System Hooks

Run custom actions in response to global xyOps activity

System Hooks let you run background actions in response to any activity across xyOps—user transactions, system events, job completions, alert triggers, and notices. Configure hooks in config.json to send web requests, execute shell commands, send emails, or create tickets.

Overview

System Hooks differ from regular Actions:

• Global scope - Fire for all activity, not just specific events or alerts
• Configuration-based - Defined in config.json, not in the UI
• Background execution - Run asynchronously without blocking the main activity
• Firehose capability - Can capture every transaction with a wildcard hook

Configuration

Define hooks in config.json under the hooks object:

"hooks": {
  "error": {
    "shell_exec": "/opt/system/bin/notify-error.sh"
  },
  "critical": {
    "shell_exec": "/opt/system/bin/notify-critical.sh",
    "email": "[e-mail]",
    "ticket": {
      "type": "issue",
      "assignees": ["admin"]
    }
  },
  "secret_access": {
    "web_hook": "mkd0t84o7hwasu0g"
  }
}

From ~/workspace/source/docs/syshooks.md:11-37.

Activity types

System Hooks can trigger on any activity type ID. Common examples:

User transactions

• user_create, user_update, user_delete - User account changes
• event_create, event_update, event_delete - Event management
• workflow_create, workflow_update, workflow_delete - Workflow changes
• alert_create, alert_update, alert_delete - Alert definitions
• secret_create, secret_update, secret_delete - Secret management
• secret_access - When a secret is accessed by a job or plugin

System events

• server_add, server_update, server_remove - Server changes
• conductor_primary - Conductor became primary
• conductor_secondary - Conductor became secondary

Log levels

• notice - Informational notices
• warning - Non-critical warnings
• error - Recoverable errors
• critical - Critical failures requiring attention

Job and alert conditions

All action conditions fire system hooks with a job_ or alert_ prefix:

• job_start, job_complete, job_success, job_error, job_warning, job_critical, job_abort
• job_tag:TAGID - Jobs with specific tags
• alert_new, alert_cleared

See the full list at Activity.action in the source documentation.

Firehose hook

Capture all activity with a wildcard * key:

"hooks": {
  "*": {
    "shell_exec": "/opt/system/bin/firehose.sh"
  }
}

Firehose hooks execute for every activity. Use with caution—high-volume systems can generate thousands of events per minute.

Hook action types

System Hooks support four action types. Combine multiple actions in a single hook—they all run in parallel.

Web hook

Two options for sending HTTP requests:

1. Simple URL - POST activity payload as JSON:

"hooks": {
  "alert_new": {
    "url": "http://alerts.company.com/api/new-alert/v1"
  }
}

2. Configured web hook - Reference a web hook ID managed in xyOps UI:

"hooks": {
  "alert_new": {
    "web_hook": "wmkd2yx4yw4ihh7lu"
  }
}

Configured web hooks support full templating, custom headers, and all features of regular Web Hooks.

Shell exec

Execute a command on the primary conductor server. Activity payload is sent to STDIN as JSON:

"hooks": {
  "critical": {
    "shell_exec": "/path/to/script.sh"
  }
}

The payload follows the xyOps Wire Protocol format (single-line JSON).

Shell exec runs on the primary conductor only. Use with extreme caution. Scripts have full system access.

Debugging shell hooks

Search for System Shell Hook in /opt/xyops/logs/Action.log. Set debug_level: 9 in config.json for verbose logging.

Send email

Send a generic activity summary email:

"hooks": {
  "critical": {
    "email": "[e-mail]"
  }
}

Multiple recipients can be comma-separated:

"email": "[e-mail], [e-mail]"

The email includes:

• Activity summary
• Full JSON payload
• Any details provided (common for notice, warning, error, critical)

For alert emails, use alert_universal_actions instead. It generates alert-specific emails with better formatting.

Create ticket

Automatically create tickets for activity:

"hooks": {
  "critical": {
    "ticket": {
      "type": "issue",
      "assignees": ["admin"],
      "tags": ["automated", "critical"]
    }
  }
}

The ticket is populated with activity details and includes a link to the related resource (job, alert, user, etc.).

Payload format

All System Hooks receive an Activity payload:

{
  "action": "event_create",
  "username": "admin",
  "ip": "192.168.1.10",
  "summary": "Created event: Daily Backup",
  "date": 1763256572,
  "details": {
    "event": {
      "id": "emi11ejdlde",
      "title": "Daily Backup",
      "enabled": true
    }
  }
}

Key fields:

• action - Activity type ID
• username - User who triggered the action (or system for automated actions)
• summary - Human-readable description
• date - Unix timestamp
• details - Activity-specific data (varies by action type)

Examples

Alert all critical errors to Slack

"hooks": {
  "critical": {
    "web_hook": "wmkd2yx4yw4ihh7lu"
  }
}

Where wmkd2yx4yw4ihh7lu is a Slack web hook configured in the xyOps UI.

Log all secret access to a security log

"hooks": {
  "secret_access": {
    "shell_exec": "/opt/system/bin/log-secret-access.sh"
  }
}

Script example:

#!/bin/bash
# Read JSON from STDIN
read -r payload
echo "$(date -Iseconds) $payload" >> /var/log/xyops-secrets.log

Create tickets for all job failures

"hooks": {
  "job_error": {
    "ticket": {
      "type": "issue",
      "assignees": ["ops-team"],
      "tags": ["job-failure", "automated"]
    }
  }
}

Email admins when servers disconnect

"hooks": {
  "server_remove": {
    "email": "[e-mail]"
  }
}

Multi-channel critical alert

"hooks": {
  "critical": {
    "email": "[e-mail]",
    "web_hook": "wmkd2yx4yw4ihh7lu",
    "ticket": {
      "type": "incident",
      "assignees": ["admin"],
      "tags": ["critical", "automated"]
    },
    "shell_exec": "/opt/system/bin/page-oncall.sh"
  }
}

This fires email, Slack, creates a ticket, and executes a paging script—all in parallel.

Best practices

Use configured web hooks

Reference web hook IDs instead of inline URLs for better control and templating support.

Avoid shell exec in production

Shell scripts run with full system privileges. Prefer web hooks and tickets for security.

Filter carefully

Firehose hooks execute constantly. Use specific activity types to reduce overhead.

Monitor hook performance

Check /opt/xyops/logs/Action.log for slow hooks that block activity processing.

Test with debug level 9

Enable verbose logging temporarily to troubleshoot hook execution.

Combine actions sparingly

Multiple actions per hook run in parallel. Ensure they don’t conflict or duplicate work.

Comparison: System Hooks vs Actions

Feature	System Hooks	Actions
Scope	Global (all activity)	Scoped to events/alerts
Configuration	config.json	UI or API
Conditions	Activity type IDs	Job outcomes, alert states
Web hooks	Simple URL or ID reference	Full template control
Shell exec	Yes (conductor only)	No
Tickets	Yes	Yes
Email	Generic summary	Templated with context
Firehose	Yes (* wildcard)	No

See also

• Actions - Event and alert action system
• Web Hooks - Detailed web hook configuration
• Tickets - Ticket creation and management
• Data Formats - Activity payload structure