Web Hooks

Send HTTP requests in response to jobs and alerts

Web hooks are outbound HTTP requests that fire when jobs complete or alerts trigger. Use them to integrate xyOps with external systems like Slack, Discord, PagerDuty, or custom APIs.

Overview

Web hooks provide:

• Fully customizable requests - Configure URL, method, headers, and body
• Templating support - Use expressions to inject job/alert data dynamically
• Action-driven - Trigger on job start, completion, specific outcomes, or alert state changes
• Observable - View request/response details and performance metrics for each execution

When hooks fire

Attach web hook actions to events, workflows, or alerts. Hooks can fire on:

Job conditions

• start - Before job launches on remote server
• complete - Any job completion (success or failure)
• success - Successful completion (exit code 0)
• error - Failed completion (non-zero exit code)
• warning - Completion with warning code
• critical - Completion with critical code
• abort - Job was aborted by user or system
• tag:TAGID - Job completed with a specific tag

Alert conditions

• alert_new - Alert was triggered
• alert_cleared - Alert condition cleared

Special conditions

• Job suspended for human intervention
• Job exceeded resource limits (CPU, memory, time, output)

Actions are deduplicated by type and target. If the same hook is referenced by an event, category, and universal actions, it only fires once.

Creating web hooks

1. Define the web hook

   Navigate to Settings → Web Hooks and click Add Web Hook.

   title · string · required

   Display name for the web hook

   url · string · required

   Target URL (http:// or https://). Supports templating.

   method · string · required

   HTTP method: GET, POST, PUT, PATCH, DELETE, HEAD

   headers · array

   Custom headers as name/value pairs. Values support templating.

   body · string

   Request body for non-GET/HEAD requests. Supports templating.

2. Configure options

   timeout · number

   Seconds to wait for response (default: 30)

   retries · number

   Automatic retries on transport errors (default: 0)

   follow · boolean

   Follow HTTP redirects (default: true)

   ssl_cert_bypass · boolean

   Skip TLS certificate validation (default: false)

   max_per_day · number

   Daily execution cap for rate limiting (0 = unlimited)

3. Attach to actions

   Add a “Web Hook” action to your events, workflows, or alerts. Select the hook and configure the condition.

4. Test the hook

   Use the Test button to send a sample request and verify connectivity.

Templating

Web hook url, headers, and body support templating with {{ ... }} expressions. Expressions are evaluated using the xyOps Expression Format.

Available context

For job hooks:

• event - Event object with id, title, category
• job - Job object with id, code, description, elapsed, cpu, mem
• server - Server object with hostname, ip, groups
• nice_server - Formatted server name
• nice_hostname - Formatted hostname (stripped)
• text - Pre-formatted notification text
• links - URLs for job details, event page, server page
• secrets - Assigned secrets (if any)

For alert hooks:

• def - Alert definition with id, title, expression
• alert - Alert invocation with message, value, started
• server - Server object
• nice_server - Formatted server name
• text - Pre-formatted notification text
• links - URLs for alert details, server page

Templating examples

Slack

POST https://hooks.slack.com/services/XXX/YYY/ZZZ
Content-Type: application/json

{
  "text": "{{text}}",
  "attachments": [
    {
      "color": "{{job.code == 0 ? 'good' : 'danger'}}",
      "fields": [
        {
          "title": "Event",
          "value": "{{event.title}}",
          "short": true
        },
        {
          "title": "Server",
          "value": "{{nice_hostname}}",
          "short": true
        }
      ]
    }
  ]
}

Discord

POST https://discord.com/api/webhooks/ID/TOKEN
Content-Type: application/json

{
  "content": "{{text}}",
  "embeds": [
    {
      "title": "{{event.title}}",
      "description": "Job {{job.code == 0 ? 'succeeded' : 'failed'}} on {{nice_server}}",
      "color": {{job.code == 0 ? 3066993 : 15158332}},
      "fields": [
        {
          "name": "Job ID",
          "value": "{{job.id}}",
          "inline": true
        },
        {
          "name": "Elapsed",
          "value": "{{float(job.elapsed)}} sec",
          "inline": true
        }
      ]
    }
  ]
}

PagerDuty

POST https://api.pagerduty.com/incidents
Authorization: Token token={{secrets.PAGERDUTY_TOKEN}}
Content-Type: application/json

{
  "incident": {
    "type": "incident",
    "title": "Alert: {{def.title}} on {{nice_server}}",
    "service": {
      "id": "PXXXXXX",
      "type": "service_reference"
    },
    "body": {
      "type": "incident_body",
      "details": "{{alert.message}}"
    }
  }
}

Generic JSON

POST https://api.example.com/events
Authorization: Bearer {{secrets.API_KEY}}
Content-Type: application/json

{
  "event_type": "job_{{job.code == 0 ? 'success' : 'failure'}}",
  "source": "xyops",
  "data": {
    "job_id": "{{job.id}}",
    "event_title": "{{event.title}}",
    "server": "{{server.hostname}}",
    "code": {{job.code}},
    "elapsed": {{float(job.elapsed)}},
    "cpu_pct": {{float(job.cpu)}},
    "mem_bytes": {{integer(job.mem)}},
    "link": "{{links.job_details}}"
  }
}

Using secrets

Store sensitive tokens as secrets and inject them via templating:

Authorization: Bearer {{secrets.API_TOKEN}}

Assign secrets to web hooks in the secret editor. Without assignment, secrets.* resolves to empty strings.

Security best practices

1. Avoid secrets in URLs - Query params may be logged by destination servers
2. Use HTTPS - Always use https:// URLs for sensitive data
3. Prefer headers or body - Put tokens in Authorization headers or request body
4. Enable TLS verification - Don’t bypass certificate checks unless absolutely necessary

Default text templates

xyOps generates a {{text}} variable for each hook execution using configurable templates. Customize these in config.json under hook_text_templates:

From ~/workspace/source/sample_conf/config.json:84-93:

"hook_text_templates": {
  "job_start": "Job started on {{nice_server}}: {{event.title}}: {{links.job_details}}",
  "job_success": "Job completed successfully on {{nice_server}}: {{event.title}}: {{links.job_details}}",
  "job_error": "Job failed on {{nice_server}}: {{event.title}}: Error ({{job.code}}): {{job.description}}: {{links.job_details}}",
  "job_progress": "Job is in progress on {{nice_server}} ({{event.title}}): {{links.job_details}}",
  "job_suspended": "Job is suspended and requires human intervention: {{event.title}}: {{links.job_details}}&resume=1",
  "job_limited": "{{action.msg}}: {{links.job_details}}",
  "alert_new": "Alert: {{nice_server}}: {{def.title}}: {{alert.message}}: {{links.alert_url}}",
  "alert_cleared": "Alert Cleared: {{nice_server}}: {{def.title}}"
}

These provide compatibility with services that expect a text, content, or message field.

Execution observability

Every web hook execution records detailed diagnostics:

• Status - HTTP status code or error message
• Timing - Total elapsed time and performance breakdown
• Request - Final URL, headers, and body with templates resolved
• Response - Status, headers, and body from the endpoint

View these in the job’s Activity tab or alert action log.

Performance breakdown

From ~/workspace/source/lib/action.js, hooks track:

• dns - DNS lookup time
• connect - TCP/TLS connection time
• send - Request transmission time
• wait - Time to first byte (TTFB)
• receive - Response download time
• compress - Request compression time
• decompress - Response decompression time

Troubleshooting

Hook doesn't fire

Check action condition: Ensure the condition matches the job outcome (e.g., error for failures).

Verify hook is enabled: Disabled hooks won’t execute.

Check daily limit: If max_per_day is set, you may have hit the cap.

Review action logs: Job Activity tab shows which actions were triggered.

Request fails with timeout

Increase timeout: Default is 30 seconds. Adjust in hook configuration.

Check network connectivity: Ensure xyOps can reach the target URL.

Enable retries: Set retries to 1-3 for transient failures.

Template variables are empty

Check context: Ensure you’re using job variables in job hooks, not alert variables.

Verify secret assignment: Secrets must be assigned to the hook in the secret editor.

Test with literal values: Replace templates with static strings to isolate the issue.

SSL/TLS errors

Check certificate validity: Ensure the destination has a valid SSL certificate.

Use ssl_cert_bypass: Only for testing—bypass TLS verification temporarily.

Update CA certificates: Ensure your OS has up-to-date root certificates.

See also

• Actions - Action system overview
• Expressions - Expression syntax and helpers
• Secrets - Secure credential management
• System Hooks - Global hooks for all activity