Notification Channels

Overview

Notification Channels bundle multiple notification targets and follow-up actions under a single reusable name. Instead of configuring individual emails, webhooks, or event launches everywhere, reference a channel and xyOps executes all configured actions together.

Channels streamline incident response by grouping related notifications into a single action.

Key Features

Reusable Bundles

Define once, reference from events, workflows, and alerts

Multi-Action

Email, webhook, event launch, and in-app notifications in one

Rate Limiting

Optional per-day caps to prevent notification floods

Rich Context

Templated payloads with job logs, metadata, and direct links

Use Cases

Channels excel at standardizing incident response:

Severity 1 Incidents

{
  "id": "sev1",
  "title": "Severity 1",
  "enabled": true,
  "icon": "bullhorn-outline",
  "users": ["oncall", "noc"],
  "email": "ops@example.com, sre@example.com",
  "web_hook": "slack_ops",
  "run_event": "auto_remediate",
  "sound": "attention-3.mp3",
  "max_per_day": 100,
  "notes": "Page on-call for P1 incidents"
}

This single channel:

Emails on-call and NOC teams
Sends external email addresses
Posts to Slack via webhook
Launches auto-remediation event
Plays audible alert in UI for connected users
Caps at 100 invocations per day

Team Notifications

{
  "id": "team_payments",
  "title": "Payments Team",
  "enabled": true,
  "icon": "cash",
  "users": ["payments_lead", "payments_oncall"],
  "web_hook": "slack_payments",
  "notes": "Notify payments team of job outcomes"
}

Creating Channels

Navigate to Channels

Go to Admin → Channels

Define Channel Properties

Set ID, title, icon, and enable status

{
  "id": "ops_critical",
  "title": "Operations Critical",
  "enabled": true,
  "icon": "alert-circle"
}

Configure Recipients

Add users (by username) and external email addresses

{
  "users": ["oncall", "sre_lead"],
  "email": "ops-alerts@example.com"
}

Add Actions

Configure webhook, event launch, and in-app notification

{
  "web_hook": "slack_ops",
  "run_event": "incident_response",
  "sound": "attention-3.mp3"
}

Set Rate Limit (Optional)

Prevent notification floods with per-day cap

{
  "max_per_day": 50
}

Set to 0 for unlimited.

Channel Actions

When invoked, channels execute these actions in parallel:

Email

Emails all channel users plus any explicit addresses:

{
  "users": ["dev", "qa"],
  "email": "external@example.com, alerts@example.com"
}

User emails are looked up from account settings
External addresses receive identical content
Email templates include job context or alert details

Web Hook

Fires configured webhook with rich, templated payload:

{
  "web_hook": "slack_ops"
}

Payload includes:

For jobs: Job ID, title, status, log excerpts, output data, direct link
For alerts: Server details, alert name, current value, threshold, direct link

Run Event

Launches follow-up or remediation event:

{
  "run_event": "auto_remediate"
}

Inherits context:

Job actions: Parent job’s output data and files passed to child
Alert actions: Alert metadata included in child job’s input data

In-App Notification

Sends UI notification to all channel users with optional sound:

{
  "users": ["oncall"],
  "sound": "attention-3.mp3"
}

Notifications appear as popups for connected users
Sound files must be .mp3 format
Links directly to triggering job or alert

Using Channels

Reference channels via actions on events, workflows, or alerts:

Event Action

Job Error Notification

{
  "id": "deploy_app",
  "title": "Deploy Application",
  "category": "prod",
  "actions": [
    {
      "enabled": true,
      "condition": "error",
      "type": "channel",
      "channel_id": "sev1"
    }
  ]
}

When the job fails, the sev1 channel executes all configured actions.

Alert Action

Alert Fired Notification

{
  "id": "cpu_high",
  "title": "High CPU Usage",
  "actions": [
    {
      "enabled": true,
      "condition": "alert_new",
      "type": "channel",
      "channel_id": "ops_critical"
    },
    {
      "enabled": true,
      "condition": "alert_cleared",
      "type": "channel",
      "channel_id": "ops_info"
    }
  ]
}

Different channels for alert states (fired vs. cleared).

Workflow Action

Workflow Completion

{
  "id": "data_pipeline",
  "title": "Data Pipeline",
  "actions": [
    {
      "enabled": true,
      "condition": "success",
      "type": "channel",
      "channel_id": "team_data"
    }
  ]
}

Rate Limiting

Prevent notification fatigue with per-day caps:

{
  "id": "auto_alerts",
  "title": "Automated Alerts",
  "max_per_day": 100
}

Counter resets daily at midnight (server timezone)
When limit is reached, channel actions are skipped
Skip event is recorded in job activity log or alert history
Set to 0 for unlimited invocations

Use rate limits for high-frequency alerts or automated jobs that might spike unexpectedly.

Behavior Details

Parallel Execution

All channel actions execute concurrently:

Email is sent
Webhook is fired
Event is launched
In-app notifications appear

Results are aggregated and logged together.

Action Deduplication

xyOps deduplicates actions by type and target:

Multiple references to the same channel run only once per trigger
Actions within a channel execute once as part of the bundle
Deduplication happens at the job/alert level

Disabled Channels

When a channel is disabled:

Actions referencing it are skipped
A message is recorded in activity logs
No notifications are sent

Use this to temporarily silence channels without removing references.

Templates

Email and webhook payloads use standard templates:

Job context: Job ID, title, category, status, elapsed time, log excerpts, output data
Alert context: Server hostname, alert name, current value, threshold, duration

Channels don’t add custom text. For custom messages, use direct email/webhook actions.

API Reference

List All Channels

GET /api/app/get_channels/v1

Get Single Channel

GET /api/app/get_channel/v1?id=sev1

Create Channel

POST /api/app/create_channel/v1

Request Body

{
  "title": "Severity 1",
  "enabled": true,
  "icon": "bullhorn-outline",
  "users": ["oncall"],
  "email": "ops@example.com",
  "web_hook": "slack_ops",
  "run_event": "auto_remediate",
  "sound": "attention-3.mp3",
  "max_per_day": 100
}

ID is auto-generated using pattern ch<random>. Optionally provide your own ID.

Update Channel

POST /api/app/update_channel/v1

Request Body

{
  "id": "sev1",
  "title": "Severity 1 Incidents",
  "max_per_day": 150
}

Shallow-merge updates. Increments revision counter.

Delete Channel

POST /api/app/delete_channel/v1

Request Body

{
  "id": "sev1"
}

Deleting a channel doesn’t remove references. Actions will skip and log warnings.

Auditing

Channel execution is fully auditable:

Job Actions

Results appear in the job’s Activity log:

Channel invocation timestamp
Each sub-action result (email sent, webhook fired, event launched)
Rate limit status
Any errors or warnings

Alert Actions

Results are stored with the alert’s action history:

Timestamp and condition
Channel details
Sub-action outcomes

Permissions

These privileges control channel management:

create_channels - Create new channels
edit_channels - Modify existing channels
delete_channels - Remove channels

Using a channel in an action doesn’t require special privileges. Only managing channel definitions requires privileges.

Best Practices

One Channel Per Severity

Create separate channels for different incident severities:

sev1 - Page on-call, escalate immediately
sev2 - Notify team, no immediate action
sev3 - Log only, no notifications

This provides clear escalation paths.

Team-Based Channels

Organize channels by team ownership:

team_ops - Operations team
team_security - Security team
team_data - Data engineering team

Each team controls their notification preferences.

Use Rate Limits Wisely

Set rate limits based on expected frequency:

High-frequency alerts: 50-100 per day
Moderate alerts: 20-50 per day
Critical incidents: 200+ per day (don’t miss important alerts)

Monitor channel invocation counts and adjust limits as needed.

Idempotent Remediation Events

If your channel launches a remediation event, ensure the event is:

Safe to run multiple times
Checks current state before taking action
Doesn’t cause cascading failures

Avoid infinite loops where remediation triggers more alerts.

Test Before Production

Create test channels and validate:

Email delivery
Webhook payloads
Event launches
In-app notifications

Use a staging environment to verify end-to-end behavior.

Available Sounds

The UI provides curated .mp3 alert sounds:

attention-1.mp3 - Short beep
attention-2.mp3 - Double beep
attention-3.mp3 - Triple beep
alert-1.mp3 - Rising tone
alert-2.mp3 - Urgent tone

Preview sounds in the channel editor before selecting.

Documentation Index

​Overview

​Key Features

Reusable Bundles

Multi-Action

Rate Limiting

Rich Context

​Use Cases

​Severity 1 Incidents

​Team Notifications

​Creating Channels

​Channel Actions

​Email

​Web Hook

​Run Event

​In-App Notification

​Using Channels

​Event Action

​Alert Action

​Workflow Action

​Rate Limiting

​Behavior Details

​Parallel Execution

​Action Deduplication

​Disabled Channels

​Templates

​API Reference

​List All Channels

​Get Single Channel

​Create Channel

​Update Channel

​Delete Channel

​Auditing

​Job Actions

​Alert Actions

​Permissions

​Best Practices

​Available Sounds

Overview

Key Features

Use Cases

Severity 1 Incidents

Team Notifications

Creating Channels

Channel Actions

Email

Web Hook

Run Event

In-App Notification

Using Channels

Event Action

Alert Action

Workflow Action

Rate Limiting

Behavior Details

Parallel Execution

Action Deduplication

Disabled Channels

Templates

API Reference

List All Channels

Get Single Channel

Create Channel

Update Channel

Delete Channel

Auditing

Job Actions

Alert Actions

Permissions

Best Practices

Available Sounds