Limits - xyOps

Overview

Limits are self-imposed restrictions you can place on your events to govern resource usage as jobs run, as well as specify options such as max number of retries, or max allowed jobs to queue up. Limits can be defined at several different levels, including directly on events, attached as workflow nodes, inherited from categories, or inherited from your global configuration file (“universal” limits).

Limit Precedence

When multiple limits of the same type are present for a job, xyOps applies them based on source priority:

Event-Defined Limits

Highest priority - limits defined directly on the event

Workflow Limit Nodes

Limits attached as nodes in workflow graphs

Category-Inherited Limits

Default limits from the event’s category

Universal-Inherited Limits

Lowest priority - global defaults from config

Single vs Multiple Limits

Single Limit Types (only first enabled limit applies):

Max Concurrent Jobs
Max Retry Limit
Max Queue Limit
Max File Limit

Multiple Limit Types (all enabled limits apply):

Max Run Time
Max Output Size
Max CPU Limit
Max Memory Limit

For example, you can emit a warning at 500MB memory usage and abort at 1GB by adding two separate memory limits.

Where Limits Are Defined

Event/Workflow Editor: Add limits directly to a specific job or workflow
Category Editor: Add default limits that all events in the category inherit
Configuration: Add universal defaults in job_universal_limits

Limit Types

Max Run Time

Enforce a soft or hard cap on total job elapsed time. When exceeded, optional actions can be taken and the job can be aborted. Parameters:

Name	Type	Description
`type`	String	Set to `time`
`duration`	Number	Maximum runtime in seconds
`tags`	Array(String)	Tag IDs to apply when exceeded
`users`	Array(String)	Usernames to email
`email`	String	Additional comma-separated addresses
`web_hook`	String	WebHook ID to fire
`text`	String	Custom text for webhook
`snapshot`	Boolean	Take server snapshot
`abort`	Boolean	Abort the job

Example:

{
  "enabled": true,
  "type": "time",
  "duration": 3600,
  "tags": ["limited"],
  "users": ["oncall"],
  "email": "ops@example.com",
  "web_hook": "slack_ops",
  "text": "Runaway protection triggered",
  "snapshot": true,
  "abort": true
}

Max Concurrent Jobs

Limit how many jobs of the same event/workflow may run at once. If the cap is reached, xyOps can queue the job if a queue limit allows it; otherwise the job is aborted. Parameters:

Name	Type	Description
`type`	String	Set to `job`
`amount`	Number	Maximum concurrent active jobs

{
  "enabled": true,
  "type": "job",
  "amount": 2
}

For workflows, scope matches the workflow’s event. For ad-hoc workflow node jobs, the queue scope includes the node ID.

Max Output Size

Cap the job’s output/log size in bytes. When exceeded, optional actions can be taken and the job can be aborted. Parameters:

Name	Type	Description
`type`	String	Set to `log`
`amount`	Number	Maximum bytes of output/log
`tags`	Array(String)	Tags to apply
`users`	Array(String)	Users to email
`email`	String	Additional addresses
`web_hook`	String	WebHook ID
`text`	String	Webhook text
`snapshot`	Boolean	Take snapshot
`abort`	Boolean	Abort job

{
  "enabled": true,
  "type": "log",
  "amount": 10485760,
  "users": ["sre"],
  "abort": true
}

Max Memory Limit

Cap total memory usage for the job (including child processes). The limit triggers only if usage stays over the threshold continuously for the sustain duration. Parameters:

Name	Type	Description
`type`	String	Set to `mem`
`amount`	Number	Maximum memory in bytes
`duration`	Number	Sustain time in seconds
`tags`	Array(String)	Tags to apply
`users`	Array(String)	Users to email
`email`	String	Additional addresses
`web_hook`	String	WebHook ID
`text`	String	Webhook text
`snapshot`	Boolean	Take snapshot
`abort`	Boolean	Abort job

{
  "enabled": true,
  "type": "mem",
  "amount": 1073741824,
  "duration": 30,
  "tags": ["memoryhot"],
  "snapshot": true,
  "abort": true
}

Sustain Period Details

Memory limits use a “sustain” mechanism:

// Source: lib/job.js:500-519
Tools.findObjects( job.limits, { type: 'mem', enabled: true } )
  .forEach( function(limit) {
    if (job.mem.current > limit.amount) {
      if (!limit.when) {
        limit.when = now;
        self.logJob(6, "Job has exceeded memory usage limit: " + job.id);
      }
      if ((now - limit.when > limit.duration) && !limit.date) {
        self.triggerActiveJobLimit(job, limit);
      }
    } else if (limit.when) {
      self.logJob(6, "Job is now under the memory usage limit: " + job.id);
      delete limit.when; // reset sustain timer
    }
  });

The limit only triggers if memory stays above threshold for the entire sustain duration.

Max CPU Limit

Cap CPU usage for the job (including child processes). The limit triggers only if CPU stays over the threshold continuously for the sustain duration. Parameters:

Name	Type	Description
`type`	String	Set to `cpu`
`amount`	Number	CPU percentage (100 = one core)
`duration`	Number	Sustain time in seconds
`tags`	Array(String)	Tags to apply
`users`	Array(String)	Users to email
`email`	String	Additional addresses
`web_hook`	String	WebHook ID
`text`	String	Webhook text
`snapshot`	Boolean	Take snapshot
`abort`	Boolean	Abort job

{
  "enabled": true,
  "type": "cpu",
  "amount": 250,
  "duration": 20,
  "users": ["oncall"],
  "web_hook": "slack_ops",
  "abort": true
}

Max Retry Limit

Control how many retries are attempted for failed jobs, and optionally how long to wait between retries. Parameters:

Name	Type	Description
`type`	String	Set to `retry`
`amount`	Number	Maximum retries (0 disables)
`duration`	Number	Delay in seconds between retries

{
  "enabled": true,
  "type": "retry",
  "amount": 3,
  "duration": 60
}

On each retry, xyOps clones the job context, increments retry_count, and optionally delays before relaunch.

Max Queue Limit

Cap how many jobs are allowed to wait in the queue when concurrency or server availability prevents immediate start. Parameters:

Name	Type	Description
`type`	String	Set to `queue`
`amount`	Number	Maximum queued jobs (0 disables)

{
  "enabled": true,
  "type": "queue",
  "amount": 25
}

Without a queue limit, jobs are aborted when they cannot start due to concurrency or server selection limits.

Max File Limit

Soft limit that prunes incoming files (from job input) before launch. Can cap the number of files, total combined size, and restrict file types by extension. Parameters:

Name	Type	Description
`type`	String	Set to `file`
`amount`	Number	Maximum files (0 means no files)
`size`	Number	Maximum total bytes
`accept`	String	Comma-separated extensions (e.g., `.json,.csv`)

{
  "enabled": true,
  "type": "file",
  "amount": 100,
  "size": 52428800,
  "accept": ".json,.csv,.tsv"
}

This limit never aborts the job; it prunes files and logs what was removed.

Max Daily Limit

Quietly prevent additional job launches if a specific daily condition count has been reached for the event. Parameters:

Name	Type	Description
`type`	String	Set to `day`
`condition`	String	Job condition to track (`complete`, `error`, etc.)
`amount`	Number	Maximum conditions per day

{
  "enabled": true,
  "type": "day",
  "condition": "complete",
  "amount": 100
}

Use Cases

Cap total runs: Set condition to complete to limit total jobs per day
E-brake on errors: Set condition to critical to stop after N critical errors
Throttle warnings: Set condition to warning to prevent warning spam

Manual job runs (by user or API key) skip this check.

Universal Limits

Set universal defaults in server config:

"job_universal_limits": {
  "default": [
    { 
      "enabled": true, 
      "type": "retry", 
      "amount": 2, 
      "duration": 30 
    },
    { 
      "enabled": true, 
      "type": "queue", 
      "amount": 100 
    }
  ],
  "workflow": []
}

You can define separate arrays for default (regular events) and workflow limits.

Limit Enforcement

Start-Time Enforcement

Before a job starts, the system evaluates:

// Source: lib/job.js:688-794
checkJobStartLimits(job) {
  // Check concurrent job limit
  var job_limit = Tools.findObject( job.limits, { type: 'job', enabled: true } );
  
  if (job_limit && job_limit.amount) {
    var jobs = this.findSimilarJobs(job, { state: 'active' });
    
    if (jobs.length >= job_limit.amount) {
      var queue_limit = Tools.findObject( job.limits, 
        { type: 'queue', enabled: true } );
      
      if (queue_limit && queue_limit.amount) {
        var queued = this.findSimilarJobs(job, { state: 'queued' });
        
        if (queued.length < queue_limit.amount) {
          // Room in queue
          job.state = 'queued';
          job.position = queued.length + 1;
          return false;
        }
      }
      
      // No room - abort
      this.abortJob(job, "Maximum concurrent jobs reached.");
      return false;
    }
  }
  
  // Check file limit and prune if needed
  var file_limit = Tools.findObject( job.limits, { type: 'file', enabled: true } );
  if (file_limit && info.input && info.input.files) {
    // Prune by count, size, and extensions
  }
  
  return true;
}

Runtime Enforcement

time, log, mem, and cpu limits are checked while jobs run:

// Source: lib/job.js:469-541
checkJobActiveLimits(job) {
  var now = Tools.timeNow();
  
  // Check dead job timeout first
  if (now - job.updated >= this.config.get('dead_job_timeout')) {
    job.retry_ok = true;
    this.abortJob(job, "No updates received, assuming job is dead.");
    return;
  }
  
  // Time limits
  Tools.findObjects( job.limits, { type: 'time', enabled: true } )
    .forEach( function(limit) {
      if (now - job.started > limit.duration) {
        self.triggerActiveJobLimit(job, limit);
      }
    });
  
  // Log size limits
  Tools.findObjects( job.limits, { type: 'log', enabled: true } )
    .forEach( function(limit) {
      if (job.log_file_size > limit.amount) {
        self.triggerActiveJobLimit(job, limit);
      }
    });
  
  // Memory and CPU with sustain periods...
}

Triggered Actions

When a limit is exceeded, configured actions execute:

// Source: lib/job.js:543-686
triggerActiveJobLimit(job, limit) {
  limit.code = 0;
  limit.date = Tools.timeNow();
  limit.elapsed_ms = 0;
  
  // Apply tags if configured
  if (limit.tags && limit.tags.length) {
    if (!job.tags) job.tags = [];
    job.tags = job.tags.concat( limit.tags );
    job.tags = [...new Set(job.tags)];
  }
  
  async.parallel([
    // Send email
    function(callback) {
      if (!limit.users.length && !limit.email) return callback();
      var sub_action = { type: 'email', template: 'job_limited', ... };
      self.runJobAction_email(job, sub_action, callback);
    },
    // Fire webhook
    function(callback) {
      if (!limit.web_hook) return callback();
      var sub_action = { type: 'web_hook', template: 'job_limited', ... };
      self.runJobAction_web_hook(job, sub_action, callback);
    },
    // Take snapshot
    function(callback) {
      if (!limit.snapshot) return callback();
      var sub_action = { type: 'snapshot', ... };
      self.runJobAction_snapshot(job, sub_action, callback);
    }
  ], function() {
    // Abort if configured
    if (limit.abort) {
      job.retry_ok = true;
      self.abortJob(job, msg);
    }
  });
}

All actions are recorded in the job’s Activity log with details.

Notes and Behavior

Start-time enforcement: job, queue, and file limits are evaluated before launch
Runtime enforcement: time, log, mem, cpu are checked while the job runs
Sustain periods: mem and cpu require sustained overages for their duration before triggering
Multiple limits: If multiple sources define the same type, event/workflow definition takes precedence for start-time checks
Queue scope: Queues are per event. For ad-hoc workflow node runs, the queue scope includes the node identifier
Queues usage: Used both when job concurrency is saturated and when no matching servers are available

Events

Learn about event configuration

Actions

Configure limit-triggered actions

Jobs

Understand job lifecycle

Workflows

Attach limits to workflow nodes

Documentation Index

​Overview

​Limit Precedence

​Single vs Multiple Limits

​Where Limits Are Defined

​Limit Types

​Max Run Time

​Max Concurrent Jobs

​Max Output Size

​Max Memory Limit

​Max CPU Limit

​Max Retry Limit

​Max Queue Limit

​Max File Limit

​Max Daily Limit

​Universal Limits

​Limit Enforcement

​Start-Time Enforcement

​Runtime Enforcement

​Triggered Actions

​Notes and Behavior

​Related Documentation

Events

Actions

Jobs

Workflows

Overview

Limit Precedence

Single vs Multiple Limits

Where Limits Are Defined

Limit Types

Max Run Time

Max Concurrent Jobs

Max Output Size

Max Memory Limit

Max CPU Limit

Max Retry Limit

Max Queue Limit

Max File Limit

Max Daily Limit

Universal Limits

Limit Enforcement

Start-Time Enforcement

Runtime Enforcement

Triggered Actions

Notes and Behavior

Related Documentation