Troubleshooting

Common issues, debugging techniques, and solutions for xyOps

This guide helps you diagnose and resolve common issues when working with xyOps.

Common Issues

Jobs Not Running

Check Scheduler Status

Ensure the scheduler is enabled:

• Navigate to the System tab in the UI
• Check if the scheduler is paused
• Resume the scheduler if needed

Verify Event Configuration

• Check that the event is enabled
• Verify triggers are properly configured
• Ensure target servers are online and connected
• Review any Max Concurrent Jobs or Max Queue Limit settings

Check Server Connectivity

• Verify satellite connections in the Servers tab
• Check network connectivity between conductor and satellites
• Review firewall rules (default ports: 5522 for HTTP, 5523 for WebSocket)

Job Failures

Review Job Logs

• Navigate to the job details page
• Check the job output and error messages
• Review the metadata log for internal errors
• Look for resource limit violations (CPU, memory, runtime)

Plugin Issues

• Verify the plugin is properly installed
• Check plugin parameters are correctly configured
• Ensure required dependencies are installed on target servers
• Test the plugin manually if possible

Connection Issues

Satellite Connection Problems

If satellites are showing as disconnected:

• Check that the satellite service is running on the worker server
• Verify network connectivity between satellite and conductor
• Review satellite logs: /opt/xysat/logs/
• Check WebSocket port (default 5523) is accessible
• Verify SSL/TLS certificates if using HTTPS

WebSocket Timeouts

If you’re experiencing frequent disconnections:

• Adjust ping_freq_sec (default: 5) and ping_timeout_sec (default: 30) in config
• Check for packet loss or network instability
• Review firewall settings for WebSocket connections

Debugging Techniques

Starting in Debug Mode

To start xyOps in debug mode, which echoes the entire debug log to the console:

./bin/debug.sh

Beware of file permissions if you run as a non-root user. Hit Ctrl-C twice to shut down the service in debug mode.

Customize log categories:

./bin/debug.sh "xyOps Transaction Error API Unbase Action Comm Job Workflow Maint Multi Scheduler SSO"

This is useful for muting extremely loud components like Storage and WebServer.

Using the REPL

By default, the debug.sh script starts a REPL in the console for live JavaScript execution.

Available globals:

Global	Description
server	The current global pixl-server instance
cli	The pixl-cli global with utility functions
xyOps (or xy)	The main xyOps server component
Storage	The pixl-server-storage component
Unbase	The pixl-server-unbase component
WebServer	The pixl-server-web component
API	The pixl-server-api component
User	The pixl-server-user component

REPL commands:

Command	Description
.echo	Add or remove echo categories, e.g., .echo add Storage WebServer
.notify	Send a notification to all users, e.g., .notify HI THERE

Log Locations

xyOps Conductor Logs:

/opt/xyops/logs/

Satellite Logs:

/opt/xysat/logs/

Log Viewer in UI:

• Navigate to System > Log Viewer
• Filter by date, log level, and component
• Search for specific patterns

Checking Service Status

Linux (systemd):

sudo systemctl status xyops
sudo systemctl status xysat

Docker:

docker logs xyops01
docker logs xysat01

Performance Issues

High CPU Usage

Job Resource Limits

• Add Max CPU Limit to resource-intensive events
• Use the “stagger” option for multiplexed jobs
• Review concurrent job limits

Database Optimization

• Navigate to System > Database Optimization
• Run optimization to clean up WAL files and compact database
• Consider external storage (S3, MinIO) for job logs and files

High Memory Usage

Configure Memory Limits

• Add Max Memory Limit to events
• Review job output sizes and consider streaming large outputs
• Check for memory leaks in custom plugins

Storage Configuration

• Configure S3-compatible storage for job data
• Enable caching with appropriate maxItems and maxBytes
• Review log retention settings

Alert Issues

Alerts Not Firing

1. Check Alert Configuration

   • Verify the alert is enabled
   • Review the monitor expression and thresholds
   • Check that the server being monitored is online
2. Test Monitor Expression

   • Use the REPL to test JEXL expressions
   • Verify data sources are available
   • Check for syntax errors in expressions
3. Review Actions

   • Ensure actions are properly configured
   • Test Web Hooks or Email actions separately
   • Check for rate limiting on notifications

Alert Stuck in Limbo

If alerts get stuck after server disconnect/reconnect:

• Check Activity Log for alert lifecycle events
• Verify server reconnection was successful
• Consider restarting the xyOps service if alerts remain stuck

Plugin Issues

Plugin Not Found

Verify Plugin Installation

• Check Plugins tab to confirm plugin exists
• Verify plugin is not disabled
• For Marketplace plugins, ensure they’re properly installed

Check Plugin Compatibility

• Verify xyOps version compatibility
• For Cronicle plugins, ensure compatibility mode is enabled
• Review plugin documentation for dependencies

Plugin Execution Errors

Always review plugin output in job logs for specific error messages.

Common Plugin Errors

• Permission denied: Check file permissions and execute bit
• Module not found: Install missing dependencies on target servers
• Timeout: Adjust Max Run Time limit or optimize plugin code
• JSON parse error: Ensure plugin emits valid JSON with xy: 1 property

Database Issues

Database Lock Errors

SQLite Specific

• Check for concurrent write operations
• Review WAL (Write-Ahead Logging) file size
• Run Database Optimization from System tab
• Consider external storage for high-throughput scenarios

Storage Migration

Migrating to External Storage

For better scalability, migrate to S3-compatible storage:

1. Configure S3 storage in config.json
2. Test connection with a small event
3. Use bulk export/import for data migration
4. Update backup scripts to target new storage

Authentication Issues

Cannot Login

Reset Admin Password

Use the command-line tool to reset the admin password:

cd /opt/xyops
node bin/storage-cli.js admin/reset_password admin NEW_PASSWORD

SSO Issues

• Verify SSO configuration in config.json
• Check SAML/OIDC provider settings
• Review group role mappings
• Check SSO logs in Activity tab

Getting Help

If you’re still experiencing issues after trying these troubleshooting steps:

• Check GitHub Discussions for community support
• Review GitHub Issues for known bugs
• Consult the full documentation at docs.xyops.io
• For urgent issues, consider the Enterprise support plan