Servers

Server management, xySat agents, metrics collection, and server groups in xyOps

Overview

Servers are the worker nodes in a xyOps cluster. Each server runs the lightweight satellite agent (xySat), maintains a persistent WebSocket connection to the conductor, collects monitoring metrics, and executes jobs on demand. A server may be a physical host, virtual machine, or container running Linux, macOS, or Windows.

Servers run xySat and act as job runners and metrics collectors. Conductors run the full xyOps stack and coordinate scheduling, routing, storage, and UI.

Key Concepts

Servers vs. Conductors

• Server: A worker node running xySat that reports host details and metrics, and executes jobs sent by a conductor. Servers may be grouped and targeted by events.
• Conductor: A full xyOps instance (primary or hot standby) that manages the schedule, routes jobs to servers, stores data, and serves the UI/API.

xySat keeps an up-to-date list of all conductors. If a server loses its primary connection, it automatically fails over to a backup and then reconnects to the new primary after election.

Metrics Collection

Servers collect two types of metrics:

• Per-second (“quick”): CPU, memory, disk, and network retained in a rolling 60-second in-memory buffer for UI
• Per-minute (monitors): User-defined monitor plugins run each minute on servers to produce numeric values that feed charts, alerts, and dashboards

Some metrics are not available on Windows. To avoid thundering herd effects, each server deterministically staggers its minute collection offset using a hash of its Server ID.

Adding Servers

You can add servers in three ways:

1. Via the UI (one-line installer)

   • Go to the Servers tab and click “Add Server…”
   • Optionally set a label, icon, enabled state, and pick groups
   • Copy the pre-configured one-line install command for Docker, Linux, macOS or Windows
   • Run it on the target host
   • The installer authenticates, installs xySat as a startup service (systemd/launchd/Windows Service), writes the config, and starts the agent
   • The server appears immediately in the cluster and begins streaming metrics
2. Automated bootstrap (API Key)

   For autoscaling or ephemeral hosts, generate an API Key and use your provisioning to call the bootstrap endpoint during first boot.

   curl -s "https://xyops01.mycompany.com/api/app/satellite/install?t=API_KEY_HERE" | sudo sh
   
3. Manual install

   Install xySat on the host and configure it with your cluster URL and secret key. Start the service to join the cluster. This method is typically only used for development, testing and home labs.

Automated Docker Workers

To automate adding new Docker-based workers:

1. Create an API Key

   Create a new API Key in the UI with only the add_servers privilege.

2. Get the installation command

   Click “Add Server” from the sidebar, select “Docker” as the target platform, and copy the installation command.

3. Replace the token

   Replace the temporary auth token (expires after 24 hours) with your API Key in the XYOPS_setup environment variable:

   services:
     worker1:
       image: ghcr.io/pixlcore/xysat:latest
       init: true
       restart: unless-stopped
       volumes:
         - /var/run/docker.sock:/var/run/docker.sock
       environment:
         XYOPS_setup: http://YOUR_XYOPS_SERVER:5522/api/app/satellite/config?t=YOUR_API_KEY_HERE
   

Groups and Auto-Assignment

Servers can belong to one or more groups used for organizing the fleet, scoping monitors/alerts, and targeting events.

How Auto-Assignment Works

• Auto-assignment: Groups can declare a hostname regular expression. When a server comes online or its hostname changes, matching groups are applied automatically.
• Multiple groups: Servers can match and join multiple groups.
• Manual assignment: If you manually assign groups to a server, automatic hostname-based matching is disabled for that server.
• Re-evaluation: Group matches are re-evaluated if a server’s hostname changes.

Targeting Events at Servers

Events specify targets as a list containing server IDs and/or group IDs. At run time, the scheduler resolves these into the set of currently online, enabled servers, then picks one using the event’s selection algorithm.

Selection algorithms:

• random - Random selection
• round_robin - Round-robin rotation
• least_cpu - Server with lowest CPU load
• least_mem - Server with lowest memory usage
• Monitor-based policy

Behavior When Servers are Offline

• Single-server target: If the target server is offline, behavior is configurable via limits (add a Queue limit to allow queuing; without one, the job fails immediately)
• Group target: Offline servers are ignored; alternate online servers from the group are selected

Alerts can optionally suppress job launches on a specific server, so a server under alert may be excluded from selection until it clears.

User Data

xyOps can store arbitrary data with each server called “user data” - a freeform JSON object that can contain any data you want (including nested objects/arrays). This data is:

• Automatically passed to all running jobs on the server
• Available for custom event targeting
• Stored in memory on the primary conductor for all active servers

Ways to update server data:

• In the UI, on the server details page, click “Edit Server”
• Call the update_server_data API
• Inside a running job by outputting a serverData object

Server UI Page

Each server has a dedicated page showing live and historical state:

Status · section

Online/offline badge, label/hostname, IP, OS/arch, CPU details, memory, virtualization, agent version, uptime, and groups

Quick metrics · live graphs

Small rolling graphs for CPU, memory, disk, and network over the last 60 seconds (per second)

Monitors · charts

Charts for all user-defined monitors and deltas, with alert overlays (per minute)

Processes · table

Current process table showing PID, parent, CPU, memory, network, and other metrics for each process

Connections · table

Current network connections showing state, source and dest IPs, and transfer metrics

Running jobs · list

Live jobs executing on the server, including workflow parents/children

Alerts · list

Active alerts affecting this server, with links to history

Lifecycle and Health

Online/Offline Status

A server is online while its xySat WebSocket is connected. If the socket drops, the server is immediately marked offline. The UI updates in real time.

Jobs are not aborted immediately when a server goes offline. Conductors wait for dead_job_timeout (default: 120 seconds) before declaring the job dead and aborting it.

Enable/Disable

Disabling a server removes it from job selection but it can remain online and continue reporting metrics.

Decommissioning Servers

To retire a server, open its detail page and click the trash can icon:

• Online: The conductor sends an uninstall command to the agent, which shuts down and removes xySat. You can also optionally delete historical data.
• Offline: You can still delete the server but must opt to delete history, as uninstall requires an active connection.

Deletions are permanent and cannot be undone.

Code Reference

Server validation and management is implemented in /workspace/source/lib/server.js:

server.js:15-38

validateServer(server) {
    // check server hostname and groups
    var self = this;
    var group_defs = this.groups;
    
    // make sure server ID is unique
    if (server.id && this.servers[server.id]) {
        var msg = "Server with duplicate ID attempting to connect to the network: " + server.id + " (" + server.hostname + ")";
        this.logTransaction('warning', msg, server);
        server.err = msg;
        return false; // failure
    }
    
    // normalize hostname for storage (and sanity)
    server.hostname = this.storage.normalizeKey( server.hostname ).replace(/\//g, '');
    
    // warn if dupe hostname -- not an error
    var dupes = this.findActiveServers({ hostname: server.hostname });
    if (dupes.length) {
        this.logTransaction('warning', "Server connecting with duplicate hostname: " + server.hostname, server);
    }
    
    return true;
}

Related APIs

• get_active_servers - List all currently online servers
• get_active_server - Get details for a specific active server
• get_server - Get server details from database
• update_server - Update server configuration
• delete_server - Remove server from cluster
• watch_server - Start a watch to take snapshots every minute
• create_snapshot - Take a snapshot of server state
• search_servers - Search server history and summaries