Off-Chain Services

Seven systemd services run alongside the on-chain program on a Hetzner CPX22 server (178.104.120.151) — six always-on Node services plus an obv-engine Python sidecar. All are deployed with automatic restart, health endpoints, and journald logging. Every operation they perform is permissionless — the services are a guarantee that work happens on schedule, not a centralized operator.

Service topology

Each service exposes a /health endpoint returning {status, uptime, lastCycle, errors} for the monitor service to poll. The obv-engine is consumed by the crank’s engine/live-processor.ts over HTTP (POST /api/live-obv/matches/{id}/{start,events,end}) when ENABLE_REALTIME_OBV=true and OBV_ENGINE_BASE_URL is set; the legacy heuristic impact-estimation path is retained as a fallback.

1. Oracle Crank + Candle API + WS (port 3456)

The central service. Three responsibilities in one process:

Crank (core loop)

Every 5 minutes:

1. Fetch season stats for all 20 teams and eligible players from the post-match REST feed.
2. Fetch recent match results (for form + PPG calculation).
3. Subscribe to the live event feed (GraphQL subscriptions) for any in-progress matches.
4. Compute composite indices via index-calculator.ts — z-scored, scaled to 100–900.
5. Persist each tick to SQLite (candles.db) for historical candle reconstruction.
6. Broadcast the tick on WebSocket to connected frontends.
7. Signal the oracle pusher that a new value is available.

Candle API (Express REST)

Exposes /candles/{market_key}?tf=1m|1H|4H|1D&limit=N returning OHLC bars. Frontend charts consume this via HTTPS proxy routes.

WebSocket server

Streams live ticks, live-match events (goals, key passes, defensive actions), and aggregate state changes. Consumed by the trading UI for real-time chart updates.

Also hot-reloads roster.json via fs.watch — any roster change takes effect on the next cycle without a restart.

2. Oracle Pusher (port 3458)

A dedicated bridge between the crank’s SQLite output and the on-chain program:

1. Reads new index values from candles.db.
2. Decides whether to push: triggered at > 0.5% change from last pushed value (> 0.1% during live matches) or at > 2-hour staleness.
3. Signs the transaction with the admin keypair (expected to migrate to multi-source 2-of-N in the mainnet path).
4. Sends update_oracle with price, confidence (DEFAULT_CONFIDENCE_BPS = 300 normal, LIVE_CONFIDENCE_BPS = 600 during live matches; flat across team and player markets in v1), and the live-match flag.

The two-process split between crank (pure calculation) and pusher (on-chain commits) is deliberate:

• Decouples failure domains. A data-feed outage pauses the crank; the pusher keeps serving cached data. An RPC outage pauses the pusher; the crank keeps producing data.
• Enables dry-run mode. Pusher supports --dry-run for testing transaction shape without committing.
• Makes mainnet migration incremental. The same crank will feed the multi-source oracle (2-of-N weighted median consensus, multi-oracle.mjs — built but not yet deployed).

3. Trigger Keeper + Funding Crank (port 3457)

Two related jobs bundled in one service:

Trigger keeper (15-second loop)

1. Loads all TriggerOrder PDAs via getProgramAccounts + memcmp filter.
2. Reads the latest MarketConfig.mark_price_ema for each relevant market.
3. Evaluates the trigger condition (see Order Types).
4. Submits execute_trigger_close or execute_trigger_open for matches.
5. Earns the keeper reward on successful execution.

Funding crank (8-hour loop)

Invokes apply_funding on every market. The on-chain program computes the plain mean of the premium samples accumulated since the last interval and updates the cumulative counters. Any trader interacting with a position after this call will have their pending funding settled. (Outlier filtering of premium samples is not implemented on-chain in v1; it is tracked as a potential future hardening.)

4. Liquidation Bot (port 3459)

The three-layer cascade keeper. Every 30 seconds:

1. Scans all UserPosition accounts, computing each position’s current margin ratio against the live mark_price_ema.
2. Classifies each position into a layer:
   • ratio ≤ 20% and > 13.33% → Layer 1 candidate
   • ratio ≤ 13.33% and insurance healthy → Layer 2 candidate
   • ratio ≤ 13.33% and insurance at cap → Layer 3 candidate
3. Executes the appropriate liquidation for each eligible position:
   • partial_liquidate — earns 5% reward.
   • backstop_liquidate — absorbs into insurance.
   • unwind_backstop — continues to unwind previously absorbed positions (10% per call), earns 3% reward.
   • auto_deleverage — picks a profitable opposing target (PnL × leverage ranking) and force-closes.

All four are permissionless. Competing keepers may run in parallel; the first transaction to land wins the reward, subsequent attempts fail cheaply with a clear error.

5. Monitor + Telegram (port 3460)

A watchdog that polls everything else:

• Health-checks the other services’ /health endpoints (crank, pusher, keeper, liquidator, indexer).
• Checks on-chain state — oracle staleness across all 68 markets, insurance fund balance, current_backstop_exposure, Program account existence.
• Sends Telegram alerts (via the NanoClaw bot integration) for: service down, oracle staleness > 2h, insurance balance < target × 0.5, backstop exposure > cap × 0.75.

Credentials for Telegram live in the service’s EnvironmentFile on Hetzner, never in the repo.

Deployment

All services deploy via systemd unit files in services/:

# on Hetzner
cp services/*.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now \
  sportsperp-crank \
  sportsperp-oracle \
  sportsperp-keeper \
  sportsperp-liquidator \
  sportsperp-monitor \
  sportsperp-indexer

Each unit file specifies:

• User=sportsperp (non-root service account).
• EnvironmentFile=/etc/default/sportsperp-<service> (data-feed credentials, RPC keys, Telegram tokens).
• Restart=on-failure with exponential backoff.
• LimitNOFILE=8192 (WS clients).

Logs flow to journald:

journalctl -u sportsperp-crank -f

See services/README.md and DEPLOY.md for the full deployment runbook.

RPC key split (Helius)

Solana RPC calls are distributed across four Helius keys, each rate-limited to 10 RPS (40 RPS aggregate):

Splitting traffic by workload prevents one runaway service from exhausting the overall rate budget. Key 3 is safe to expose in the frontend’s NEXT_PUBLIC_SOLANA_RPC_URL; keys 1, 2, and 4 stay server-side.

Degraded operation

Each service is designed to fail visibly rather than silently:

• Oracle crank: if the REST data feed fails, last-known values are preserved; oracle_is_live flag remains false; Telegram alert fires.
• Oracle pusher: if RPC fails, retries with kind-specific backoff; /health exposes lastCycleAt, lastSuccessfulPushAt (alias lastPushedAt), bootstrapStatus, backlogSize, lastFailureKind, circuitBreakerOpen, and quotaExhausted. Monitor alerts on quota exhaustion immediately, on degraded state across 2 polls, on backlog growth across 3 polls, on stale lastSuccessfulPushAt while backlog is non-empty, and on the pusher’s lastCycleAt lagging the crank’s lastCycleAt.
• Trigger keeper: if RPC or program fails, skipped this cycle; next cycle retries.
• Liquidator: same; plus, liquidations are permissionless so third parties can step in.
• Monitor: if the monitor itself fails, there’s no metaobserver — so the monitor is watched by a simple external uptime check.

No degraded state is ever silent. Traders can check oracle_timestamp on any market PDA to verify the feed isn’t stale, regardless of our services’ status.

Live↔REST ID bridge

The data partner’s REST and live-event APIs use independent ID spaces — e.g. a team might be REST id 1 but live id 21; a player might be REST 39461 but live 106232. Every market in roster.json is keyed by REST (canonical) id, so live events must be translated before they can be attributed. Note that live-side IDs are not guaranteed stable across matches for the same entity — the bridge resolves the mapping against each match’s lineup, not against a fixed cross-match table.

engine/id-bridge.ts is a fail-closed translator: any unmapped Live id drops the event and increments a counter rather than mis-attributing it. Operator overrides live in engine/id-bridge-overrides.json. Behaviour is gated by USE_LIVE_REST_BRIDGE_V1, USE_LIVE_REST_BRIDGE_V2, SHADOW_DROP_DIAG, and ID_BRIDGE_OVERRIDE_STRICT env vars.

Phase 2 (USE_LIVE_REST_BRIDGE_V2=true) was enabled on Hetzner on 2026-05-05. Phase 3 verification runs every 15 minutes Sat/Sun 10–22 UTC via sportsperp-bridge-verify.timer; the first full matchday verification window is Saturday 2026-05-09 (Liverpool–Chelsea KO 11:30 UTC).

Further reading

• services/README.md — deployment runbook.
• DEPLOY.md — full infrastructure setup guide.
• Oracle Design — detail on how the crank → pusher pipeline feeds the program.
• docs/plans/2026-05-04-live-rest-id-bridge-plan.md — Live↔REST bridge design and verification cadence.