Off-Chain Services

A set of services runs alongside the on-chain program on a hardened backend host — always-on Node services plus an obv-engine Python sidecar. All are deployed with automatic restart, health endpoints, and centralized logging. Every operation they perform is permissionless — the services are a guarantee that work happens on schedule, not a centralized operator.

Service topology

Each service exposes a /health endpoint returning {status, uptime, lastCycle, errors} for the monitor service to poll. The obv-engine is consumed by the crank’s live processor over an internal HTTP contract when the real-time OBV bridge is enabled; the heuristic impact-estimation path is retained as a fallback.

1. Oracle Crank + Candle API + WS

The central service. Three responsibilities in one process:

Crank (core loop)

Every 5 minutes:

1. Fetch season stats for all 20 teams and eligible players from the post-match REST feed.
2. Fetch recent match results (for form + PPG calculation).
3. Subscribe to the live event feed (GraphQL subscriptions) for any in-progress matches.
4. Compute composite indices via the off-chain index calculator — z-scored, scaled to 100–900.
5. Persist each tick to SQLite (candles.db) for historical candle reconstruction.
6. Broadcast the tick on WebSocket to connected frontends.
7. Signal the oracle pusher that a new value is available.

Candle API (Express REST)

Exposes /candles/{market_key}?timeframe=1m|5m|15m|1H|4H|1D&limit=N returning OHLC bars. Frontend charts consume this via HTTPS proxy routes.

WebSocket server

Streams live ticks, live-match events (goals, key passes, defensive actions), and aggregate state changes. Consumed by the trading UI for real-time chart updates.

Also hot-reloads roster.json via fs.watch — any roster change takes effect on the next cycle without a restart.

2. Oracle Pusher

A dedicated bridge between the crank’s SQLite output and the on-chain program:

1. Reads new index values from the candle store.
2. Decides whether to push, based on a minimum price-change threshold (tighter during live matches) or an internal staleness bound well inside the on-chain 2-hour limit.
3. Signs the transaction with the hot oracle_authority — the deployed pusher runs 2-of-N multi-source consensus on devnet (fully independent sources are the remaining mainnet step).
4. Sends update_oracle with price, a confidence interval (wider during live matches), and the live-match flag.

The two-process split between crank (pure calculation) and pusher (on-chain commits) is deliberate:

• Decouples failure domains. A data-feed outage pauses the crank; the pusher keeps serving cached data. An RPC outage pauses the pusher; the crank keeps producing data.
• Enables dry-run mode. Pusher supports --dry-run for testing transaction shape without committing.
• Makes mainnet migration incremental. The same crank already feeds the multi-source oracle (2-of-N weighted-median consensus), which is deployed and live on devnet, with the mainnet cutover still ahead.

3. Trigger Keeper + Funding Crank

Two related jobs bundled in one service:

Trigger keeper (15-second loop)

1. Loads all TriggerOrder PDAs in one program-account read, then filters by the active roster.
2. Reads the latest MarketConfig.mark_price_ema for each relevant market.
3. Evaluates the trigger condition (see Order Types).
4. Submits execute_trigger_close or execute_trigger_open for matches.
5. Earns the keeper reward on successful execution.

Funding crank (8-hour loop)

Invokes apply_funding on eligible markets. The on-chain program averages the premium samples accumulated since the last interval, clamps the result to the configured cap, updates the cumulative counters, and pays the flat funding-crank reward when pool headroom allows. Any trader interacting with a position after this call will have their pending funding settled.

4. Liquidation Bot

The three-layer cascade keeper. Every 30 seconds:

1. Scans all UserPosition accounts, computing each position’s current margin ratio against the live mark_price_ema.
2. Classifies each position into a layer:
   • ratio ≤ 20% and > 13.33% → Layer 1 candidate
   • ratio ≤ 13.33% and insurance healthy → Layer 2 candidate
   • ratio ≤ 13.33% and insurance at cap → Layer 3 candidate
3. Executes the appropriate liquidation for each eligible position:
   • partial_liquidate — earns the Layer 1 bps reward.
   • backstop_liquidate — absorbs into insurance and earns the Layer 2 bps reward.
   • unwind_backstop — continues to unwind previously absorbed positions (10% per call) and earns the flat unwind reward.
   • auto_deleverage — picks a profitable opposing target (PnL × leverage ranking), force-closes, and earns the flat ADL reward.

All four are permissionless. Competing keepers may run in parallel; the first transaction to land wins the reward, subsequent attempts fail cheaply with a clear error.

5. Monitor + Alerting

A watchdog that polls everything else:

• Health-checks the other services’ /health endpoints (crank, pusher, keeper, liquidator, indexer, snapshot-pusher).
• Checks on-chain state — oracle staleness across all 68 markets, insurance fund balance, current_backstop_exposure, program account existence.
• Sends operator alerts for: service down, oracle staleness past the on-chain limit, insurance balance below its target buffer, backstop exposure approaching its cap.

Alerting credentials are held server-side in a restricted secrets store, never in the repo.

Deployment

All services are deployed as managed, auto-restarting system units under a dedicated non-root service account, with credentials supplied from a restricted secrets store and logs shipped to centralized logging. Restart-on-failure with exponential backoff is configured per service. The full deployment runbook is maintained privately by the operations team.

RPC

Solana RPC runs on a single managed provider plan with a shared rate budget; demand is kept well under that budget by conservative per-service polling cadences. Only a read-only, domain-locked frontend key is exposed client-side (NEXT_PUBLIC_SOLANA_RPC_URL); the backend key stays server-side.

Degraded operation

Each service is designed to fail visibly rather than silently:

• Oracle crank: if the REST data feed fails, last-known values are preserved; the oracle_is_live flag remains false; an operator alert fires.
• Oracle pusher: if RPC fails, retries with kind-specific backoff; /health exposes cycle, push-success, backlog, and circuit-breaker state. The monitor alerts on quota exhaustion, sustained degraded state, backlog growth, stale push-success while backlog is non-empty, and the pusher lagging the crank.
• Trigger keeper: if RPC or program fails, skipped this cycle; next cycle retries.
• Liquidator: same; plus, liquidations are permissionless so third parties can step in.
• Monitor: the monitor’s own liveness is covered by independent external uptime checks and on-call escalation.

No degraded state is ever silent. Traders can check oracle_timestamp on any market PDA to verify the feed isn’t stale, regardless of our services’ status.

Live↔REST ID bridge

The data partner’s REST and live-event APIs use independent ID spaces — the same entity has a different id on each feed. Every market in roster.json is keyed by the canonical (REST) id, so live events must be translated before they can be attributed. Live-side IDs are not guaranteed stable across matches for the same entity — the bridge resolves the mapping against each match’s lineup, not against a fixed cross-match table.

The bridge is a fail-closed translator: any unmapped live id drops the event rather than mis-attributing it, and the miss is counted for observability. A small operator-curated override table covers known edge cases, and a periodic verification job validates mappings during matchdays.

Further reading

• Oracle Design — detail on how the crank → pusher pipeline feeds the program.