Skip to main content

Dead Man Switch

The Core-owned Dead Man Switch (DMS) is enabled in the live private-beta deployment, with a bounded BSL Spot expiry canary passed on September 12. Its contract is to cancel explicitly protected resting orders when their owner's signed heartbeats stop. Orders are not enrolled automatically by enabling DMS.

The September 12 incident exposed divergent order IDs in API/WS follower replay: the Core canceled the protected order, but readers retained a ghost order. The corrected readers use sealed compact-frame identity and verify the actual V4 state root. A live Direct TCP canary subsequently confirmed the same order ID in the execution report and account API, followed by an exact Canceled report, absence from openOrders, and an empty tested book level after heartbeat expiry. This bounded test used one 5-USDC PostOnly Spot bid; it does not establish a cancellation SLA or coverage of every market and client. Signed HTTP, reconnect, blocked SQL history and process-kill recovery have separate local test evidence. See Status.

Scope and supported clients

Protection belongs to an account, authenticated principal, logical session, epoch and writer term. A transport reconnect retains the strategy's logical session. Other strategies and unmarked orders in the account remain outside that scope.

ClientSigned control endpointHow an order opts in
Delegated HTTP agentPOST /api/v1/trading/dmsInclude the signed dms binding on supported delegated order requests
BSL SessionKey v1POST /api/v1/bsl/dmsSet the DMS flag and epoch before signing the compact order frame

Legacy wallet-signed BSL frames cannot opt in. FIX/FIXP session cancel-on-disconnect is a separate mechanism; its presence does not enroll orders in this HTTP/BSL DMS. Conditional and algorithmic parents without supported durable child ownership are rejected.

Arm, heartbeat, reconcile

  1. Send a signed status control for the intended owner to discover its current epoch and writerTerm.
  2. Send arm with a newer epoch and the current writer term. timeoutMs must be between 100 and 60000 milliseconds. Wait for a successful response before submitting protected orders.
  3. Mark the orders and send signed heartbeat controls well within the timeout, with a strictly increasing heartbeatSequence. For example, a 3000 ms timeout with heartbeats every 500 ms leaves scheduling margin; choose values suitable for the client's measured network and scheduling.
  4. Set a fresh expiresAtMs on every control, at most 30 seconds ahead. This is the signed request's validity window, separate from the DMS timeout. Repeated arm requests and replayed heartbeats do not extend the lease.
  5. On timeout or heartbeat failure, reconcile status and order reports. A late heartbeat cannot revive an expired epoch. Wait for its completion before arming the next epoch.

TCP keepalives, an open socket, order traffic and SessionKey runner lifetime do not replace signed DMS heartbeats. See Session keys.

Request fields

Both controls require operation (arm, heartbeat, status, disarm), heartbeatSequence, timeoutMs, expiresAtMs and signature.

EndpointAdditional required fields
Delegated HTTPagentId, account, dms: { logicalSession, epoch, writerTerm }; logicalSession is 16 bytes encoded as hex
BSLsessionKeyId, account, gatewayId, sessionId, epoch, writerTerm

Use the same authenticated principal and account as order entry. BSL controls use the registered Ed25519 SessionKey and chain binding. Set the epoch before computing the full frame signature: the existing 192-byte frame binds flag bit 4 and the positive u32 epoch in bytes 20–23.

The registered BSL policy must include cancel in allowedActions, even for status, arm, and heartbeat. It must also authorize the gateway. Otherwise the control returns DMS_AUTH_REJECTED. DMS controls authorize a cancellation service; spot_place alone is insufficient.

Coordinate action nonces across every process using the same account. A nonceHoles read is a snapshot, not a reservation: another process can consume the hole before submission. Reject 205 (NonceAboveWindow) means that order never entered the book. Do not treat completion of its empty epoch as a passed canary. Require the exact order's successful execution report and DMS liveOrders > 0 before stopping heartbeats; a COMMAND ACK is insufficient.

The published TypeScript SDK @sentico-labs/[email protected] provides buildAgentDmsMessage, dmsSigningSuffix, signBslDmsControl, client.trading.controlDms and client.trading.controlBslDms. Rust provides the corresponding control types and prepare_bsl_dms_frame_for_session_signing in its repository distribution. Upgrade older npm installations to 0.3.0 to use these helpers. See SDKs.

What completion means

The deployed status includes everTrackedOrder: true means a canonical protected resting order was observed, false means a fresh epoch has not enrolled one, and null means that history is unavailable after recovery. Older servers omit it. Do not use this field or completed as a substitute for an exact order cancellation report. A rejected order can leave a correctly completed empty epoch. An unknown cancel target supplies no terminal-order proof and must remain fenced while it remains in protected membership.

The live empty-epoch test confirmed everTrackedOrder: false and lastCommandSequence: 0; the protected-order test confirmed true before expiry and after completion.

One canary immediately after the Core restart failed its 10-second wait for the New execution report, despite canonical placement and subsequent expiry cancellation. The later steady-state canary passed all report/API/book checks. The immediate-post-restart report-delivery observation remains an open acceptance item; do not equate server readiness with a verified client report channel. Keep protection active and reconcile when a report is missing.

PhaseMeaning
disarmedNo active protection for this owner
armedValid heartbeats can renew the lease; marked orders can enter
cancellingNew protected exposure is rejected; accepted commands and exact-order cancellations are being reconciled
completedCanonical protected-order membership and pending commands are empty

Inspect liveOrders, pendingCommands, epoch, writerTerm and, when present, completedAtResultSequence. A COMMAND ACK, an admitted cancellation count, or a disconnect notice is not terminal cancellation evidence. Fills can occur before cancellation. DMS does not reverse fills or liquidate positions.

completed covers only the requested owner and epoch. An epoch without enrolled orders can also complete. It does not mean every order on the account was canceled. Reconcile the exact protected order IDs with canonical terminal reports and the orderbook before treating a connectivity-loss test as accepted.

disarm removes protection from existing orders; it does not cancel them. It is allowed while armed with no pending accepted commands. Cancel and reconcile first when the intention is to stop quoting without resting orders.

Errors and recovery

ErrorResponse
DMS_CONTROL_EXPIREDRefresh the signed request validity window
DMS_INVALID_SCOPE / DMS_AUTH_REJECTEDCorrect the scope, account, key or signature
DMS_DISABLEDActivation or recovery is incomplete; protection cannot be assumed
DMS_CONTROL_BUSYControl capacity is full; back off and reconcile before retrying
DMS_CONTROL_REJECTEDInspect the reason and current epoch/term/phase before retrying

Writer loss fences admission. On a supported Core restart, canonical recovery reconstructs ownership and expires previous volatile leases. During a complete Core outage an in-process switch cannot execute cancellations. Reconcile after recovery and use the returned writer term for the next arm.

The current DMS release supports one shard and restart recovery. Multi-shard and hot-standby/follower promotion are explicitly unsupported. There is no fixed cancellation-latency SLA or blanket zero-overhead claim. Heartbeat handling avoids matching-engine locks and per-heartbeat SQL writes; expiry cancels still consume normal execution capacity.