Dead Man Switch
The Core-owned Dead Man Switch (DMS) is enabled in the live private-beta deployment, with a bounded BSL Spot expiry canary passed on September 12. Its contract is to cancel explicitly protected resting orders when their owner's signed heartbeats stop. Orders are not enrolled automatically by enabling DMS.
The September 12 incident exposed divergent order IDs in API/WS follower replay:
the Core canceled the protected order, but readers retained a ghost order.
The corrected readers use sealed compact-frame identity and verify the actual
V4 state root. A live Direct TCP canary subsequently confirmed the same order ID
in the execution report and account API, followed by an exact Canceled report,
absence from openOrders, and an empty tested book level after heartbeat expiry.
This bounded test used one 5-USDC PostOnly Spot bid; it does not establish a
cancellation SLA or coverage of every market and client. Signed HTTP, reconnect,
blocked SQL history and process-kill recovery have separate local test evidence.
See Status.
Scope and supported clients
Protection belongs to an account, authenticated principal, logical session, epoch and writer term. A transport reconnect retains the strategy's logical session. Other strategies and unmarked orders in the account remain outside that scope.
| Client | Signed control endpoint | How an order opts in |
|---|---|---|
| Delegated HTTP agent | POST /api/v1/trading/dms | Include the signed dms binding on supported delegated order requests |
| BSL SessionKey v1 | POST /api/v1/bsl/dms | Set the DMS flag and epoch before signing the compact order frame |
Legacy wallet-signed BSL frames cannot opt in. FIX/FIXP session cancel-on-disconnect is a separate mechanism; its presence does not enroll orders in this HTTP/BSL DMS. Conditional and algorithmic parents without supported durable child ownership are rejected.
Arm, heartbeat, reconcile
- Send a signed
statuscontrol for the intended owner to discover its current epoch andwriterTerm. - Send
armwith a newer epoch and the current writer term.timeoutMsmust be between 100 and 60000 milliseconds. Wait for a successful response before submitting protected orders. - Mark the orders and send signed
heartbeatcontrols well within the timeout, with a strictly increasingheartbeatSequence. For example, a 3000 ms timeout with heartbeats every 500 ms leaves scheduling margin; choose values suitable for the client's measured network and scheduling. - Set a fresh
expiresAtMson every control, at most 30 seconds ahead. This is the signed request's validity window, separate from the DMS timeout. Repeated arm requests and replayed heartbeats do not extend the lease. - On timeout or heartbeat failure, reconcile status and order reports. A late heartbeat cannot revive an expired epoch. Wait for its completion before arming the next epoch.
TCP keepalives, an open socket, order traffic and SessionKey runner lifetime do not replace signed DMS heartbeats. See Session keys.
Request fields
Both controls require operation (arm, heartbeat, status, disarm),
heartbeatSequence, timeoutMs, expiresAtMs and signature.
| Endpoint | Additional required fields |
|---|---|
| Delegated HTTP | agentId, account, dms: { logicalSession, epoch, writerTerm }; logicalSession is 16 bytes encoded as hex |
| BSL | sessionKeyId, account, gatewayId, sessionId, epoch, writerTerm |
Use the same authenticated principal and account as order entry. BSL controls use the registered Ed25519 SessionKey and chain binding. Set the epoch before computing the full frame signature: the existing 192-byte frame binds flag bit 4 and the positive u32 epoch in bytes 20–23.
The registered BSL policy must include cancel in allowedActions, even
for status, arm, and heartbeat. It must also authorize the gateway.
Otherwise the control returns DMS_AUTH_REJECTED. DMS controls authorize a
cancellation service; spot_place alone is insufficient.
Coordinate action nonces across every process using the same account. A
nonceHoles read is a snapshot, not a reservation: another process can consume
the hole before submission. Reject 205 (NonceAboveWindow) means that order
never entered the book. Do not treat completion of its empty epoch as a passed
canary. Require the exact order's successful execution report and DMS
liveOrders > 0 before stopping heartbeats; a COMMAND ACK is insufficient.
The published TypeScript SDK @sentico-labs/[email protected] provides buildAgentDmsMessage,
dmsSigningSuffix, signBslDmsControl, client.trading.controlDms and
client.trading.controlBslDms. Rust provides the corresponding control types and
prepare_bsl_dms_frame_for_session_signing in its repository distribution.
Upgrade older npm installations to 0.3.0 to use these helpers.
See SDKs.
What completion means
The deployed status includes everTrackedOrder: true means a canonical protected
resting order was observed, false means a fresh epoch has not enrolled one,
and null means that history is unavailable after recovery. Older servers
omit it. Do not use this field or completed as a substitute for an exact
order cancellation report. A rejected order can leave a correctly completed
empty epoch. An unknown cancel target supplies no terminal-order proof and
must remain fenced while it remains in protected membership.
The live empty-epoch test confirmed everTrackedOrder: false and
lastCommandSequence: 0; the protected-order test confirmed true before
expiry and after completion.
One canary immediately after the Core restart failed its 10-second wait for the New execution report, despite canonical placement and subsequent expiry cancellation. The later steady-state canary passed all report/API/book checks. The immediate-post-restart report-delivery observation remains an open acceptance item; do not equate server readiness with a verified client report channel. Keep protection active and reconcile when a report is missing.
| Phase | Meaning |
|---|---|
disarmed | No active protection for this owner |
armed | Valid heartbeats can renew the lease; marked orders can enter |
cancelling | New protected exposure is rejected; accepted commands and exact-order cancellations are being reconciled |
completed | Canonical protected-order membership and pending commands are empty |
Inspect liveOrders, pendingCommands, epoch, writerTerm and, when
present, completedAtResultSequence. A COMMAND ACK, an admitted cancellation
count, or a disconnect notice is not terminal cancellation evidence. Fills can
occur before cancellation. DMS does not reverse fills or liquidate positions.
completed covers only the requested owner and epoch. An epoch without enrolled
orders can also complete. It does not mean every order on the account was canceled.
Reconcile the exact protected order IDs with canonical terminal reports and the
orderbook before treating a connectivity-loss test as accepted.
disarm removes protection from existing orders; it does not cancel them.
It is allowed while armed with no pending accepted commands. Cancel and
reconcile first when the intention is to stop quoting without resting orders.
Errors and recovery
| Error | Response |
|---|---|
DMS_CONTROL_EXPIRED | Refresh the signed request validity window |
DMS_INVALID_SCOPE / DMS_AUTH_REJECTED | Correct the scope, account, key or signature |
DMS_DISABLED | Activation or recovery is incomplete; protection cannot be assumed |
DMS_CONTROL_BUSY | Control capacity is full; back off and reconcile before retrying |
DMS_CONTROL_REJECTED | Inspect the reason and current epoch/term/phase before retrying |
Writer loss fences admission. On a supported Core restart, canonical recovery reconstructs ownership and expires previous volatile leases. During a complete Core outage an in-process switch cannot execute cancellations. Reconcile after recovery and use the returned writer term for the next arm.
The current DMS release supports one shard and restart recovery. Multi-shard and hot-standby/follower promotion are explicitly unsupported. There is no fixed cancellation-latency SLA or blanket zero-overhead claim. Heartbeat handling avoids matching-engine locks and per-heartbeat SQL writes; expiry cancels still consume normal execution capacity.