--- title: "Primary authority and failover" description: "Understand direct peer health, controller confirmation, recoverable candidate selection, and fenced transfer to a new primary." section: "Architecture" source_url: "https://sqlstyle.com/docs/primary-and-failover" --- SQLStyle separates evidence of failure from authority to replace a writer. A capture owner can determine that its direct connection to the holder has stopped responding, but that observation alone cannot distinguish a failed primary from a network partition. Only the hosted controller can end the current grant and issue the next fencing token. There are no controller roles. [Node capabilities](/docs/node-roles) configure voting, storage work, promotion, capture contention, and capture-loss cleanup independently. ## A fencing token identifies the current writer Every primary grant contains a token newer than the token issued to the previous primary. The runtime admits writes only while its local authority state matches the current grant and its deadline remains valid. A recent database is evidence of recoverable state, not permission to write. If a former primary reconnects after replacement, its token cannot reopen writer admission, advance current replication sessions, or publish recovery progress for the new lineage. ## Direct peer streams detect loss first Each holder-to-capture-owner connection has a dedicated bidirectional health stream bound to the database, both node identities, and the current primary token. Both ends ping once per second. Five seconds without a successful round trip marks that peer unhealthy in the local view. A capture owner then treats holder health as unknown and may report a concern for the exact token and topology it observed. It does not declare the primary dead or start an election. If direct health recovers before the controller confirms the failure, the concern is cleared. A concern is counted only when the reporting connection has `vote: true` and its local token-bound channel has previously completed a round trip to that holder. A nonvoter can report an advisory route failure, which may prompt an immediate holder heartbeat but cannot create, refresh, join, or accelerate a fast-fence incident. ## The controller challenges the primary The controller maintains its own heartbeat for the current grant. The first concern for a token creates a short incident, records the holder's current heartbeat, asks the primary to heartbeat immediately, and tells the reporter to retry after 250 milliseconds. For the current topology: - `C` is the current holder plus every other connected connection with `vote: true`; - `R` is the number of matching reports from those connected voters; and - `H` is `1` when the challenged primary heartbeat has not advanced, otherwise `0`. The controller ends the grant on the fast path only when: ```text 2(R + H) > C ``` The counts are derived from current connections, not retained in the incident. If topology changes, the incident is discarded. Zero-capability ASP sessions do not appear in topology and cannot affect the calculation. If the condition is not met, the controller returns the remaining health window and retains the grant; the ordinary holder-heartbeat timeout remains the fallback. Once the controller has confirmed the holder unhealthy for a token, a later peer ping cannot revive that grant. ## The primary fences itself during suspected isolation The primary evaluates the same direct-health topology from the other side. If the missing vote-capable peers together with the primary constitute a strict majority of current voters, it asks the controller to confirm its own heartbeat. If the runtime cannot queue and confirm that challenge within 300 milliseconds, it closes local writer admission. The controller grant remains allocated during this local fence, so a later confirmation for the same token and topology can restore writes without issuing another token. A topology or authority change starts a new decision. This rule prevents a primary that may be isolated from a majority of the deployment from continuing to accept writes solely because its previous lease deadline has not yet expired. ## Candidate selection starts from a required transaction An unexpected control-channel close first reserves the existing token for a short reconnect window. The same node may resume that grant if it reconnects in time. A holder timeout already confirmed by the health path ends the grant immediately instead of waiting for that reservation. When replacement is required, the controller publishes the minimum transaction the next primary must recover. Connected `promotable` runtimes evaluate their local image and object-storage state and report a frontier only for the matching token. A runtime offers after the decision delay only when its image is usable and its frontier covers the requirement. If the selected candidate still needs to materialize the required state, the controller places it in recovery. The candidate validates and restores a continuous frontier, reports the recovered transaction, and remains read-only until the controller durably issues the next token. Connections without `promotable` never offer, and duplicate or racing offers cannot bypass the controller's current durable state. ## Process takeover does not require node failover A process crash on the primary host may leave the node and its controller grant intact. Another local process can acquire capture ownership and preserve that grant only while its identity, token, and absolute deadline still validate. This transition recreates process-local services; it does not select a new primary. See [multi-process operation](/docs/multi-process). ## Planned handoff changes holder and token atomically For a requested handoff, the controller reserves one connected `promotable` target and asks the current primary to drain and release. It then updates the holder and token in one durable transition rather than publishing an intermediate state with no primary. The replacement admits writes only after it receives the new grant. The former primary's drained state and earlier token remain fenced. ## Authority also fences backup history A former primary can finish uploading an object after failover, but object arrival does not advance the recovery state readers may trust. The controller accepts storage progress only for the current token or through an explicit recovery transition. [Backup and recovery history](/docs/backup-and-recovery-history) describes that read fence and the distinction between uploaded and authoritative history.