Transaction State Machines for Multi-Chain Custody: What Actually Breaks
Every wallet backend has a transaction status column. Very few have a transaction state machine. The difference shows up the first time a chain does something the happy path did not anticipate, which on real chains is roughly every week.
A status column answers "what is it now". A state machine answers three harder questions: which transitions are legal, what triggers them, and what happens when the world moves backwards. In custody, where every transition can move client money, those questions are the system.
The states you actually need
The shape that has survived contact with BTC, EVM chains and Solana looks roughly like this:
CREATED → POLICY_APPROVED → SIGNED → BROADCAST → PENDING →
CONFIRMED(n) → FINALIZED
with exits to FAILED, REPLACED and EXPIRED,
and one deliberately uncomfortable arrow: CONFIRMED → PENDING, for reorgs.
A few of these deserve defending:
- POLICY_APPROVED is a separate state, before SIGNED. Policy evaluation (limits, allowlists, quorum) happens before a key is ever touched, and the approval is recorded as its own transition with its own audit entry. If policy and signing are one step, you cannot prove to an auditor which rule approved a transfer, and you cannot fail closed cleanly when the policy engine is down.
- BROADCAST and PENDING are different states. Broadcast means "we handed the transaction to a node". Pending means "the network acknowledges it exists". Between the two, transactions get dropped silently: the node was behind, the mempool evicted it for low fees, the RPC provider accepted it and lost it. If your system treats broadcast as success, these transactions simply vanish from your books while the signed bytes still exist and can confirm later. That combination is how you get a withdrawal that pays out twice.
- CONFIRMED is parameterized, FINALIZED is policy. Confirmations accumulate differently per chain: depth on Bitcoin, finalized checkpoints on Ethereum, commitment levels on Solana. FINALIZED is where your business logic says "this can never come back". That threshold is a per-chain policy decision that belongs in reviewed configuration, not in an engineer's head.
- REPLACED is a real state, not an error. Fee-bumping (RBF on Bitcoin, same-nonce replacement on Ethereum) produces a new transaction that competes with the old one. Both exist until one confirms. Model the pair explicitly, or your monitoring will report a phantom failure every time remediation does its job.
Backwards transitions, or: reorgs are not exceptional
The transition everyone forgets is the one that goes backwards. A block containing your confirmed transaction can stop being part of the canonical chain. On fast chains with probabilistic finality this is routine; on Bitcoin it is rare but priced in; on Ethereum it is bounded by the finalized checkpoint. When it happens, the transaction must walk back to PENDING, any ledger effect must be reversed with a compensating entry (append-only ledgers make this provable, mutable balance columns make it archaeology), and someone should be paged.
Idempotency is the other half of the design
Most real incidents are not exotic chain behaviour. They are retries. A timeout on broadcast, an operator clicking again, a queue redelivering a message. Every one of those must be safe, which means:
- Every transition is idempotent. Applying "mark as broadcast" twice results in one state change and one audit entry, not two.
- Every externally-triggered action carries an idempotency key. A withdrawal request that arrives twice, through any path, produces one transaction. The dedup happens at the boundary, on a key the client controls, stored with the transaction itself.
- Signing is exactly-once by construction. The signed payload is persisted before broadcast. A retry re-broadcasts the same bytes; it never re-signs. Re-signing on retry is how you end up with two valid competing transactions and, on UTXO chains, a very educational afternoon.
Where each chain bends the model
- Ethereum: the nonce queue. Transactions from one address confirm in nonce order. One underpriced transaction blocks everything behind it, so "stuck" is contagious. The state machine needs a per-account view, not just per-transaction: remediation of transaction N is meaningless if N-1 is the one that is stuck.
- Solana: transactions expire. A transaction references a recent blockhash and dies after roughly 150 slots if unconfirmed. EXPIRED is a first-class terminal state, and the retry path builds a new transaction with a fresh blockhash. For offline or slow signing flows, durable nonces exist precisely because signed transactions cannot wait around.
- Bitcoin: nothing is yours until it confirms. Mempools are local opinion, eviction is silent, and replacement is a feature. The PENDING state carries an age timer, and remediation (fee-bump or child-pays-for-parent) is a policy-driven transition, not an ops improvisation.
The audit trail is not a log file
In a regulated environment, every transition needs: previous state, new state, trigger (who or what), timestamp, and the evidence (block hash, node response, policy decision id). Stored as data, queryable, immutable. When a client asks why a withdrawal took forty minutes, or an auditor asks who approved it, the answer is a query, not a grep through application logs that rotated last month.
What actually breaks: a short list from production
- Treating broadcast as done. The transaction vanishes from the mempool and from your books at the same time.
- No REPLACED state. Fee-bumps look like failures; monitoring cries wolf until people stop listening.
- Re-signing on retry instead of re-broadcasting persisted bytes.
- Per-transaction thinking on Ethereum, ignoring the nonce queue.
- No EXPIRED state on Solana; transactions rot in PENDING forever.
- Crediting on CONFIRMED with one global threshold for every chain.
- No backwards transition for reorgs, so reconciliation finds the hole weeks later.
None of these are hard to fix on a whiteboard. All of them are expensive to fix after the money moved.