Operations & Integration

Friday, 11 a.m.: the fittings manufacturer sends its payment run to the wallet. Fourteen suppliers are to receive 212,000 euro stablecoins in total. After 30 seconds the interface reports a timeout, and the treasurer starts the run again. On Monday the first supplier calls because it was paid twice. All fourteen were. The first run had long been executed; only its response was lost on the way.

A response gets lost

Nothing is broken at either end. The ERP system sent correctly, the wallet paid correctly. The fault lies in between: the order arrived, the confirmation did not. Repeating it while the other side has no memory of the first order creates a second one. With token payments, no bank recovers the money. Each of the fourteen suppliers has to return the duplicate amount voluntarily, and until then the sum is missing from liquidity.

For a CFO, the question is therefore not whether such failures happen. Networks lose responses, services restart, queues deliver twice. What matters is whether a retry can have a second effect. This article describes operations between the ERP system and the ledger in both directions: collecting events and sending payments. Which fields a statement carries is described in Data Standard & Account Statement. Who approves a payment, and above which amount a four-eyes principle applies, is covered in Custody & Safekeeping. This article is about the approved payment reaching the signature exactly once and unchanged.

When an event counts

The interface of an Ethereum node knows three states for every query: the latest block, one marked as safe, and a finalized one. The latest block is the most current, but it can still be replaced. If the network later agrees on a different block, this is called a reorganization. An event read from the replaced block then no longer exists. Anyone receiving events through a filter gets it delivered again, with the flag removed. Anyone who simply queries again no longer finds it. An import that neither evaluates the flag nor re-checks blocks already read keeps a payment that never happened.

A finalized block, by contrast, can only be reverted if an attacker loses at least a third of all staked ether, according to the Ethereum documentation. This leads to a simple rule: read everything, but post only what is finalized. Anything before that appears in the statement as provisional. Other networks have other rules, and some implementations count a fixed number of confirmations instead of using the marker. Which threshold applies is a decision like the day cut-off. It belongs in the procedural documentation, as described in Treasury, Risk & Reporting.

Collecting or being notified

Events reach the company in two ways. The integration regularly asks a node for transfers of a token to its own address. Or a provider, such as the custodian or a payment service provider, reports them by webhook. The notification is faster. But it has a weakness: if the company's system was unreachable at the wrong moment, it depends on the provider whether and how often it resends. The query is slower, but it can be repeated for any block range as often as needed.

Combining both has proven itself. The webhook gives the trigger, the query at the block provides the evidence. After every notification, and also at a fixed interval, the integration reads the block range itself. A lost notification then shows up at the next interval at the latest. How a webhook is protected against forged and replayed messages is explained in the glossary.

One key per event

Both ways deliver duplicates. The provider reports the same event twice, two query windows overlap, a range is read again after a restart. Every event therefore needs a key under which it is stored exactly once. The ledger supplies it: Chain ID, transaction hash and the position of the event within the transaction. The position is needed because one transaction can contain several transfers, for example a batch payment, as described in Reconciliation & Payment Flows. If an event arrives a second time with a known key, the import table discards it.

The ERP side is harder. If its interface accepts an idempotency key when a posting is created, the integration passes on the event's key. If it does not, the integration has to check before every posting whether it already exists, for example via the transaction hash in the posting text. Checking and posting are then two steps, and two runs at the same time could both find nothing. So only one process posts per wallet. One implementation is HashBuch by Setrion GmbH, which also operates this knowledge hub. It stores each event exactly once under transaction hash, position and wallet, and for that it re-reads the most recent blocks in every run.

What remains after the last attempt

Many errors pass on their own: the node is overloaded, the database briefly unreachable. A retry helps then, but not immediately. The intervals should grow, say one minute, then five, then thirty, so that an overloaded service can recover. Other errors do not pass. The ERP system rejects a posting because the period is locked or the counterparty is missing. After a fixed number of attempts, the event therefore goes into a separate queue with its reason, and someone clears it every day.

One rule matters more than any interval. The marker for how far blocks have been read must never move past an event that has not been stored. Otherwise the event is lost silently, with no error message and no entry in any queue. The queue is also not the suspense account. The suspense account holds posted amounts whose assignment is open. The queue holds events that have not been posted at all. At month end, both belong on the list of differences.

When the other side changes

An integration depends on three counterparts that change without asking: the node provider, the provider's interface and the token contract. A provider announces a new version of its interface and switches off the old one on a set date. If a field is renamed there, it goes missing from the import without a trace. The integration therefore states explicitly which version it expects and tests a new one before the deadline.

With the token contract, the change is less visible. Some contracts are built so that their program logic can be replaced while the address stays the same. The ERC-1967 standard for such contracts provides that a replacement is announced with the event Upgraded. The integration watches for this event on every token it posts and alerts treasury. A test network helps only to a limited extent. It runs a test token issued by anyone, not the issuer's contract.

The nonce as a payment run number

On the outgoing side, Ethereum already has a lock built in. Every transaction from an account carries a nonce, a counter that rises by one with each transaction. Only one transaction with a given nonce can be executed per account. The interface explicitly allows a pending transaction to be overwritten with the same nonce. If a payment is sent a second time with the same nonce, no second payment results. Only a new nonce pays twice, and that is exactly what a wallet does when it assigns the nonce itself at sending.

This yields a building block: approval in the ERP system fixes the nonce for each payment and stores it with the payment. The nonce thus becomes the idempotency key of the outgoing side, and the network enforces it. The limit lies in the counter. It runs in sequence, so a payment with nonce 42 waits until 41 has been executed. If a payment is stopped after approval, it holds up all later ones. That lasts until a replacement transaction with the same nonce takes its place, for example a transfer of zero to the company's own address.

The ERC-4337 standard relaxes this limit. It splits the nonce into a key and a sequence, and each key counts on its own. A payment run can thus get its own key, and a stopped run does not hold up the next one. As with the paymaster, the wallet has to be run as a smart contract, as described in Posting, Valuation & Tax. On its second sending, Friday's run would have carried the same fourteen nonces, and the network would not have executed any of the fourteen payments a second time.