Three backend jobs whose first version is wrong and passes review

*Building Supero, part 1 of 4.*

---

---

Ask an engineer to estimate a B2B application and you get a number. The number is usually not wrong. It is a number for the features — the screens, the entities, the reports someone drew on a whiteboard.

Then there is the rest. The tenant column on every table, the audit row nobody reads until they need it, the invoice that has to survive a plan change mid-month. Most of it is boring and known. It does not appear on the estimate because everyone in the room assumes it is already handled — by the framework, by the platform, or by whoever did it last time.

Three pieces of it are not boring. They are the ones where the first version is wrong in a way careful review does not catch, because the defect is a race or a lost obligation, and neither shows up under a load test that never kills a process mid-transaction.

Each section below ends with something you can run against code you already own.

*Disclosure: I work on Supero, which generates multi-tenant application backends. The three checks below need no account and are about your code, not ours. The part where I talk about our own system is at the end, flagged as such.*

The saga pattern is taught with a clean picture. Do A. Do B. B fails. Undo A. The undo is drawn as a primitive.

That drawing hides two different operations with different preconditions, and which one applies depends on a state you do not own.

An authorization becomes a capture. A capture becomes a settlement. On some processors a capture can be voided before it settles; on Stripe it cannot, because cancel applies to an uncaptured intent, and a same-day refund is converted to a reversal behind the scenes instead. Either way the money does not move — but do not tell your support team nothing appears on the customer's statement, because the authorization hold usually stays visible in their banking app until the issuer releases it, which can take days unless you send an explicit reversal. After settlement your only instrument is a refund, and a refund is not an undo. It is a new money movement in the opposite direction with its own lifecycle, its own asynchronous confirmation, and its own fee treatment. The processor may keep your original fee.

The boundary between voidable and refundable moves on the processor's clock. Usually a nightly batch. Not yours.

So the naive compensator is this, and it is in a lot of production code:

try:
    charge = psp.capture(auth_id, amount)
    reserve_inventory(order)      # fails
except Exception:
    psp.void(charge.id)           # what state is this actually in?

Four things go wrong, and only the first is obvious.

**The void is rejected because the capture already settled.** Your client library raises, your retry wrapper does what retry wrappers do, and it tries again against an object that will never become voidable. You have to resolve *which* compensation applies at compensation time rather than deciding at failure time from what your code remembers — and note that asking the processor is itself a read that can go stale between the answer and your write. The sturdier shape is to attempt the void and branch on the specific error. Better still, use an endpoint that decides atomically, the way Adyen's cancel-or-refund does.

**The compensator itself fails.** Network timeout, processor 500, pod evicted. Now ask where the record lives that says a compensation is owed. If the answer is "the `except` block's stack frame", the obligation dies with the process and the money stays captured. This is the one that produces a support ticket six weeks later with nothing in your logs that looks like an error, because the error was raised by a process that no longer exists.

**A refund succeeds and the response is lost.** You retry. Without an idempotency key that is two refunds — and having a key is not the same as having one that works, which I come back to at the end.

**The arithmetic is not what you think.** With partial captures, the refundable amount is the captured amount minus prior refunds, and prior refunds must include the pending ones. Count only settled refunds and a fast second click refunds the customer twice, legitimately, through the front door.

Underneath all four sits one point. **Compensation is not symmetric.** A capture can be voided. A refund cannot be un-refunded. Some steps have no inverse at all — you cannot un-send an email, un-sign a document, or un-ship a pallet. So the design is not "undo A". Every compensation is a persisted obligation with its own state machine, its own retry schedule, and a terminal state that is a human work queue rather than a silent give-up.

Write that down and you have written an orchestrator. The only question is whether you write one, or twenty slightly different ones inside every service that touches money.

Ours does not do all of that, and the gap is worth naming because it is the failure mode two paragraphs up. The walk compensates in reverse order and, if a compensation itself fails, it halts and surfaces the errors for an operator rather than continuing. That is a deliberate choice — carrying on after a compensation has failed is how you make the state worse — but it is a human queue without a retry schedule behind it.

**Check this yourself, it takes thirty seconds.** Find the line where your payment provider returns a successful capture, and the line where you commit that fact to your database. `SIGKILL` the process between them, then bring it back up. Does anything reconcile? In most hand-rolled implementations nothing does, because the only record that a capture happened was the row that never got written.

Everybody's first version is one statement, and it is genuinely correct:

UPDATE items SET qty = qty - 1 WHERE id = ? AND qty >= 1;

Row-level lock. Atomic. Correct for decrementing now. Not correct for "hold this unit for twelve minutes while the customer finds their card", which is the actual requirement.

Separate the decision to take stock from the decision to keep it, and four problems arrive together.

**Expiry is a write, and something has to perform it.** Lazy expiry computes availability as stock minus unexpired reservations at read time. Correct, and it turns every availability check into an aggregate on your hottest path. Eager expiry runs a sweeper instead. Fast, and it opens a window in which a reservation is expired by wall clock and still live in the table. So the sweeper releases a row at the same instant checkout commits it, both transactions read a row that is valid at the moment they read it, both write, and you have now sold stock you also released.

**The multi-item deadlock.** Two orders each want the last unit of A and the last unit of B, in opposite order. Each reserves its first item and blocks on the second. The fix is a deterministic lock ordering: sort the basket by primary key before reserving anything. Almost nobody does that in the first version. It never shows up in tests either, because almost nobody writes a concurrent test against the same two rows.

**A reservation is a saga resource,** so it is compensable, so release must be idempotent. Release twice and stock goes up by two. The reservation now needs a state, which makes it a state machine, which puts you back in section 1.

**The two error directions are not equally bad.** Underselling costs a sale. Overselling produces a customer-facing failure and a refund, and in some categories a regulatory conversation. Any design that treats the reservation counter as eventually consistent has quietly chosen to oversell. Fine for concert merchandise. Not fine for prescriptions, or for a numbered seat at a specific venue on a specific night, and it should be an explicit choice rather than a property of your cache TTL. Airlines are the counter-example people reach for, and they are the wrong one. They oversell on purpose, with a compensation regime built on top of it.

Then the two sections meet. What happens when a reservation expires *between* the authorization and the capture? Somebody else took the last unit, you are holding a live authorization against goods that no longer exist, and the instrument you need to reverse it depends on whether the processor has run its nightly batch since you charged the card. Void or refund? Not your decision. That composition is where hand-rolled versions die, and it dies most reliably when payments and inventory belong to two teams who each tested their own half.

**Check this yourself.** Run your checkout path from two processes at once against the last unit, no sleeps, no mocks on the database. Then run it again with a two-item basket ordered in opposite directions. Most codebases survive the first and fail the second, and the failure presents as a timeout rather than as a correctness bug, which is why it stays open.

Loyalty points. Store credit. A wallet. It always starts as `users.balance integer`, and for about four months that is genuinely fine.

Then the first support ticket arrives: "why do I have 340 points?" You cannot answer it. So a transactions table gets added for display, and you now have two sources of truth that can disagree. They will, because one is written in the request path and the other in a logging call somebody wrapped in a `try/except: pass` during an incident.

Then a correction. An award was wrong by 200 points and the customer has already spent some of it. You cannot delete the row, because the spend depends on it. The only correct operation is a new opposing entry. What happens instead, when the history table is decorative, is that somebody opens a console and fixes the balance column by hand. From that moment the history is a lie every later reconciliation inherits.

Nobody writes that down.

Then expiry. The 500 points earned in January expire before the 200 earned in March, so redemption must consume in order. A single integer cannot express that. You need lots and a consumption order, at which point the balance is already a derived aggregate over unexpired lots and you have simply not admitted it.

The double-entry design is old and dull. Every movement is at least two entries against named accounts — one debit, one credit — summing to zero. Balances are derived, never stored as truth.

The reason it wins is not elegance:

SELECT transaction_id FROM ledger_entries
GROUP BY transaction_id, currency
HAVING SUM(amount) <> 0;     -- must return nothing, always

Per entry, not global. A global `SUM` over the whole table is the version everyone writes first, and it is too weak: two offsetting mistakes in different transactions cancel out, and posting to the wrong account never perturbs it at all. Group by currency too, or you are adding points to dollars.

That is an invariant you can assert continuously, in production, on real data. An unbalanced ledger is *detectable*. An incorrect balance column is not detectable by any query, because there is nothing to compare it against. Everything else follows from that one difference.

It costs you reads that become aggregates or maintained snapshots, and more code. Real ledgers do store balances — as period closing figures and running snapshots — and re-derive them to check. What they do not do is treat the stored number as the truth. If your points are non-refundable, non-expiring and non-auditable, a column is the right call. The moment one of those three stops being true, it is not.

**Check this yourself.** Recompute every user's balance from your history table and compare it to the stored balance. Count the mismatches. That number tells you whether you have a ledger or a cache that someone has been calling a ledger.

None of the decisions above is a *bug* at the moment it is written. The `except` block that calls `void` is a reasonable line. The single-statement decrement is correct. The balance column is the right call for four months. Each is a correct line in a correct file.

What they share is the failure signal. When these go wrong nothing raises, nothing turns red, and the numbers stay plausible. Review reads for wrongness, and none of these is wrong on the page.

Review looks at one diff.

We have a scar that makes the point better than the argument does. Our Stripe checkout handler derived its idempotency key from the per-invocation request id. A different value every time, so functionally no key at all. It reads as careful code; the word "idempotency" is right there in the variable name. A double-click would re-enter with a new request id and create a second payable session. It now hashes the record UUID and the canonicalised form, so the same intent produces the same key — wherever the call carries a record to anchor on. Where it does not, it still falls back to the per-invocation id, so the scar is narrowed rather than closed.

Go and look at what your idempotency key is derived from. A surprising number are unique per attempt, which is the same as not having one.

The rule: **invoked twice, applied once.** Two mechanisms get you there and they are not the same one. Sending an idempotency header puts the dedup on the provider's side, inside whatever window they keep — about a day, for Stripe. Owning it yourself means a claim row keyed on the record, the action and the idempotency key. Its *insert is the gate*: uniqueness is enforced by the store, so a racing duplicate raises a conflict. A check-then-act is neither. And even the claim row only gets you at-most-once *claiming*: the process that claimed and then died is section 1 again.

A vendor telling you how many services it ships is making an argument you should treat as a liability rather than a feature: it is a lot of surface written by people you have never met. The number, since you will ask, is 22 transactional services and 117 operations, roughly 30 state machines. Two questions follow, and the second one is the one I would ask.

**How much of it is tested?** The core suite is 1,613 end-to-end contract tests across 123 files, and the state machines have their own file: `test_34_state_machine_transitions.py`, 48 tests. Invalid transitions return 409.

I am not going to quote you a coverage percentage, and the reason is better than the refusal: we have two defensible ways of counting assertions on error contracts in that suite and they disagree by more than three times. A number that moves that much under a careful reading is not a number.

Here is the residual instead, which is worse for me and more useful to you. That file carries six skipped classes, and four of its tests are `def test_placeholder(self): pass` under a docstring headed "Tests planned:". Those four are every composite state-machine class in the file: document signature, approval, loyalty points, inventory. The last two are sections 2 and 3 of this article. The inventory stub's docstring lists what it meant to check — `reserve_stock`, `commit_reservation`, `release_reservation` as dual-state idempotent — which is the exact release-twice bug I described above.

So the two hardest things I just lectured you about have a planned test and an empty body.

There is a better one. `test_38_shopping_cart_e2e.py:43` used to carry the line *"Real saga compensation (rollback on payment failure): Out of scope here — covered by test_50_saga_walker_e2e.py."* That file had not existed since May. Its source was never committed; only a `.pyc` survived.

So the suite carried a written claim that some other file covered saga compensation. The file was gone. Every run reported green, and the broken inverse templates shipped underneath it — the same fourteen the contract suite now reports, still broken. There is now a guard test, `test_00_coverage_claims.py`, whose entire job is to fail when a suite delegates coverage to a module that does not exist. A dangling delegation is worse than no comment, because it stops the next reader from looking.

**Which of these are actually harder than what you would write?** Not all of them. Some of the 22 really are thin, and writing one yourself costs an afternoon rather than a quarter. I was going to name `comment` as the example and it turns out to declare seven operations, a state machine and one of those fourteen broken inverses, which is its own small lesson about how the thin ones get counted. The three above are different in kind, for the reason in the previous section.

So the argument is narrower than "22 services, therefore write nothing". It is that compensation logic implemented centrally beats the same logic reimplemented per service by whoever picked up the story. Be precise about where it is tested, though: not in the suite I just quoted. Search those 123 files for a test function with `saga`, `compensat` or `rollback` in its name and you get nothing.

There is a separate contract suite that checks every inverse template against the operation that is supposed to publish it. It is red, on purpose, and has been since it landed. Three of its six tests fail and report fourteen real violations — nine output-schema, four input-schema, one orphaned key — across shipped manifests including inventory adjustment and the loyalty accrual and redemption paths. It is detection for an unfixed defect rather than coverage of a working one, landed ahead of the fix so the violations stop being invisible. I would rather tell you that than let the word "suite" do work it has not earned. The count is a consequence of that, not the point of it.

Three things to know before taking any of this seriously, and they are not the complete list. The export is one-way: you can pull the source down as an archive or push it to a repo you own, but nothing flows back, so the moment you fork it you are maintaining it. Generated apps take the SDK by floor rather than by pin. A running application can change underneath you when we publish, and you cannot hold it still. And the platform is designed for SOC 2 and HIPAA controls without being certified: no BAA, and no third-party penetration test published.

Sixteen applications built this way are running at real URLs, indexed at supero.dev/apps.

---

The three checks cost under an hour between them. If all three come back clean, this was handled on your system and you can go and do something else.

*Friday, part 2: what happens when you ask a model to generate this code, and why the thing that catches its mistakes cannot itself be a prompt.*