Buying a phone number is a distributed transaction

The API makes it look trivial.

const number = await carrier.numbers.buy({ phone_number: "+1..." });
await db.insert("rented_numbers", { user_id, e164: number.phone_number });
await stripe.subscriptions.create({ customer, price });

Three lines, one number, done. Ship it.

What you actually wrote is a distributed transaction across three systems. They share no transaction log, they have no two-phase commit, and none of them can roll back the others. The carrier will keep charging you for a number your database has never heard of. Stripe will stop charging for a number your database still thinks is paid up. Neither one is going to mention it.

I run a virtual phone number product. Below are the failure modes that actually cost us money, roughly in order of how much.

The orphan taxonomy

Write down the states first, because the interesting ones are the states nobody designs for. Three systems, each holding an opinion about a single number:

Every row under the first one is reachable from a plain network timeout at a bad moment.

The first orphan class is the worst, because you cannot see it from inside your own product. No row, no user, no support ticket. The number sits in the carrier’s inventory producing an invoice line every month until somebody actually reads the invoice.

The second class is the one that generates a complaint. The rest leak money in one direction or the other, quietly.

Reconcile, don’t prevent

The instinct is to armour the write path. Sagas, compensating transactions, idempotency keys on everything. Use idempotency keys, they cost nothing and they do help. But the gap does not close there, because “carrier says yes, then the process dies before the insert” is always reachable. No ordering removes it. Orderings only change which orphan you end up with.

So the real answer is a scheduled reconciler, and how you design that matters more than the write path does. Ours runs daily. Three properties I would keep in any rebuild:

Diagnosis is read-only and lives apart from action. One function reads all three systems and returns a delta. Separate code decides what to do with it. The point is being able to run the diagnosis from a script, against production, on a Sunday, with zero chance of it changing anything.

It acts only when the situation is unambiguous. DB says pending_cancellation, Stripe says canceled? A customer.subscription.deleted webhook got dropped, the intent is not in question, finalize locally. DB says cancelled but the carrier still holds the number? Log an anomaly and let a person look. That case is rare, it means a release call failed, and the obvious automated fix is destructive if the diagnosis was wrong. Anything stranger goes into a deltas[] array for admin review instead of into a clever branch nobody will ever read again.

It searches every provider account you own. We keep more than one account per carrier. Our first orphan scan walked the primary account only, and orphan inventory sat undiscovered in account two for weeks. If your provider registry can list accounts, iterate the list rather than the default.

Two smaller things that paid for themselves.

Give the reconciler a real timeout budget. Ours makes four passes with per-row calls out to a carrier, Stripe and an app store. On serverless, the default function timeout will cut the run partway through, and the passes at the end are the money backstops. A truncated reconciliation looks exactly like a clean one in your logs.

Alert on deltas, not on fixes. Nobody reads “reconciled 3 rows” every morning for a year. What you want is a ping when the reconciler finds something it refused to touch.

A number is not a number

The second category of pain: “phone number” is not a type. It is a family of products with very different behaviour, and the search API flattens them into rows that look interchangeable.

Capability flags describe the number, not your setup. A search result carrying sms: true means the number is technically SMS-capable. Whether a message actually lands depends on your account configuration, the messaging profile it sits on, and in some regions on a registration you have not filed. We shipped numbers that could not receive a text, sold as numbers that could. The flag was not wrong. It answered a different question than the one our UI was asking.

The same product name behaves differently per carrier. Toll-free is the clearest case. Identical label, materially different rules depending on who sold it to you, particularly around permitted use and how it interacts with caller ID verification. If you run more than one carrier, toll-free cannot be a single branch in your code.

Listed is not the same as in stock. Our provider lists over a hundred countries. When we enumerated actual purchasable inventory, fewer than half had any. A country picker built from the provider’s country list advertises numbers you cannot sell, and every empty result reads to a user as a broken product.

Some countries need a regulatory bundle. Proof of address, identity documents, sometimes a local presence, filed with the carrier and reviewed by a human over several days. This is not a deferred edge case, it is a second product surface: document upload, a submission state machine, a review queue, rejection handling. Two notes for my past self. Persist the identifiers a bundle submission hands back before you send anything else, because losing a requirement_group_id mid-flight strands a live submission you can neither finish nor cancel. And build the flow after somebody asks for one of those countries, not before. Ours has been used zero times.

Inbound is a separate product from outbound

Buying a DID does not make it ring. The number has to be attached to a voice application on the carrier side, and until that happens, calls to a number you own and pay for simply fail. It catches people out because outbound starts working immediately, so the number feels live.

Then there is the part I did not see coming.

Answering a call costs money, even to say no

While our inbound bridge was still behind a flag, we handled incoming calls by answering the leg and playing “this number can’t take calls yet.” Polite, and also billed. An answered leg carries a 60-second minimum from the carrier, so every robocaller scanning a freshly issued DID cost us a full carrier minute to read a canned sentence to a machine. It is unbillable by construction as well, since charging the number’s owner for our own error message would be wrong.

Roughly half the inbound legs we saw in a month were exactly that.

Fresh DIDs get scanned. The scanning is automated, it starts within days of issue, and it will find every number you buy. The fix is one line of intent. Decline at the signalling layer instead of answering.

// Costs a 60-second minimum, every time.
await call.answer();
await call.speak({ payload: "This number can't take calls yet." });
await call.hangup();

// Costs nothing.
await call.reject({ cause: "CALL_REJECTED" });

One of our two decline paths already knew this and said so in a comment. The other did the opposite. Which is its own lesson: a rule that lives only in a comment on one branch is not a rule.

Settle your own row before you touch the provider

Last one, and it travels well beyond telephony.

Our decline path used to mark its database row terminal after three sequential round-trips to the carrier’s control API. The carrier, meanwhile, fires an answered webhook the moment our own answer command lands. That webhook regularly won the race, found a live non-terminal row, and stamped it answered. Downstream, a monitor watching for “answered calls that escaped the minimum charge” started firing on calls nobody had answered.

The old code carried a comment asserting the terminal guard made that race impossible. It did, in tests, where the mock adapter returns instantly.

The fix was to settle the row first, final state and zero cost, and only then talk to the provider. The invariants downstream code relies on are now true before any event can arrive, in whatever order they arrive.

Most of this reduces to one idea. Your database is the only system in the transaction whose consistency you control, so give it its final answer first and treat everything else as something you reconcile toward. Carriers, payment processors and app stores are eventually consistent with your intentions at best, and only if you go and check.

If you are about to add numbers to a product: write the state table before you write the write path, schedule the reconciler in the first sprint rather than the fourth, and never answer a call you do not intend to be paid for.

I build Twin-Phone, a virtual phone number product. Every failure mode above is ours. Questions welcome in the comments.