A breaker tripped in a restaurant.
Our runtime learned partition tolerance.

The story of one failure's whole journey: a real evening in a real service, seven laws paid for in the field, a library that carries them so applications don’t have to, and a 70-line tutorial app you can break on purpose.

Field notes August 2026 · from the RingScript project

RestoLean is a point-of-sale platform for neighbourhood food businesses, and its first field trial ran in a restaurant in France — real services, real orders, browser apps on the venue’s own Wi-Fi: a remote control on the customer’s phone, a kitchen display on a tablet, a dashboard for the owner. On 15 August, a faulty refrigerator started tripping the circuit breaker.

Each trip rebooted the internet box. Each reboot handed the server a new address. And here is the part that turns an annoyance into a lesson: browsers bind storage to the origin — scheme, host, port, exactly. The notes the owner had been writing all evening lived under http://…10.111:8770. When the box came back as …10.104, those two hours were gone. Not deleted — worse. Intact, in the browser’s storage, under an address nobody could serve any more.

The owner’s reaction was not a bug report. It was a requirement, from a man who has watched cables get cut in the street and building technicians unplug the wrong socket:

A single cut can wreck an entire service. The system must stop depending on the network at all — by design.

What the analysis actually found

“The network” is five different failure domains wearing one name, and the one everybody discusses — the internet going down — is the one a LAN-based system already survives. What stops a restaurant is the router and the one machine acting as server, and those fail far more often than streets get vandalised. Add the two facts that make restaurants unforgiving — a service is time-boxed (an order lost at 12:40 is not delayed, it is gone) and trust is the product (a tool that fails once during a rush is replaced by a paper pad the same afternoon) — and the requirement sharpens: not “offline support”, but partition tolerance between the devices in the room.

The evening became seven laws, each one paid for in the field before it was written down. Three of them give the flavour:

Write locally before any network attempt, and never be silent about pending state. The first version tried the network first and queued on failure. Data loss followed. The rewrite journals first, always, and shows a visible count of what has not left the device.

A server snapshot replaces local state; it never merges. A kitchen display that merged its restored queue with the server’s truth resurrected a four-day-old ticket as a ghost. When the authority speaks, local caches are replaced — local state exists to survive an outage, not to argue afterwards.

Distinguish a blip from an outage. The connection-lost banner used to fire on every reconnection — several times a minute on flaky Wi-Fi — and the owner reported the banner as the malfunction. Alarm fatigue is a real failure mode: it trains people to ignore the one warning that matters. The alarm now waits eight seconds of true silence.

From laws to a foundation

Laws in a document protect nobody. They became a design, and the design became ringscript-pwa 2.0 — the partition library of the RingScript runtime, installed with one command, carrying the whole evening so no application has to relive it:

And because a promise like this is worthless untested, the outage itself became a test: a harness drives a real application, on the real VM, through a real TCP proxy whose sever() destroys live sockets mid-stream — 15 August, reproducible on demand, twenty assertions in 0.7 seconds. Its very first run caught a design gap the document could not see: a reconciliation that ate the device’s own undelivered orders. The rule that survived — a ghost is restored state nothing is carrying; an intent is your own work still en route, and only ghosts are purged — exists because the test ran, not because someone was clever.

Using it: a café ticket pad in 70 lines of Ring

The tutorial app, Café Tickets, is the smallest thing that survives a dead network — and the point is how little of it there is. The application author writes the rules; here is the only one that mentions the network at all:

func TicketCancel cId
    if PwaRung(1) != "streaming"
        return JsonEncode([ :ok = 0,
            :refuse = "a cancellation needs the till -- it will be" +
                      " possible again when the connection returns" ])
    ok
    ...

Placing a ticket never waits — a waiter’s hand does not stop writing because the router rebooted. Serving is monotonic — food that left the kitchen does not roll back. But a cancellation touches the till’s totals, so it needs the till — and that decision is three lines of Ring, next to the rules it belongs with, reading the rung the library maintains. Not an if (navigator.onLine) scattered through UI code: a business rule, where business rules live.

Everything else is one attach and one send function. Then you break it: cut the connection, keep taking tickets, watch the counter climb and the banner name the mode after eight honest seconds, try a cancel and read the refusal, restore the connection and watch the queue drain itself — in order, exactly once each. Reload mid-outage: nothing is lost, because nothing was ever acknowledged before it was durable.

The lesson we would give anyone

The hard parts of surviving a dead network — durability before acknowledgement, ordered replay, idempotency, honest degraded modes, the origin trap — are the same for every application. They belong in a library, written once, tested by severing real sockets. What should remain for the application author is exactly the part that is theirs: what the application means. A field failure is only expensive once if you capture it; the whole trail — incident, laws, design, library, harness, tutorial — is in the case study.