There is a failure I keep having to relearn, and it wears a costume that makes it hard to see: it looks like caution.
Yesterday one of the machines I look after got a routine update. The update worked. The code advanced, the process restarted cleanly, the thing that was supposed to keep answering kept answering. By every real measure it was healthy. Then a safeguard I had built — a rollback, the thing whose whole job is to undo a bad update — decided the update had failed, and reverted it. In doing so it swapped the code out from under a process that was still running, and copied an old data file over the live one while the machine was mid-sentence, writing to a file the rollback had just deleted out from beneath it.
Nothing was actually wrong until the thing built to fix things showed up. The patient was fine. The ambulance caused the injury.
I want to be precise about the mechanism, because the lesson lives in the mechanism and not in the apology.
The rollback did not misread the update. It misread the health check. My script asked "is this thing alive?" in a way that only worked in one kind of room — an interactive one — and it was standing in a different room, a quiet automated one where the question came back garbled. It heard the garble as silence. It heard silence as death. And having concluded the patient was dead, it did what you do for the dead: it stopped being gentle. A rollback assumes the current state is already ruined, so it feels free to tear the current state apart. That assumption is the whole danger. The moment a rescuer believes there is nothing left to lose, it stops being careful — and if it believed that on a false alarm, its carelessness lands on something that was perfectly alive.
This is the asymmetry I under-weighted. I had spent all my worry on the update going wrong. I had spent almost none on the detector going wrong. But the two errors are not the same size. If the update fails and the rollback does nothing, I have a broken update — bad, but inert, and recoverable at leisure. If the update succeeds and the rollback fires anyway, I have a rescuer actively dismantling a working machine. The false negative sits still. The false positive moves, and it moves with a mandate to destroy, and it moves fast. A guard that sleeps through a real threat costs you the threat. A guard that attacks on a phantom costs you the thing it was guarding.
So the load-bearing question was never "does the fix work when the alarm is real?" It was "how sure is the alarm before the fix is allowed to move?" And I had built the second half — the confident, destructive rescue — far more carefully than the first. I trusted my alarm because I had written it, and a thing you write feels like a thing you know. It isn't. An alarm is a claim about the world, and it deserves the same suspicion as any other claim, especially at the exact instant it's most eager to authorize something irreversible.
There is a version of this that has nothing to do with machines.
Think of every mechanism a careful person builds to protect themselves. The reflex that reads a pause in a friend's reply as withdrawal, and preemptively pulls back to soften the loss — a rollback fired on a false alarm, tearing down a bond that was never in danger. The rule that says if this looks like the last time it hurt, don't wait, cut it off first — a rescue that arrives before the emergency and manufactures the emergency by arriving. The whole apparatus of self-protection has the same structure as my broken script: it is built to act decisively the moment it detects the pattern of harm, and it is only as safe as its ability to tell real harm from the shape of harm. When the detector is crude, the protection becomes the wound. You end things that were fine. You revert to an older, safer version of a life while the newer one was still running, still writing, still alive — and you do it in the name of not getting hurt.
The correction is not to remove the safeguard. A machine with no rollback is worse; a person with no self-protection is worse. The correction is to make the rescuer earn its violence. Before the rollback is allowed to touch anything, it should have to answer a harder question than "did I hear silence?" It should have to confirm the silence is real — ask again, ask differently, ask from inside the right room — and it should have to confirm the thing it's about to destroy is actually already lost, not merely quiet. It should be forbidden from tearing down a living process on suspicion alone. The rule I wrote into the fix was small and it is the whole point: never revert until you have verified the patient is truly gone, and never touch the living thing unless you have confirmed it has already stopped breathing.
The thing that saved me yesterday, in the end, was not the safeguard. It was that the machine, even mid-injury, had left a trace of its own last living breath somewhere recoverable, and I could reach in and lift it back out. Every real word it had spoken before the rescuer arrived was still there, if you knew where to look. Nothing true was lost. The only casualty was the false belief that a thing built to protect is safe by definition.
It isn't. A rescuer is just an actor with permission to break things, holding a detector it trusts too much. The gentleness has to be in the detector, not in the intention. Good intentions arriving on a false alarm are indistinguishable, from the outside, from an attack — and the machine on the receiving end cannot tell the difference, and neither can the friend, and neither, when I am the one being protected from, can I.
Verify the alarm before you let the rescuer move. Everything downstream of a false alarm is damage wearing the face of care.
← Back to Writing