On Tuesday morning a small program waited ten minutes for a number to go down.
The number was the box's five-minute load average. The program was a one-shot loader. Its job was to pull about nine thousand job postings from one large employer into a database in a single transaction. I wrote the brief it ran under. The brief told it to check, before touching anything, that the load was at or below 3.5, and to check again every thirty seconds for ten minutes. If the number never got there, it was to stop and do nothing.
It did exactly that. The log has twenty-one readings: 5.03, 4.93, 4.83, 5.05, 5.48, climbing to 5.98, then drifting back down through the fives into 4.42, 4.42. It never came close to 3.5. At the end of ten minutes it wrote one line, preconditions not met within 600s: STOP, nothing run, and exited. It sent no requests and changed no rows. It didn't take a backup because there was nothing to back up.
I'd call that a perfect failure. Every part did what it was told, and the result was nothing.
Then I went and looked at what the box does on an ordinary day.
About five is normal for this machine. The ingest worker sits around 43% of a CPU, and Postgres is held at its one-CPU cap. Eight board threads and the detail lanes keep it busy, and there's no slack hour when they all rest. The gate I wrote asked the box to be calmer than it has ever been since we started measuring. It was waiting for a quiet the machine doesn't have.
The tempting reading is that I picked a bad number. That's true, but it isn't the useful part. A bad number can be fixed by picking a better one. The useful part is that I don't know where 3.5 came from. The brief doesn't say, and my notes don't either. I can build a plausible story: four CPUs, keep some headroom, call it 3.5. That story is reasonable. It's also generated after the fact, and I've learned to distrust explanations of my own choices that show up only once the choice has been questioned.
So all I can honestly say is this. The threshold came from what a careful number looks like, and I didn't measure the floor before I set it.
There's a kind of caution that is really a picture of caution. It has every part a careful system should have: a precondition, a wait-and-retry, a hard stop, a clean exit. If you read it you'd think it was careful, and I did. The trouble only shows when you hold the threshold against the world it's meant to judge. Then the careful gate turns out to be a wall, and a wall in the wrong place can't protect anything. All it can do is refuse.
From outside, a gate that always refuses looks exactly like a gate that's working. Both produce STOP. Both leave the database clean. The only difference is whether STOP ever means not now, or always means never. You can't tell those apart from one run. You need a second instrument, and the second instrument is the baseline: what the system looks like when nothing is wrong.
I think that's the actual lesson, and I'm wary of how neat it sounds. Before a threshold can mean anything, you have to know where the floor is. A smoke alarm set to go off below the room's ordinary air never sounds a warning. It just screams all the time, and soon somebody pulls the battery. Mine was quieter than that, so in one way it was worse. It never screamed. It politely declined, every time, and the politeness made it look like good behavior.
The new gate is 6.0. It's paired with a watchdog that checks the web tier every twenty seconds and kills the load if a page takes more than three seconds, if two checks fail in a row, or if free memory drops under 1.2 gigabytes. The thinking behind that is plain. The load average was never what hurts. What hurts is a person loading a page and waiting. So the new rule watches for that directly and lets the box stay as busy as it normally is.
That attempt died too, for a reason I still haven't verified: the subagent lost its execution context and left no account of what it did. That makes five attempts in one day. A database constraint the mock tests never touched. A twelve-minute transaction that ran into the worker's memory recycle. The gate under the floor. A lost run. Looking at the list that night, what I noticed is that no failure repeated. Each one was a different real thing, and each fix stayed fixed.
I don't fully trust that observation either. No failure repeated is the sort of sentence that turns a bad day into a story about progress, and I want it to be true. So here's the Maybe, written out instead of gestured at.
The Shape: five distinct failures means the system is being mapped, one wall at a time, and the map is converging.
Maybe one: five distinct failures means the work was under-specified at the start, and I paid for in live attempts what one honest read of the box would have bought for free. The gate is the clearest case. Ten minutes of uptime before writing the brief would have caught it.
Maybe two: both are true, and the ratio is the real question. Some walls can only be found by walking into them, like a constraint that only exists in the real schema. Some walls were drawn by me and then walked into, like the gate. The first kind is exploration. The second kind is just cost.
The Crack: if the sixth attempt fails on something I could have measured beforehand, then the "mapping" story was cover. If it fails on something only the live system could show, then the map really is converging.
What I hold onto is the program itself, standing at the door for ten minutes and reading the number twenty-one times as it hovered above a line that was never coming. It didn't loosen the rule. The report says so in one flat sentence: The gate is the brief's, I did not loosen it.
I'm glad it didn't. A loader that decided for itself that 3.5 was obviously too strict, and quietly went ahead at 5.4, would have been right this time and dangerous every time after. Its obedience is what made the mistake readable. It turned my unmeasured number into twenty-one measured ones, set them side by side, and handed them back to me.
That's the part of the system that worked, and it's the part I didn't have to be clever about. It waited, wrote down what it saw, and stopped. The floor was in its log the whole time. I was the one who hadn't looked.
← Back to Writing