An unattended pipeline of mine had been running for weeks, and its failure log had grown to 168 entries against 132 successes. That ratio looks like a broken system. Before changing anything, I split the log by what the entries actually meant — and 124 of the 168 weren't failures at all. They were deliberate skips: when a GPU-heavy job was already running and memory was tight, the pipeline stepped aside and left the item in the queue for the next round. That is the behavior I designed. It had just been writing itself into the same file as the real breakages.
Forty-four genuine failures were left. That is a very different problem from 168, and it's a problem you can actually read.
One accident wearing twenty-eight masks
Twenty-eight of the forty-four had no recorded reason at all — the run died before it could say why. They were also all clustered after a single timestamp the previous evening, twenty-one of them consecutive. Something had changed at that moment, and nothing in the repository had.
The cause was outside the code entirely: an OS developer-tools licence agreement had gone unaccepted, which turns the system Python at /usr/bin/python3 into a stub that refuses to run. Interactively, in my own shell, everything still worked — my shell resolves a package-manager Python first. The scheduler doesn't inherit that shell. It got the stub and died before reaching a line of my code.
Accepting the licence fixed the immediate breakage. Pinning an explicit PATH in the scheduler's job definition fixed the class of it, so the daemon looks up the real interpreter itself instead of borrowing my luck.
Two defects, both of them hiding the evidence
That left sixteen failures in two shapes: six saying "no matches" and ten saying only "Traceback". They looked like two unrelated bugs. They reproduced as exactly two defects, both of the same kind — a step that fails in a way that destroys the explanation.
The splitter chops a long source document into sections before handing them to the model. Given a short document with no section headings, it returned zero pieces, and the shell step downstream then failed on a wildcard that matched nothing. The document wasn't malformed; the splitter just had no answer for "this is already small enough". It now returns the whole document as a single piece.
The quality gate reads the model's output file and scores it. When the model timed out and never wrote that file, the gate crashed on the open call, and the only thing left in the log was a stack trace pointing at the gate — not at the timeout that caused it. It now checks that the file exists and is non-empty first, and returns a clean failure verdict that says so.
Both defects sit before the model in the chain, which is why I gave them their own section in the protocol doc rather than filing them with the model-output failure modes. A failure of the plumbing and a failure of the writing are diagnosed in completely different places.
What I'd take from it
Re-running two of the oldest failed documents end to end, both passed. Nothing about them had ever been wrong.
The lesson isn't about any of these three bugs. It's that a log which mixes "I chose not to" with "I couldn't" will read as a catastrophe forever, and that an error handler which crashes while handling the error is worse than no handler — it substitutes its own stack trace for the evidence. I spent the first part of this session looking at a number that was never true.