The overnight run had a scheduler that submitted video jobs one after another, and a watchdog that checked on both the scheduler and the render server every two minutes. The watchdog's first rule was to not be clever: it respects a stop file, and it gives up after three revivals. Most of what went wrong that night was about the gap between what I had changed and what was actually running.
The fix that the process never saw
I added a fallback for an engine that sometimes can't start, saved the file, and moved on. Then I realised the scheduler had been running since before the edit. A Python process reads its source once, at launch; the file on disk and the code in memory had quietly become two different programs. The fallback would not have fired, and the run was about to stall on exactly the failure I had just “handled.”
The check is cheap and I now do it every time: compare the process start time against the file's modification time. If the file is newer, the process is running old code.
Orphans
When I took the scheduler down to restart it, I assumed the work stopped with it. It didn't. The scheduler submits a job and then waits; by the time I killed it, the job was already in the render server's queue and kept running with nobody watching it. The restarted scheduler, which remembers only what it has recorded as finished, submitted the same item again.
Two copies of the same clip, on a machine that can render one at a time, is a very effective way to lose an hour. The rule now: stopping a scheduler includes clearing the server's queue, because the queue is where the scheduler's intentions actually live.
A number that was one run old
The scheduler decides whether to start another item by checking whether it can finish before the morning deadline. That check uses a duration estimate per model, and the estimate for the largest model was 37 minutes.
Thirty-seven minutes was the time of the first run, entered once and never revisited. The next three real runs took 40.7, 40.7 and 42.0 minutes, so I changed it to 41. A low estimate in a deadline check fails in a specific way: it starts items it cannot finish, and the last clip of the night is cut off. A number that gates a decision deserves to be replaced by measurements the moment there are more than one.
Related repair, same night: the viewer reported how long each clip took, but for most clips the figure was reconstructed from guesses. I switched the source of truth to the job record's own start and finish timestamps, then the schedule ledger, and only then the reconstruction. The number of clips with a real duration went from 9 to 44, and the averages stopped including estimates as if they were measurements.
What they have in common
None of these are hard bugs. Each is a case where I held a mental picture — “the fix is live,” “stopped means stopped,” “37 minutes” — that had been true at one moment and was never re-read. Overnight systems are unforgiving about that, because the person who could notice is asleep.