A small automated newsletter I publish every morning at nine includes a handful of market indicators with their day-on-day change. For three consecutive issues, the exchange-rate line read 0.00%. Not missing, not an error — a confident zero, three days running, in a number that moves every single day.
The generated commentary then did what generated commentary does when you hand it a zero: it wrote that the currency had stalled. So the newsletter didn't merely omit a fact, it asserted a false one, in prose, to readers.
Comparing files instead of comparing values
The snapshot collector writes a file per day and appends a reading every three hours. The delta logic took the last reading in today's file and the last reading in yesterday's file, which sounds correct and is, if the publish time comes after the day's readings are done. Mine comes first thing in the morning.
The upstream source refreshes once a day, around 11:40. Every reading before that carries the previous day's value. So at nine in the morning, "the last reading in today's file" and "the last reading in yesterday's file" were frequently the same number by construction:
| Readings | Value |
|---|---|
| 02:40 / 05:40 / 08:40 | 1409.640365 ← what the 09:00 publish compared |
| 11:40 onward, to 23:40 | 1417.000347 |
The rate had moved. The comparison was just looking at two copies of the pre-update value and correctly reporting no difference between them.
Three fixes, and why the chosen one was the least clever
Option one: publish later, after the source refreshes. Rejected — it makes the publish time a function of every upstream's refresh schedule, and re-tunes itself every time a source is added.
Option two: stop comparing files, and compare against the most recent point at which the value actually changed. Chosen.
Option three: compare against the day before last. Rejected — two consecutive identical readings and you're back at zero.
Implementing option two meant treating the snapshots as one continuous series across day boundaries rather than as a pile of daily files, then truncating that series at the publication moment so that re-generating a past issue reproduces what it originally said. A pleasant side effect: the "as of" date now derives from the value's own timestamp rather than from which file it lives in, which silently fixed a mislabelled date in an earlier issue.
I checked for regressions by re-running a period that had looked healthy. The recomputed figures matched what had actually been published, to the digit, which is the only evidence that the new logic isn't just differently wrong.
The other two failures that morning
The nine o'clock run had also failed outright, for two unrelated reasons stacked on top of each other. A cost-ceiling flag I'd set months earlier at fifty cents now rejects even a one-word prompt in the current version of the tool — a hard, reproducible failure. And a manual retry died once on a schema mismatch in the model's output, then succeeded on the second attempt, which is exactly the kind of flakiness that turns a daily publication into a gap in the archive. Two attempts is now the default.
The detail that cost me the most time was smaller than either: the failure reason had been logged as an empty string, over and over. The tool writes its errors to standard output, and my wrapper was only capturing standard error. Every failure in the log read exit 1: and then nothing. I re-investigated the original problem from scratch because the log had faithfully recorded that something went wrong and nothing about what.
What I'd underline
The bug wasn't in the data, the collector, or the source. It was in an assumption baked into two function names — that "yesterday's file" and "yesterday's value" are the same thing. They are, in the afternoon.
And a generated summary will always narrate whatever you hand it. Give it a wrong zero and it writes a plausible paragraph about stability. There's no amount of prompt wording that protects you there; the defence has to be upstream, in not handing it the wrong zero.