Eval Notes

Why I keep a set of three boring prompts

Posted 30 October 2026 · from notes kept since spring

Every time a new model comes out I run the same three prompts before reading anything about it: summarise a long, messy email thread; fix a small bug in an unfamiliar script; and explain a tax rule to a non-specialist.

None of them is clever. That is the point. They are the tasks I actually do, and after a year of keeping the old answers side by side, the differences between models are much easier to see than on any public chart.

The one rule: never edit the prompts. The moment you "improve" them, the history stops being comparable.