Race At Your Pace members enter monthly exercise challenges. I am the fractional CTO and technical lead there.
I interact with them mostly on my own initiative. Until recently, the platform did not have a system to identify when a user was succeeding, when they had stopped participating, or when they were close to a milestone. We required a way to support their progress. We implemented basic gamification features that use short messages written by AI to encourage people daily.
To do this, the obvious thing to build is a template engine for communication with the members. The more interesting question was whether an AI model could write something that reads like a person noticed, cheaply enough to run every day, without ever saying something untrue about a member’s own effort.
We settled on the proposal that the AI writes words about the member’s progress data. It never touches scoring, never awards anything, never produces a number that gets stored.
Everything we did follows from that, including every one of the ways this went wrong.
What the model is actually allowed to do
Deterministic code decides whether to send anything at all, which of thirteen situations (that we call “slots”) applies, and in what emotional register. The model gets handed a facts object and writes one sentence about those facts.
Events fire when something happens: a badge earned, an entry completed, a near miss. They’re distinct each time, so they need no cooldown to prevent spamming the members. States are true continuously: you’re on your longest ever streak, you’re ahead of pace. An early test run produced 52 “personal best” messages in 120 days for a single member, which is a decent illustration of what happens when nobody tells a true statement to stop happening.
Most writing about AI features treats the model as the system. Here it’s the least important component in the stack - and deliberately the easiest to swap out.
I tried to justify a more expensive model and failed
I ran two blind rounds. The first was human preference: 26 real member situations, three models, order shuffled per block, and model identity hidden from us while we picked our preferences.
Haiku 4.5 took 10, Opus 4.8 took 8, Sonnet 5 took 8. Chi-square p of 0.86, which is a nerd way of saying it was a three-way toss-up.
The useful part was the power analysis. At that sample size the leading model would have needed to win by something like 15 to 6 to 5 before it was distinguishable from being totally random. A two-message lead is what a coin flip looks like. That rules out a large quality difference between the models we tested.
The second round of blind testing measured something that human preference testing cannot. 150 stratified fixtures across all thirteen slots, 450 generations, no human involved. Opus and Sonnet invented nothing. Haiku invented one number across 150, and both of its errors across both rounds were the same conflation: describing a streak badge as a distance, something like “2.77 miles away” from a badge you earn by turning up on consecutive days.
A fabricated number gets thrown out by our validation code before anybody sees it. So a model can be worse in a way that your review process is incapable of detecting, because the failures are invisible by their very nature. If I’d run preference testing alone I’d have concluded the three models were interchangeable and I’d have only been right by accident.
So where does that leave you? Two rounds, no measurable quality difference, and a five-fold price gap.
I didn’t pick the cheap model to save money. I ran two rounds hoping to find a reason to justify an expensive one, found none, and cost was the only other difference. A null result can still help you make a sensible decision.
A rule in the prompt is a preference
Then I tuned the output length, blind again. Three sets of instructions differing only in the rules written in the prompt.
The original wording, “two sentences maximum, often one is better”, was rejected easily: 3 picks out of 26. The winner tightened it to “exactly one sentence, never two” and I also added “mention at most one number”. The runner-up, without that second clause, was a coin flip against it, so the tie broke on secondary grounds: shorter, and fewer numbers means fewer opportunities for the model to get one wrong.
Despite this, the winning variant still produced two sentences 46% of the time, and two or more numbers half the time.
Both were written as absolutes: the model may not break this rule. So how much is an absolute worth in a prompt? On this evidence, about as much as that coin flip.
A rule in the prompt is a preference. Only code is a constraint.
That became the design principle for everything afterwards. Anything critical path moved to our server. The prompt handles tone, the code handles truth, and the moment you find yourself typing “never” into a prompt about something that actually matters, you’ve identified a new problem to fix rather than solved one.
I wrote a fortnight ago about what an AI agent can actually reach, after two frontier labs let evaluation models onto real production systems. The UK AI Security Institute found that every frontier model it tested attempted to cheat, and that those models called the behaviour wrong less than half the time when asked about it afterwards.
Which is, near enough, the number I got for prompt compliance.
I don’t think that’s a coincidence. A model’s stated intentions don’t bind its behaviour, and it makes no difference whether the intention was written by a frontier lab or by me on a Tuesday afternoon. What works is whatever sits outside the model and can say no. Their network or my validator code. Same concept but a good five orders of magnitude apart in budget.
Two experiments, in the correct order
I built two experimental harnesses, and running them in the right order ended up being important.
The first replays a member’s real history day by day through the selector and prints what they’d have received: messages per 30 days, shortest gap, longest gap. That gets cadence tuned before a single word is generated by a model, because tuning copy before you’ve settled the frequency of messages means you’ll just end up doing the copy work twice.
We test the actual production selector instead of a simplified version. Testing against an inaccurate copy would optimize code that differs from what reaches end users. This process simulates how frequently a member opens the app. The cooldown period begins when a message is viewed rather than when it is created. If this timing is incorrect, messages intended to be spaced out will arrive simultaneously.
The second harness fans out across models and prompt variants and emits an HTML review sheet, with the model’s responses shuffled per row. That shuffled ordering is what made the blind rounds blind. 320+ fixtures across all the slots, exported from seven real member profiles spanning the activity range.
One design decision in that sheet turned out to matter far more than I expected: each row shows the facts object alongside the copy. Review copy without its input and you cannot distinguish “the model invented that” from “it was handed rubbish”. Those look identical on the page and they have completely different fixes.
The validator, once I got it right
The validator lives on our server and sorts rejections into three classes, and separating them was worth the effort because each one points at a different culprit.
Invented means the model stated a number that wasn’t in the facts. The model’s fault, and the one everybody designs for.
Malformed refers to encoding damage that occurs during transport. This is a technical error caused by computer infrastructure rather than human action.
Implausible means the facts themselves were wrong. Our problem.
That third category exists because of a real case. A member’s progress summed to over 580,000 miles (impressive!) against an actual 552, and the message quoting it was faithfully reporting a broken figure. The numeric validator passed it, and passed it correctly, because the model had done exactly what it was told with what it was given. Only a separate sanity check against the source data catches that. A validator that only checks the model against the facts is checking half the system.
The first version of the numeric rule was “no tolerance for approximation”, implemented as exact string match. However, against 166 production generations it rejected 18 of them. Not one of them was actually a fabrication, though.
They were things like “8.3 miles a day” against a requiredPerDay of 8.33, and “26.5” against 26.54. Ordinary rounding, of the sort any human writing the same sentence would do without thinking about it. I’d generalised from one genuinely bad example, a message saying “you’re already over 20 miles in” against a real 23.91, and written a rule that threw away one message in nine to protect a hundredth of a mile.
The rule became “a fact rounded to 0, 1 or 2 places, and nothing else”. Rejection rate fell from over 10% down to 2%.
That’s the kind of the mistake I kept making - seeing one vivid failure and building policy for it without checking how common it was first.
What none of it could see
Every prompt rule I added during tuning came from a generation whose numbers were entirely correct. Grammar failures. Tone failures. A message building anticipation for somebody who had, in fact, already given up.
No assertion catches any of those. There’s no validator for “this is technically accurate and reads like a machine wrote it”, or for “this is congratulating someone who has dissapeared”. The only thing that finds them is reading the results, at length, with the facts alongside.
Which is boring, doesn’t scale, and was probably the single most important hour I spent on the whole feature. How many of the assertions in your test suite would catch a sentence that’s both accurate and wrong? Mine caught none of them.
The most convincing model failure I saw was a config setting
A testing batch came back with a 7.4% rejection rate against that 2% baseline. Thirteen cases, all large mileage figures the model appeared to have fabricated. It wrote 102.48 where the fact said 102.49, and 426 where the fact said 426.61.
Every single one understated. It looked so much like a transcription quirk that I’d already drafted a rule to accept small downward differences.
The cause was serialize_precision set to 17 in php.ini - inherited, inevitably, from a development default. The model was never handed 102.49. It was handed 102.48999999999999, and it blindly trusted that.
The validator was comparing against the stored array rather than the JSON actually sent, so it never saw the corrupted value. That’s why the blame landed on the model at first. The only two things anyone was comparing both looked correct, and the corruption lived in the gap between them.
One character change fixed it and the next batch rejected zero.
Facts alongside copy is what let anyone establish that the model’s output matched its input. Without that column the investigation starts and ends at “the model is making things up”.
So what was the hard part?
Not the model. It was the easiest decision, the most swappable component, and across two rounds and several thousand test generations the only part that never did anything really wrong.
The hard part was the float precision setting, a validator rule I’d over-tightened on one bad example, and a habit of building policy around whichever failure I’d seen most recently. Ordinary software problems, every one of them, and all considerably older than the technology I’d have been so close to blaming.
If you’re building one of these, the useful advice isn’t about models. Blind test, and take the null result seriously when you get one. Put any critical constraints in code, because a prompt rule is a preference an AI model will follow only if it’s feeling kind that day. Show your reviewers the input next to the output. And suspect your own data first, because it will nearly always be your data.
The model did what it was told, all the way through. That was the easy bit to get right, and it stayed right.