We tried letting AI improve its own prompts

We pointed AI at its own instructions to see if it could improve them, and the short answer is that the rewrites read great and we couldn’t trust one of them. Every failure traced back to the same hole: no evidence. The loop had no idea how the instructions performed in the real world, so “better” meant “sounds better”, which is not a measurement.

Calling it an experiment is generous, for what it’s worth. It was tinkering: a suggested rewrite next to the original, and us deciding by eyeball. Plenty of teams run a version of this loop today, by hand or on a schedule. The failure modes below travel well.

What we actually tried

An AI step in Tallyfy executes straight from its written instructions, so the instructions are the process. That makes self-improvement look one prompt away: hand the instructions to a model, ask for a tighter version, apply. We tried it on our own steps while designing Stern Stella, and the lesson arrived fast.

The rewrites came back shorter and cleaner, and they trimmed lines that looked redundant on the page. Some of those lines were redundant. Others were load-bearing, and nothing in the text marks which is which. Something we learned the hard way: whether a line is dead weight is recorded in the runs, and the rewriter had never seen a run.

That’s failure mode one: rewording blind. The editor with the least context gets the most authority.

A rewrite that reads better can still behave worse

Did any single rewrite look wrong? No. That was the unsettling part, because wrong-looking rewrites would have been easy to police. The diff looked great. The diff always looks great.

Failure mode two is the missing proof. Suppose the new wording changes the result one time in twenty. Reading will not catch that, and spot-checking three outputs probably won’t either. It surfaces weeks later, downstream, when something built on those outputs looks off, and by then nobody connects it to a prompt edit. Quiet drift is the specific nightmare here, worse than loud failure, because loud failure at least files its own report.

After sitting with the diffs for a while, we saw the deeper problem: we had no definition of “same”. Same-ish words and same downstream decisions are different promises, and we hadn’t picked one. Without that definition, “did anything change” has no answer, and a safety story with no answer to that question is a vibe.

Where else does the loop leak?

Three more places, quickly.

The old wording disappears. Most DIY loops rewrite in place, in a document or a config file, so the previous version survives only if someone remembered to copy it somewhere first. The sentence that quietly mattered is now gone, and you learn it mattered a month after it stopped existing. Version control fixed this for code long ago. Almost nobody wires it up for process text.

Somebody babysits it forever. The loop doesn’t run itself; it’s a kludge of scripts and reminders held together by one person’s attention. That person triggers it, reads the diffs, pastes results, remembers to check the aftermath. Attention is the one input that never scales, and when that person goes on holiday, the improvement program goes with them.

Every team starts from zero. Whatever your loop learns about which edits are safe lives in one person’s intuition, so the team next door starts the same trial-and-error from scratch, and so does everyone else running this experiment behind their own walls. Nothing accumulates. Compare how improvement gains decay in general: same disease, prompt-sized.

Serious tooling asks you for a measure

We want to be fair to the research here, because it’s real and moving fast. Open source projects like DSPy exist to do prompt improvement properly, and the newest approach in that world, called GEPA, is impressive work. Its documentation is also upfront about the entry fee: to run it, you implement a metric that scores every attempt, and GEPA improves against that score. The tooling brings the search. You bring the measurement.

That requirement is the tell worth reading: measurement is the expensive ingredient, and no library can generate it for you. For engineering teams with evaluation data, fair enough, they can pay it. For an operations team whose prompts are SOP steps written in plain English, nobody is writing a scoring function per step. The measurement has to come from somewhere else.

Keep every version, gate every change

It comes from the runs. A step that executes inside a workflow generates its own evidence, like how long it took and whether the next person sent the work back. That run history is a measurement nobody had to build, and it flips every failure mode above. Rewrites stop being blind because the evidence sits next to the text. Behavior change becomes checkable against last month’s runs instead of against taste. The old wording stops dying because every version is kept, forever, with one-click rollback, and the babysitter gets replaced by an approval task that comes to the step’s owner.

A mistake we made early on was treating “AI improves the prompt” as one problem. It’s two. Generating a plausible rewrite is nearly free now. Proving the rewrite is safe, and surviving it when it isn’t, is the entire cost, and that’s the half the DIY loop skips. Stern Stella is our answer to the second half, built on the three-tier sort between changes that can touch the output and changes that provably can’t.

An AI that edits its own instructions without measurement is mutating rather than improving, and mutation with no memory of prior versions breeds things you can’t debug. Improvement needs a memory, a measure, and a no. The odd part was discovering that the memory and the measure were sitting in the run history all along, waiting for someone to treat them as the point.