The Upgrade You Didn't Ask For
Model upgrades can quietly break a working AI system. Here is how to catch the regression before your customers report it.
Your AI system worked on Friday. On Monday it still returns answers, still responds in under two seconds, still passes every health check on the dashboard. And yet the answers are subtly different — longer, hedgier, formatted in a way that breaks the parser downstream, or confidently wrong about a policy it used to get right.
Nothing failed. Your provider shipped a model upgrade.
This is the failure mode that surprises leadership teams most, because it looks nothing like an outage. There is no red light. The system reports itself healthy while the quality of its output has moved underneath you. Your customers notice before your monitoring does.
Why upgrades break things that were working
A newer model is better on average and different in particular. Providers optimize for aggregate benchmarks; you deployed against your own narrow slice of reality. When the model changes, the behaviors you tuned around change too — the phrasing your prompt relied on, the JSON structure your integration expects, the refusal boundary your compliance team signed off on, the length that fit your UI.
Upgrades arrive in three ways. The provider retires a model version and migrates you. You pinned a floating alias that silently resolves to the newest version. Or you deliberately upgraded to cut cost or latency and assumed quality would only improve. All three produce the same result: behavior changed, and no one measured what changed.
Build the thing that tells you
The single control that matters is a fixed set of test cases with known good answers — a golden set. Two hundred real examples drawn from your actual traffic, covering your common cases, your edge cases, and the situations where being wrong is expensive. Each with the answer your business considers correct.
Run that set against the current model and record the results. That recording is your baseline. When a new model appears, run the same set again and compare. The comparison is the whole product: you now have evidence instead of impressions.
Building this takes a few days and needs your operators, not your engineers, to define what "correct" means. Most organizations skip it because it feels like overhead before launch. It is the cheapest insurance in the entire system.
Control when the change reaches you
Pin explicit model versions in every production configuration. A pinned version turns an ambush into a decision you schedule. Then treat each upgrade as a small project:
- Run the golden set against the new version and compare, case by case, against the baseline
- Have the business owner review every case where the answer changed, not just where it got worse — improvements can also break downstream systems
- Send a small share of live traffic to the new version and watch the operational signals: escalation rate, handle time, edits made by staff, thumbs-down clicks
- Keep the old version reachable for a rollback window, and know who has authority to pull the switch
- Record the decision, the date, and the evidence, so the next upgrade starts from a known state
Note what is not on that list: a demo. Watching a new model answer five questions impressively tells you nothing about the twentieth percentile of your traffic, which is where your complaints come from.
Watch the signals your business already generates
Between upgrades, quality drift shows up in numbers you likely already collect. Escalations to a human. Average handling time. How often staff rewrite the system's draft before sending it. Repeat contacts on the same issue within a week. Any of these moving three or four points in a fortnight is worth a look at what changed in the model layer.
Give one person the standing job of reviewing those numbers weekly against the model version in production. Not a committee, not a quarterly review. A named owner and a fifteen-minute check.
Put it in the contract
Ask your provider for written notice before a version is retired, a documented minimum support window for pinned versions, and access to the previous version during migration. Vendors who serve regulated industries generally offer this. Vendors who do not will tell you so, and that answer is useful before you build on them rather than after.
The honest position
You cannot stop your provider from improving their models, and you would not want to. What you can do is make sure the improvement lands on your terms: measured, staged, reversible, and owned by someone.
That is the unglamorous weight of running AI in production rather than piloting it — and it is the part we carry for clients who would rather spend their attention on the business. Either way, build the golden set. Everything else depends on it.