Repair or Rebuild: Five Signs an AI System Has Run Its Course

An AI system almost never dies in a way that forces a decision. It keeps answering, keeps scoring, keeps producing a file every morning. Meanwhile the team routes around it, the change requests pile up, and the annual spend grows without anyone choosing to grow it. The end of life arrives as a slow increase in the cost of keeping the thing alive, which is exactly the kind of cost that never appears on an agenda.

So the decision gets postponed. Rebuilding sounds expensive and disruptive; carrying on sounds free. It is not free, and the job of a leader here is to notice the moment when carrying costs more than replacing.

The five signals

These are the ones that hold up across industries and system types. Any single one is a prompt to look closer. Three together means the decision has already been made for you.

• Small changes stopped being small. A pricing rule that took two days to adjust in the first year now takes three weeks and touches four other things. The system has accumulated enough special cases that nobody can predict the blast radius of a change.
• The workarounds have become the process. People export the output and fix it in a spreadsheet before anyone uses it. That spreadsheet now has an owner, a naming convention, and a backup. The real system is the workaround; the official one is a data source.
• You cannot answer basic questions about it. Why did it decline this customer in March? Which version was running when the numbers shifted? If nobody can answer without a week of archaeology, you are carrying regulatory and reputational exposure you have not priced.
• The knowledge lives in one or two people. Every change waits on the same person. They are not hoarding anything; they are simply the only surviving copy of the design decisions. That is a staffing risk disguised as a technical one.
• The constraint moved. The system was built around a limitation that no longer exists — a context window, a latency ceiling, a per-call price, a capability that needed three components and now needs one. Much of what you maintain is scaffolding for a problem that went away.

What is not a signal

A single bad quarter of output quality is usually a data problem, not a design problem. A new model release is not a reason to rebuild anything; if your system is structured well, swapping the model is a configuration change and you should test whether it is. A noisy incident is not evidence either — incidents cluster around visible systems, not fragile ones.

The distinction that matters is between a system that is performing badly and a system that has become expensive to change. The first is fixable. The second is a structural verdict, and no amount of patching reverses it.

Price the carry, not just the rebuild

Rebuild proposals lose because only one side of the ledger gets a number. Put a figure on the other side before you decide.

Add up what the current system consumed over the last twelve months: engineering hours on maintenance and incidents, the vendor and compute bill, the hours people spend on manual correction downstream, and the value of the changes the business asked for and did not get. That last item is usually the largest and is almost never counted. Then look at the same twelve months two years ago. If the line is rising while output is flat, you have your answer, and you have it in the only currency a board argues in.

Rebuild in slices, and keep the data

A rebuild does not mean a two-year programme with a switchover date. The systems that replace well are replaced one capability at a time: stand up the new path for a narrow slice of traffic, run it beside the old one, compare outputs on real cases, move the slice over, repeat. The old system stays up until the last slice leaves. At no point does the business depend on a single cutover weekend.

The one thing worth carrying forward is the data — the labelled examples, the corrections, the record of what the system got wrong and how people fixed it. That accumulated judgement is the genuinely valuable asset. The code around it is replaceable, and at some point should be replaced.

Most organisations wait about a year too long. The cost of that year is rarely visible, which is precisely why it needs to be measured. We built our practice around carrying that weight for clients, but the measurement is yours to do, and it is worth doing before the next budget cycle rather than after.