Your AI Needs a Flight Recorder, and You Build It on Day One
The handful of records that let you explain any AI output months later, and spot quality drift before a customer does.
When an AI system reaches production, the first hard question is never how it works. It is why it said that. A customer disputes a quote. An auditor asks how a decision was reached. Someone on your team notices the summaries got worse this month and cannot tell you when it started.
At that moment you either hold the record or you do not, and you cannot reconstruct it afterwards. Rerunning the same question today gives you today's answer from today's model, today's prompt and today's data. That is not evidence of anything.
Logging is a day-one decision. Added later, it only covers what happens next.
The six things that produce an answer
An AI response is the product of a small set of inputs. Capture all of them, on every call, from the first request:
- The request — a unique ID, a timestamp, which person or system asked, and the exact text or data submitted.
- The context — every document, record or row the system retrieved and placed in front of the model, identified individually, with the version of each.
- The instructions — the prompt template plus anything assembled at runtime, and the template's version.
- The model — the exact model identifier, not the family name, along with the settings used: temperature, token limits, which tools it could call.
- The output — the raw response before any cleanup, and the processed version if the two differ.
- The consequence — what happened next: shown to a user, emailed to a client, used to approve a claim, and whether a person accepted, edited or rejected it.
Drop the context and the instructions and you are left with half a conversation. You can see what the system answered and never establish why that answer was reasonable.
Versions are the part everyone skips
Three things change underneath an AI system without anyone filing a change request. Someone edits a prompt on a Tuesday afternoon. The vendor updates the model behind the same API name. The knowledge base gets re-indexed and the system starts retrieving different documents for the same question.
Each of those can move quality, and none of them shows up in an output log. Record a version identifier for the prompt, the model and the data index on every call, and the question "what changed between the good week and the bad week" takes minutes to answer. Without them it takes a week of guesswork, usually while someone senior waits.
Human corrections are your best signal
When a person edits an AI draft before sending it, they have told you exactly what was wrong, in the most specific terms available. Store the original alongside the edited version.
Do this and two things follow. The edit rate becomes a quality measure you can chart weekly, and it moves before complaints do. And the corrections accumulate into a set of examples you can use to improve the system, built from your own work rather than bought or invented.
Drift is a comparison, not an alarm
Nothing will notify you that outputs got worse. Degradation is gradual and each individual answer looks plausible.
The practical method is to keep a fixed set of real requests — a hundred is enough, drawn from actual traffic and covering your awkward cases — and run it again every month. Compare the new answers against the stored ones. Where they differ, a person reads both and says which is better. This only works if you kept the inputs, which is the whole argument for logging them.
Pair that with the numbers you already have: edit rate, rejection rate, escalation to a human, the share of answers where the system found no relevant documents. A rise in that last one usually means your data went stale, not that your model did.
Decide retention and redaction first
These logs contain whatever your customers typed, which means they carry the same obligations as any other personal data. Classify fields as they are written, mask or tokenise what does not need to be readable, and store the logs separately from the application so a failure in one does not take the record of it with it.
Set retention deliberately. For decisions with contractual or regulatory weight, keep the record for as long as the exposure lasts, and make that store append-only so nobody can quietly revise history. For routine traffic, shorter is better and cheaper. Put one person's name against the policy.
Storage is the cheap part. The expensive part is retrofitting all of this into a live system while you are already being asked to explain an output you cannot account for. We build these records in from the start, because they are what turns an AI system from something you trust to something you can demonstrate.