How model drift happens
A model is a snapshot. It was built from data gathered up to a point in time, and it keeps behaving as though that time is still now. Drift is the gap that opens between the snapshot and reality, and it widens quietly.
It arrives from two directions. The world can change: you add services, rename products, enter a new market, or customers start asking about something that did not exist when the system was set up. Alternatively the system underneath can change: a provider updates the model behind the same name, an integration returns a slightly different field, or a prompt is edited by somebody in a hurry. In both cases nothing breaks. The output still looks fluent and confident, which is precisely why it goes unnoticed.
Why model drift matters
Drift is dangerous in proportion to how much you have stopped watching. A chatbot answering questions about a service you no longer offer, a lead-scoring rule tuned to last year’s enquiry mix, or an ad-copy generator still using a strapline you retired all keep producing tidy output while doing damage.
The commercial version is worse than the technical one. A scoring model that quietly stops separating good leads from bad does not throw an error; the sales team just starts to feel the leads are weaker, months before anyone traces it back. Anything you have automated and stopped reviewing is where drift will find you.
Where model drift goes wrong
The first mistake is assuming a working system stays working. Set-up is treated as a project with an end date, and nobody owns it afterwards.
The second is having no record of what good looked like. Without a saved set of test inputs and the answers you were happy with, you cannot tell drift from imagination, and every argument about whether the tool has got worse becomes a matter of opinion.
The third is pinning the blame on the model when the cause is upstream. Very often the model is fine and something feeding it has changed — a form field renamed, a webhook now sending a different shape of data, a knowledge base nobody has updated since launch.
What to do about it
Treat any AI system as something maintained, not something finished. Keep a small, fixed set of realistic test inputs with the answers you consider correct, and run them on a regular cycle and after every change to the prompt, the model version or the data source. Pin the model version where the provider allows it, so upgrades happen when you choose rather than overnight.
Sample real output regularly rather than only reading complaints, since most drift never generates a complaint. Give one named person the job of reviewing the system each quarter, and write down what changed and when. Building that review into your AI automation work from the start costs far less than rebuilding trust after a year of confident, slightly wrong answers.