What "Production" Actually Means

There is a phrase I have heard so many times in the last two years that I have stopped being polite about it. The phrase is we've built it, it's working, we'll roll it out next month. It is almost never true. What the team has built is a demo that works in the conditions they happened to test it in. That is not a production system.

The gap between a working demo and a production system is the gap where most AI projects in small businesses go to die. They die quietly. Six months after the launch announcement, the tool is being used by one and a half people, the team has gone back to the old workflow, and nobody quite knows how to say that out loud.

I want to make the gap visible, because once you see it you cannot stop seeing it, and once you build for it deliberately your hit rate triples.

An AI workflow is in production when it passes four tests that almost no demo passes. Each one is mundane. Each one quietly destroys most builds.

A demo works on a Tuesday at 3pm when the builder is watching. A production system works at 2am on Sunday when nobody is watching, the upstream API has changed its rate limit, the model has been silently rolled to a new version, and the user is a tired person on a phone in a different time zone.

The question to ask of any AI build is: what happens at 2am on Sunday? If the answer involves a person checking, the system isn't in production. It's a person doing work, with AI assistance, on a schedule that includes weekends and nights.

Every AI system inside an SMB is built by someone who understands it deeply because they built it. They know which prompts work, which inputs break it, what to do when it loops, where the rough edges are. The system performs well because they are performing well around it.

Now imagine the person who joins next month. They have never seen this tool. They have not been in the meetings. They do not know that the system gets weird when you upload a PDF over twelve pages. They will use the tool the way the documentation suggests. Does it still work?

If the answer requires a tribal apprenticeship — Sarah will show you the tricks — the system isn't in production. It's a personal artefact that is briefly useful while Sarah is in the building.

Every input you have tested is the happy path. The edge cases — the malformed file, the angry customer message, the document in the wrong language, the case the legal team flagged last week — were not in the test set, because nobody thinks of them in advance.

A production system has a defined behaviour for the edge case. It refuses the input cleanly. It flags it for human review. It returns an error the user can act on. It does not silently produce confident nonsense. Most AI builds, when fed an edge case, produce something that looks correct enough to pass review, which is the worst possible failure mode.

This is the test almost nobody designs for, and it has caught me twice. The model you built on will change. The provider will roll out a new version. Behaviour will shift. Outputs will become slightly different. Latency will move. A prompt that worked beautifully will start producing structured JSON in a different shape, or hedging where it didn't hedge before, or refusing requests it used to handle.

If your AI workflow has no test set, no regression check, and no way to compare outputs across model versions, you will discover the update by way of customer complaints. A production system has the scaffolding to notice the regression on the day it happens and to roll back or adapt before the business feels it.

Building this scaffolding is unglamorous. It takes more time than the original build. It is the thing that separates a workflow that survives twenty model updates from one that breaks on the first.

The fix is not to add a production hardening phase at the end of an AI project. The fix is to start, on day one, with the four tests in mind, and to refuse to call anything done until it passes all four.

Practically: have a documented happy path, edge cases, and a test set. Have a runbook for the new joiner. Have an alerting mechanism for the 2am Sunday case. Have a snapshot of outputs that you can re-run against a new model version. These four artefacts, more than the prompt itself, are what makes an AI feature production-grade.

If that sounds like overkill for a small business, I would point out that the alternative is building things that quietly stop working three months in and never get diagnosed. Production isn't a phase. It's a standard, and it's what separates the SMBs whose AI work compounds from the ones whose AI work decays.