Shipping AI Features That Hold Up in Production
A demo takes an afternoon. A feature your customers can rely on takes more. What changes when an LLM feature goes from prototype to production, including what Canadian privacy law means for the data you send to a model.
The gap between the demo and production
An AI prototype is easy to love: a few prompts, a model API, and it answers questions about your product in an afternoon. Production is different. Real users ask odd questions, some try to break it, the model provider has outages, costs scale with usage, and every answer is one your company has to stand behind. Treat an LLM feature like any other part of the system, with tests, limits and monitoring.
Decide what 'good' means, then test it
Build an evaluation set: real questions your users ask, each with the answer you expect or the criteria a good answer must meet. Include the awkward ones, like questions it should refuse or questions with no answer in your data. Run the set every time you change a prompt, a model or the data, the same way you run unit tests. Without it, every prompt tweak is a guess, and a fix for one question quietly breaks three others.
Ground answers in your own data
For questions about your products, policies or documents, retrieval-augmented generation (RAG) retrieves the relevant passages from your own content and gives them to the model along with the question. Have the model cite where an answer came from, and tell it to say when it doesn't know rather than guess. Keep the knowledge base current, and respect permissions: the assistant shouldn't be able to quote a document the user couldn't open themselves.
Know where the data goes
Every question sent to a hosted model leaves your systems. Under PIPEDA, your company stays responsible for personal information it hands to a provider for processing, and Quebec's Law 25 adds a privacy assessment before personal information leaves the province. Check where your provider processes and stores data, how long it keeps it, and whether it can use it for training. Some services, such as Azure OpenAI, can run in a Canadian region. And strip out personal details the model doesn't need before they reach it.
Plan for the provider to fail
Model APIs time out, hit rate limits and go down, so design for it: set timeouts, retry carefully, and have a fallback, whether that's a second provider, a smaller model or a clear message that hands off to a person. The demo assistant on this site works this way: it asks one provider and, if that fails or takes too long, asks a second one running the same model, so visitors rarely notice.
Put limits around it
Anything that costs money per request needs limits: per-user rate limits, caps on input size and answer length, and a daily ceiling on total spend. Public endpoints also need bot protection. Keep instructions and conversation history on the server so users can't rewrite them, and give the model no more access than the job needs. Prompt injection is much less dangerous when the model can't do anything harmful. The demo on this site, for example, keeps each conversation on the server, limits each visitor's questions per day, and asks the browser to solve a small puzzle before chatting.
Watch it like any other service
Log what goes in and out (with care for privacy), along with response time, cost per request, error rates and which provider answered. Give users a way to flag bad answers, and read a sample of real conversations regularly. Most of the improvements worth making come from what real users actually ask.
Start narrow
The features that hold up start with one well-defined job, such as answering questions from your help centre or pulling fields out of one type of document, with a person in the loop wherever a mistake would be costly. Once that works and you can measure it, widen the scope. You can try the demo assistant on this site to see what a narrow, well-fenced assistant feels like.
Want a second opinion on your setup?
Book a free 30-minute call. Tell me what's slowing your team down, and you'll leave with a clear next step, whether that's the 2-week review or something you can fix yourselves.
Book a free call