Most teams that adopted large language models in 2023 and 2024 are now on their second or third architecture. The first version was glued to a single provider, paid by the token with no budget guardrails, and shipped without a believable answer to the question "what do we do when the model is wrong?"
The good news is that a sober pattern has emerged.
Treat the model as a dependency, not a feature
The teams that ship reliably build their LLM surface area as a self-contained service with three explicit contracts: a typed input schema, a typed output schema, and an SLA. Behind that boundary you can swap providers, tune prompts, and add retrieval without leaking change into your product code.
We deploy this as a small Node or Python service with a queue for batch work and a synchronous endpoint for foreground requests. Observability is wired in from day one — every prompt, every response, every token count.
Make cost a first-class metric
A single feature that runs unchecked on the most expensive model can quietly burn six figures a quarter. The fix is boring but effective:
- Cap tokens per request at the model layer.
- Route by intent: cheap models for classification and extraction, expensive models for generation.
- Cache aggressively, including semantic caching for paraphrased prompts.
- Surface cost per request in the same dashboard you use for latency.
Plan for being wrong
Hallucinations and refusals are not bugs in the conventional sense. They are part of the operating envelope.
Every LLM feature we ship has at least one of:
- A deterministic fallback (e.g. keyword search if vector search returns nothing useful).
- A human-in-the-loop escalation path with a clear queue and SLA.
- A read-only mode the product gracefully degrades to under budget or outage.
Evaluate continuously
LLM outputs drift. Providers retrain. Your prompts age. The teams that stay out of trouble run a small, automated eval suite on every release and on a daily schedule. The suite is unglamorous — a few hundred labeled examples and a dozen rubrics — and it has saved us from shipping regressions more times than we can count.
The takeaway: treat the model like a third-party payment processor. Wrap it. Measure it. Have a plan for when it fails. Do that, and LLMs become a healthy part of your roadmap instead of a permanent fire.