Shipping AI features responsibly
Adding an LLM is easy. Making it reliable, fast, and respectful of the user is the real work.
- LLMs
- Streaming UX
- Evals
- Reliability
A prototype AI demo looks magical in a five-minute meeting. Making it run reliably for fifty thousand customers with latency under a second and zero hallucinations requires rigorous engineering.
in short
Key takeaways
- 01
Evals beat vibes every time
Never change a prompt without running automated regression tests over hundreds of benchmark cases.
- 02
Streaming is a UX necessity
A 3-second wait feels like an eternity; a response that streams in 200ms feels instantaneous.
- 03
Graceful fallbacks are mandatory
When the model times out or hallucinates, the UI must degrade to predictable deterministic tooling.
the details
The deep dive
Moving past prompt tinkering
When most teams begin building with language models, their iteration loop consists of changing a prompt, testing three examples by hand, saying "looks good to me", and deploying. Inevitably, fixing edge case A breaks edge cases B and C in production.
We treat prompt changes with the same reverence as database migrations. Every prompt is backed by an automated eval suite with graded output criteria, semantic similarity assertions, and latency budgets that run on every CI build.
// Test suite executed in CI before shipping prompt changes
export async function runPromptEval(promptVersion: string) {
const benchmark = await loadDataset("customer-inquiries.json");
const results = await Promise.all(
benchmark.map((testCase) => evaluatePrompt(promptVersion, testCase))
);
const passRate = results.filter((r) => r.score >= 0.95).length / results.length;
if (passRate < 0.98) {
throw new Error(`Eval failed: Pass rate ${passRate * 100}% below 98% threshold`);
}
}Streaming as psychological reassurance
Human perception of latency is deeply nonlinear. A spinner that rotates for three seconds causes users to wonder if the server crashed. But when tokens stream into the interface within 150 milliseconds, the user reads as the model thinks.
We pair streaming server components with WebSockets or Server-Sent Events, rendering markdown incrementally with smooth opacity fades so the UI doesn't stutter or jump as text flows in.
Don't pretend AI is magic. When it's unsure, tell the user. Humility builds far more trust than false certainty. ā Marcus Chen, Head of engineering