all insights
Oct 14, 20267 min readai

Shipping AI features responsibly

Adding an LLM is easy. Making it reliable, fast, and respectful of the user is the real work.

Marcus Chen

Head of engineering

  • LLMs
  • Streaming UX
  • Evals
  • Reliability

A prototype AI demo looks magical in a five-minute meeting. Making it run reliably for fifty thousand customers with latency under a second and zero hallucinations requires rigorous engineering.

in short

Key takeaways

  1. 01

    Evals beat vibes every time

    Never change a prompt without running automated regression tests over hundreds of benchmark cases.

  2. 02

    Streaming is a UX necessity

    A 3-second wait feels like an eternity; a response that streams in 200ms feels instantaneous.

  3. 03

    Graceful fallbacks are mandatory

    When the model times out or hallucinates, the UI must degrade to predictable deterministic tooling.

the details

The deep dive

Moving past prompt tinkering

When most teams begin building with language models, their iteration loop consists of changing a prompt, testing three examples by hand, saying "looks good to me", and deploying. Inevitably, fixing edge case A breaks edge cases B and C in production.

We treat prompt changes with the same reverence as database migrations. Every prompt is backed by an automated eval suite with graded output criteria, semantic similarity assertions, and latency budgets that run on every CI build.

eval-pipeline.ts
// Test suite executed in CI before shipping prompt changes
export async function runPromptEval(promptVersion: string) {
  const benchmark = await loadDataset("customer-inquiries.json");
  const results = await Promise.all(
    benchmark.map((testCase) => evaluatePrompt(promptVersion, testCase))
  );

  const passRate = results.filter((r) => r.score >= 0.95).length / results.length;
  if (passRate < 0.98) {
    throw new Error(`Eval failed: Pass rate ${passRate * 100}% below 98% threshold`);
  }
}

Streaming as psychological reassurance

Human perception of latency is deeply nonlinear. A spinner that rotates for three seconds causes users to wonder if the server crashed. But when tokens stream into the interface within 150 milliseconds, the user reads as the model thinks.

We pair streaming server components with WebSockets or Server-Sent Events, rendering markdown incrementally with smooth opacity fades so the UI doesn't stutter or jump as text flows in.

Don't pretend AI is magic. When it's unsure, tell the user. Humility builds far more trust than false certainty. — Marcus Chen, Head of engineering

next essayThe case for boring tech