← All posts

Start the answer early: making an AI assistant feel faster without showing the wrong answer

By Kai Khalid, Nairobi · 15 October 2026 · 8 min read

I recently made a chat assistant answer faster, in the sense of reducing AI chatbot latency measured as time to first token, by starting the answer before it knew whether it was allowed to. It worked... a little.

The assistant sits on an insurer's website. Visitors ask about cover, and sometimes want to book a callback. I built the change with Sonnet 5.5 High. This is the longer write-up: the pattern, the safety rails and the honest numbers.

The problem: a slow time to first token

Before the change, a visitor waited a median of 2.02 seconds over six turns before anything appeared. The slowest was 2.88 seconds.

Everything ran in a line. A small model call checked the message for booking signals. A search found the relevant pages. A model call wrote the reply. Nothing showed until the whole reply was finished.

Here's what I noticed. For an ordinary question, the search and the writing get the same inputs whichever way the check comes out. Only booking turns need something different. So most of that wait was work that could have started earlier.

There's a hard limit, though. A booking turn must not show a reply written without the booking state. An early reply shown when it shouldn't be is worse than a slow one.

The pattern: stream the answer alongside the check, gate on the verdict

The check and the answer start together, and a gate waits for the verdict before any words reach the screen. Three moving parts:

• Start both at once. When a message arrives, the check starts. It's an LLM judge running on Gemini 3.5 Flash. At the same moment, a worker thread runs the search and starts writing the answer. The words it produces go into a queue and wait.
• Gate on the verdict. When the check finishes, its verdict decides. If the turn is ordinary, the queued answer is released. If it's a booking turn, the early answer is thrown away unread and a fresh one is written with the booking facts.
• Stream the words. Streaming LLM responses as server-sent events sends the reply to the browser a piece at a time. When it finishes, the page swaps in the final reply with its sources and any callback offer.

The first word now waits for whichever is slower, the check or the answer's first piece, rather than both added together. That's the whole gain.

One smaller fix helped: keeping one connection open between model calls. In a probe, five calls on a fresh connection took 1.12 to 2.27 seconds to the first piece; five on a kept-open connection took 0.72 to 1.78.

Keeping it safe: guardrails that never show a thrown-away answer

The wrong answer must never show. Three things handle that.

Read only the verdict's three fields. The check already returns three fields: extra booking context, a flag to skip the search, and the text it used. The gate reads those and nothing else. If all three say ordinary, the early answer goes out.

The code never reads the visitor's words to make this choice. No phrase matching, no keyword lists. Word rules break on phrasing you didn't expect, and quietly become a second classifier nobody owns.

Test for leaks. On a booking turn, the test deliberately holds the check back until the early answer has already produced a piece, the worst case, then confirms none of those words reached the visitor.

Keep the old route. The whole-reply route is unchanged. It's the fallback if the stream can't be reached, and the reference the stream is tested against. The same message must give the same reply, sources, callback offer and booking status on both routes.

For the record: the live booking check read 14 of 14 reference messages fully right (run once, since each run costs fourteen paid calls), and 815 of 817 tests pass, with the two failures being older guards unrelated to this work.

What happened (honest numbers)

Now the honest bit. Each streamed run measured six turns and recorded the time to the first word (time to first token, TTFT).

Run 1 was slower than the baseline. Run 2 was faster. The first request in run 1 came straight after a container rebuild and took 5.42 seconds. I didn't isolate the cause.

The spacing was uneven too: baseline turns were 40 seconds apart, streamed turns 5 seconds. At the same 5-second spacing the old route's median was 2.28 seconds. So it isn't a clean comparison.

Inside the running app, on ordinary turns:

• First word took 1.14 to 1.31 seconds in-process, and 1.24 to 1.34 seconds from a client. The web layer adds no measurable time.
• The check takes 0.8 to 1.4 seconds per message.
• The search takes about 0.14 to 0.32 seconds, and the first model piece follows about 0.9 to 1.6 seconds after it.
• On a flagged turn the answer is written twice. First word then took about 2.5 to 4 seconds.

So on ordinary turns the first word lands in about 1.1 to 1.4 seconds, against 2.02 seconds for a whole reply before. Different measurement methods, so it shows the direction. It doesn't prove anything yet.

Also not covered: home or mobile connections (the tools ran on the same machine as the app), real visitors' questions (I wrote them), concurrent visitors, and cost under real traffic.

When to use it and when not to

Use it when:

• Your assistant runs a check before it answers and that check is slow enough to feel.
• Most turns are ordinary, so the early answer is usually the one you keep. Ordinary turns cost nothing extra.
• Your check returns a structured verdict you can gate on.

The price is one extra model call on flagged turns. A flagged turn is answered twice, an ordinary turn once. For me that's cheap insurance for never showing the wrong answer.

Watch the flagged share, though. The verdict on the same question varied between runs, so I don't yet know how many real turns get answered twice. Flag a lot, and you pay twice often.

Skip it when:

• The check is already fast. Then there's nothing to overlap.
• You can only gate on free text, not a structured verdict.
• You can't write the leak test. If you can't prove thrown-away words never reach the screen, don't ship it.

One more thing: how often the double answer happens depends on the booking rules, such as whether a product question that mentions premiums counts as a booking turn. That's a process decision for the business, not a code change. It's why I look at builds through people, process, strategy and technology.

What's next: a faster decision model, and the network hop from Nairobi

The weak spot is the check itself. It takes up to 1.4 seconds, and the answer still has to wait for its verdict. I'm aiming for sub-second first words. I'd love to hit milliseconds.

Next, I want to swap the LLM judge for TypeSafe's Jev decision model. TypeSafe says Jev returns structured decisions in 70 to 500 milliseconds, with calibrated confidence scores. If that holds for a booking check, the gate stops being the bottleneck, and a confidence score is a better thing to gate on than a plain yes or no.

The catch: TypeSafe's launch post for Jev says its service is currently based on the US West Coast, and its published timings were run from there. I'm in Nairobi, Kenya. The network hop could eat all my gains. LLM inference in Africa is still mostly far away, which hurts latency from Nairobi. We really need inference in Nairobi, yho!

The short version, with all six diagrams, is on X: my thread on starting the answer early.

I also want to measure from a remote client, with fixed spacing and enough runs to show the spread rather than one median.

Try it on your own assistant

If your assistant checks before it answers, try starting the answer alongside the check. Gate on the verdict, write the leak test first, keep the old route as your reference, and measure with more than six turns.

If you've tried this, or you're running inference close to East Africa, send me your numbers. Message me on LinkedIn or X.