Production checklist for an AI voice agent
A voice agent that demos well and a voice agent that survives real calls are different systems. These are the things that only break once someone who did not build it starts talking — latency, interruption, silence, consent, failure — and what to have in place before launch.
What this is based on: We build and operate our own voice agent: it answers on our scheduling page and joins consultation rooms over LiveKit, running a speech-to-text → language model → text-to-speech pipeline. Every failure mode below is one we hit ourselves, in that order, roughly.
Have a latency budget, not a latency hope
The single thing that decides whether a voice agent feels usable is the gap between a caller finishing a sentence and the agent starting to speak. Past roughly a second, people assume the line is dead and start talking again — which makes the problem worse, because now the agent is answering a question that has been superseded.
That gap is not one number. It is a chain, and every link in it is separately controllable: detecting that the caller stopped, finalising the transcription, the model producing its first token, the speech synthesiser producing its first audio, and the network carrying it. A team that measures only the total has no idea which link to fix.
- Write down a target for each stage before building, and instrument each stage separately from day one. Retrofitting per-stage timing is far harder than adding it early.
- Stream everything that can stream. Waiting for the complete model response before starting speech synthesis adds the entire generation time to the perceived gap, for no benefit.
- Keep the model’s first sentence short by design. The caller hears responsiveness in the first half-second; the rest of the answer can arrive while they are already listening.
- Measure from a real network on a real device, not from a development machine on the same LAN as the server.
- Watch the tail, not the average. A 95th-percentile pause of four seconds is a broken product even if the average looks fine.
Retrieval belongs inside this budget too. A lookup that takes 800 ms is invisible in a chat interface and fatal in a conversation.
Interruption is a feature, and it is harder than it sounds
People interrupt. They interrupt to correct a wrong assumption, to answer before the question finishes, and to stop an agent that has started down the wrong path. An agent that cannot be interrupted is experienced as rude within about two turns, and callers hang up on it.
Handling it properly means more than stopping the audio. When the agent is cut off mid-sentence, the conversation state has to reflect what the caller actually heard — not what the agent intended to say. Otherwise the agent later refers to information it never delivered, and the caller has no idea what it is talking about.
- Stop playback immediately on detected speech, and stop generation too — continuing to generate tokens nobody will hear wastes budget and delays the next turn.
- Truncate the conversation history at what was actually spoken before the interruption, not at the full intended response.
- Tune the barge-in threshold against real recordings. Too sensitive and background noise or a cough silences the agent; too insensitive and it talks over people.
- Decide explicitly what a very short interruption means. "Mm-hm" is acknowledgement and should not stop the agent; "no, wait" should.
Plan for silence, noise and the things people actually say
A demo call has one person, in a quiet room, speaking clearly, saying something the builder anticipated. Production has none of those guarantees, and each missing one is a distinct failure mode with a distinct fix.
- Silence: the caller says nothing. The agent needs a prompt after a set interval, and an end-of-call behaviour after a second one, rather than waiting indefinitely.
- Background speech: a television, an open-plan office, a second person in the room. The agent will transcribe it and try to answer it unless voice activity detection is tuned for this.
- Accents and code-switching. Transcription quality varies by accent, and callers switch languages mid-sentence far more often than test scripts assume.
- Numbers, spellings and identifiers. Postcodes, order references and email addresses are where transcription fails most, and they are exactly the things a business agent needs to get right.
- People talking to the agent as though it were a person: "hang on", "let me get that", "sorry, what?". Each needs a sensible response, and none of them is a question.
For anything an error would be expensive on — a reference number, a spelled-out email — read it back and confirm. That is a design decision, not a model capability.
Decide what the agent may do, not just what it may say
The moment an agent can call a function — book a slot, look up an order, send an email — the risk profile changes. A wrong sentence is an annoyance. A wrong booking is a business problem, and a wrong refund is a bigger one.
The safe pattern is boring and it works: read-only actions can happen freely, and anything that writes, charges, cancels or communicates is confirmed out loud with the caller before it happens, and is separately reversible afterwards.
- Split tools into read and write. Different confirmation rules, different logging, different rate limits.
- Confirm every write action verbally before executing it, using the caller’s own words back to them.
- Make every write idempotent. A retry after a network blip must not create a second booking.
- Log every tool call with its arguments and result, so a disputed call can be reconstructed without the audio.
- Cap what a single call may do. An agent that can issue one refund is a feature; one that can issue forty in a minute is an incident.
Consent and privacy, decided before launch rather than after
Voice is personal data, and a recording is more sensitive than a transcript. The rules differ by jurisdiction and we do not give legal advice — but every deployment has to make the same set of decisions explicitly, and the time to make them is before the first real call, not after a complaint.
- Tell the caller they are speaking to an automated system, at the start, in plain words. This is increasingly a legal requirement and it is good practice regardless.
- Decide whether audio is retained at all. Retaining transcripts and discarding audio is often enough for quality work and is a much smaller exposure.
- Set a retention period and enforce it automatically. A policy that depends on someone remembering to delete is not a retention period.
- Redact identifiers from transcripts before they reach any log, dashboard or analytics tool.
- Have an escalation path to a human, and make it easy to reach. "Talk to a person" should always work.
- Confirm with the client where audio and transcripts are processed and stored, and put it in writing. Do not claim a certification you do not hold.
We treat this as a scoping question with the client rather than a technical default, because the honest answer depends on their sector and their own legal advice.
Design the failure path as carefully as the happy path
A voice agent depends on several external services at once — transport, transcription, the model, synthesis — and in production one of them will be slow or unavailable while a caller is mid-sentence. What happens next is a design decision, and if nobody makes it, the default is silence, which is the worst option.
- A timeout on every external call, shorter than the caller’s patience rather than shorter than the provider’s SLA.
- A spoken holding response when something is slow. "Give me a moment" is enormously better than dead air.
- A second provider configured and tested for the components that support it, with a periodic test that the fallback path actually works — an untested fallback is a fallback that does not exist.
- A defined terminal behaviour: transfer to a human, take a message, or offer a callback. Never simply drop.
- Graceful degradation to text where the surrounding product allows it.
Monitor conversations, not just uptime
Standard monitoring tells you the service is running. It will not tell you the agent has been confidently giving a wrong answer to the same question for a week, and that is the failure that costs money.
What is needed is a small set of conversation-level signals, reviewed by a person often enough to catch drift.
- Per-stage latency percentiles, not averages, tracked over time so a provider slowdown is visible.
- Interruption rate and repeat rate — callers repeating themselves is the clearest early signal that transcription is failing.
- Escalation rate to a human, split by topic. A rising escalation rate in one topic points straight at the gap.
- Call abandonment in the first fifteen seconds, which usually means latency or a bad opening line.
- A sampled human review of real conversations every week. There is no automated substitute for reading what the agent actually said.
- An alert on the failure path being used at all, because a fallback firing regularly means something upstream is broken.
Every one of these can be collected without storing what an individual caller said. Aggregate signals first; keep transcripts only where there is a stated reason and a retention period.
The pre-launch list
Before a voice agent takes calls from people who are not on the project, we want to be able to answer yes to all of these.
- Is there a per-stage latency budget, is it instrumented, and is the 95th percentile within it on a real network?
- Can the agent be interrupted mid-sentence, and does its memory reflect what the caller actually heard?
- Is there defined behaviour for silence, background speech and a caller who says something unrelated?
- Are write actions confirmed verbally, idempotent, logged and reversible?
- Is the caller told they are speaking to an automated system?
- Is there a retention period for audio and transcripts, enforced automatically?
- Does every external dependency have a timeout, a spoken holding response and a tested fallback?
- Is there a working, easily reachable route to a human?
- Are conversation-level signals being collected, and is someone actually reading a sample every week?
- Is there a documented way to disable the agent immediately without a deployment?
The last one gets forgotten most often and matters most. If the agent starts behaving badly on a Friday evening, someone needs a switch, not a release process.
Questions this raises
How fast does a voice agent actually need to respond?
The useful target is that the caller does not notice a gap, which in practice means the agent begins speaking well inside a second of the caller finishing. What matters more than the exact figure is measuring each stage separately — end-of-speech detection, transcription, model, synthesis, network — because a total latency number tells you there is a problem without telling you where it is. Watch the 95th percentile rather than the average.
Why does the agent talk over people, or refuse to stop?
Almost always barge-in tuning. Too sensitive and background noise stops the agent mid-answer; too insensitive and it keeps talking while the caller is trying to correct it. It has to be tuned against recordings of real calls in the environment the agent will actually be used in, and it needs a deliberate decision about whether a short "mm-hm" counts as an interruption.
Should a voice agent be allowed to book, cancel or refund?
It can be, provided write actions are treated differently from read actions: confirmed out loud with the caller before execution, made idempotent so a retry cannot duplicate them, fully logged, reversible, and rate-limited so one call cannot do unlimited damage. Read-only lookups need none of that. The split between the two is the most important design decision in an agent that can act.
Do we have to record calls?
Not necessarily, and often you should not. Retaining transcripts while discarding audio is frequently enough for quality review and is a considerably smaller privacy exposure. Whatever is chosen needs a stated retention period enforced automatically, identifiers redacted before anything reaches a log or dashboard, and a clear disclosure to the caller. Specific legal obligations vary by jurisdiction and sector — that is a question for your own legal advice, not for a development team.
What happens when the model or the speech service goes down mid-call?
Whatever you designed, or silence — and silence is the worst outcome available. Each external call needs a timeout shorter than the caller’s patience, a spoken holding response when something is slow, a tested fallback where a second provider exists, and a defined terminal behaviour such as transferring to a human or taking a message. A fallback path that has never been exercised should be assumed not to work.
How do you tell whether the agent is doing a good job?
Uptime will not tell you. The signals that do are conversation-level: per-stage latency percentiles, how often callers repeat themselves, how often they escalate to a human and on which topics, and how many calls are abandoned in the first fifteen seconds. Alongside those, someone has to read a sample of real conversations every week. There is no automated substitute for that, and it is where the genuinely embarrassing failures get caught.
Can an existing phone system be kept?
Usually, yes — most deployments put the agent behind the existing number rather than replacing the telephony. What matters is where the audio is bridged, what that adds to the latency budget, and whether the transfer-to-human path works through the same system. Those three questions are worth answering during scoping rather than during integration.
The services this connects to
AI Voice Agent Development
What we build, what we do not claim, and how human review sits over the agent’s output.
Read moreAI Video Consultation Agents
The same pipeline with a video surface, used for scheduled consultations rather than calls.
Read moreConversational AI Development
When text is the right channel, and how escalation to a person is designed.
Read moreTalk to the agent we built
Our own consultation agent runs the pipeline described here. Start a session and see how it handles interruption, silence and a handover to a person — then tell us what you are trying to build.