How to Build an AI Voice Agent with Python
A practical architecture for a responsive voice assistant using Python, streaming audio and tool calling.
Voice agents are moving from novelty to real business use: booking appointments, answering support calls and qualifying leads around the clock. The hard part is not the AI model — it is making the conversation feel fast and natural. Here is an architecture that works.
The four building blocks
Every voice agent is a loop of four steps:
- Speech-to-text (STT) – turns the caller's audio into text.
- Language model (LLM) – decides what to say or do next.
- Tools – functions the model can call, such as checking a calendar or looking up an order.
- Text-to-speech (TTS) – turns the reply back into audio.
The trick is to stream every step instead of waiting for each one to finish.
Why streaming matters
People notice a pause longer than about one second. If you wait for the full transcript, then the full model reply, then the full audio file, the delay quickly adds up to three or four seconds.
With streaming:
- STT sends partial transcripts while the person is still speaking.
- The LLM starts replying as soon as the person stops.
- TTS starts speaking the first sentence while the rest is still being generated.
A simple Python structure
async def handle_call(audio_stream):
async for text in stt.transcribe(audio_stream):
if text.is_final:
async for sentence in agent.reply(text.value):
await tts.speak(sentence)
Use asyncio throughout so audio input and output never block each other.
Tool calling
The model should not invent answers about bookings or orders. Give it tools instead:
tools = [
{"name": "check_slots", "description": "Find free appointment slots for a date"},
{"name": "book_slot", "description": "Book a slot for the caller"},
]
When the model calls a tool, run the function, return the result and let the model phrase the answer.
Handling interruptions
Real callers interrupt. Detect when the caller starts speaking while the agent is talking (often called barge-in), stop the audio immediately and listen again. This one feature makes an agent feel far more human.
Production checklist
- Log every conversation and tool call for review.
- Set a clear hand-off to a human for anything the agent cannot handle.
- Keep replies short — two sentences is plenty on a call.
- Test with real accents and background noise, not only clean recordings.
Conclusion
A good voice agent is mostly good plumbing: streaming, tools and interruption handling. Get those right and the model can focus on being helpful.
Want a voice agent for your business? Start a project and tell me about your calls.