Nitesh Gupta

Building Real-Time Voice AI with FastAPI and WebSockets

•9 min read

Architecting low-latency, full-duplex conversational voice agents.

Conversational AI is moving from text interfaces to voice. However, the architecture required for a real-time voice agent is fundamentally different from a standard chat application.

A text-based RAG system might have a latency budget of 3 to 5 seconds. In voice communication, any delay over 800 milliseconds feels unnatural. This article breaks down the architecture required to build a full-duplex voice agent using FastAPI and WebSockets.

The Voice Architecture

A modern voice pipeline consists of several stages that must operate asynchronously and continuously:

  1. Audio Ingestion: Receiving chunks of raw audio from the client.
  2. Voice Activity Detection (VAD): Determining when the user starts and stops speaking.
  3. Speech-to-Text (STT): Transcribing the user's audio (e.g., using Whisper or Deepgram).
  4. LLM Processing: Generating the agent's response.
  5. Text-to-Speech (TTS): Synthesizing the LLM's text back into audio (e.g., using ElevenLabs or OpenAI TTS).
  6. Audio Egestion: Sending audio chunks back to the client.

WebSockets for Full-Duplex Streaming

HTTP request/response cycles are too slow for this pipeline. The connection must be persistent and bidirectional. FastAPI provides excellent, asynchronous WebSocket support out of the box.

from fastapi import FastAPI, WebSocket

app = FastAPI()

@app.websocket("/ws/voice-agent")
async def voice_agent_endpoint(websocket: WebSocket):
    await websocket.accept()
    try:
        # Start background tasks for STT, LLM, and TTS
        while True:
            # Receive audio chunks from the client
            audio_data = await websocket.receive_bytes()
            # Feed into the pipeline
            audio_buffer.append(audio_data)
    except WebSocketDisconnect:
        # Clean up resources
        pass

The WebSocket serves as the transport layer. Client audio arrives as bytes, and synthesized TTS audio is sent back over the same socket.

Voice Activity Detection (VAD)

A naive system simply waits for the user to stop talking for 2 seconds before transcribing. This creates terrible latency.

A robust system uses Voice Activity Detection (like Silero VAD) running continuously. As soon as the VAD detects the end of speech, the accumulated audio buffer is immediately dispatched to the STT engine.

Handling Interruptions

The hardest part of voice AI is handling interruptions (barge-in). If the agent is speaking and the user interrupts, the system must:

  1. Immediately stop playing audio on the client.
  2. Cancel the ongoing TTS generation.
  3. Cancel the ongoing LLM generation.
  4. Truncate the conversation history so the LLM knows exactly where it was interrupted.

This requires tight coordination. The WebSocket loop must listen for inbound audio while simultaneously sending outbound audio. If inbound speech is detected by the VAD while the agent is speaking, a cancellation signal must propagate through the backend pipeline.

The Latency Budget

To achieve sub-second response times, you cannot wait for the LLM to generate the entire response before generating audio.

The pipeline must be entirely streaming:

  1. The LLM streams text tokens.
  2. The backend buffers tokens until a complete sentence or phrase is formed.
  3. The phrase is sent to the TTS engine.
  4. The TTS engine streams audio bytes back.
  5. Audio bytes are streamed over the WebSocket to the client.

Scaling and Backpressure

A single voice session requires holding open a WebSocket, running VAD, and maintaining connections to external STT/LLM/TTS providers.

When scaling with FastAPI and Uvicorn, you must be careful about blocking the event loop. Any synchronous CPU-bound task (like processing audio arrays) must be offloaded to a thread pool or external process. If the event loop blocks, audio chunks pile up, buffers overflow, and the connection experiences severe jitter.

Conclusion

Building real-time voice agents is an exercise in latency optimization and asynchronous engineering. FastAPI's WebSocket capabilities provide a solid foundation, but the true engineering challenge lies in orchestrating the pipeline, managing streaming buffers, and handling the complex state required for natural, interruptible conversation.