A voice agent that answers in under a second

The problem I started from

Four things make voice agents hard, and most examples solve only one of them. The first is latency, because the conversation dies past about a second. The second is orchestration, since three separate AI services have to run as one stream. The third is extensibility: the agent needs to call business logic, and most examples cannot. The fourth is visibility. When a response is slow, nothing tells you which stage was slow.

So the goal was one working example that handled all four, with the code and the deployment guide someone else could pick up.

---

What I decided

Pipecat over a custom pipeline. I evaluated both on day one. Pipecat's stages stream, so audio never waits for a full utterance before moving on, and that wait would be most of the latency budget.

GPT-4o-mini, chosen for speed. A larger model answered better but more slowly, and in voice, slower loses.

Metrics collected outside the pipeline. Measuring each stage's time to first byte, or TTFB, must not add to it. So collection runs beside the pipeline, not in it, and the React panel redraws as little of the page as it can.

Tools registered by schema. Each function the agent can call is declared as a typed schema and runs async, so adding a tool cannot block the audio stream or break the pipeline. It ships with one example tool, a weather lookup, because the pattern is the point.

Context managed explicitly. Conversation history is collected turn by turn, with a fallback if the context window overflows. More context makes the conversation better but adds latency, so I set that tradeoff on purpose, not by default.

---

What exists

---

What I would do differently

More than one tool. The weather lookup proves the pattern, but a booking or a lookup against a real system would prove it is useful.