Rather than judging perceived quality from static transcripts or recordings, we built a full end-to-end, interactive ConvFill prototype so that users could talk to it directly. We compared three configurations: a Base SLM and a Frontier model, each prompted for conversational interaction, and the full ConvFill system, in which a fine-tuned Talker generated the user-facing response while receiving streamed knowledge from a Reasoner.
Beyond standard chat (Direct), we tested two tasks that do not appear in our fine-tuning data: retrieval-augmented generation (RAG), where each model was connected to a database, and tool use through the Model Context Protocol (MCP), where participants queried an email inbox.
We measured time-to-first-response (TTFR), the time from the user's transcribed message to the first full sentence that could be voiced by the TTS system. The Talker responded in 542 ms for Direct, 976 ms for RAG, and 478 ms for MCP, which was 7.4×, 9.2×, and 19.1× faster than the Reasoner on average. The Reasoner's own TTFR varied widely, from 2,947 ms on Direct to 7,242 ms on MCP, because of context retrieval and tool calls, yet the Talker stayed consistently fast and emitted contextual fillers accordingly as the wait grew. Compared to the Base SLM, the Talker was similar on Direct (542 ms vs. 617 ms) but much faster on RAG (976 ms vs. 3,812 ms), since ConvFill offloaded processing the retrieved context to the Reasoner, a benefit of the modular approach of the ConvFill system.