Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents

1University of Washington
*Equal contribution

News & Updates

  • October 2026: Released a live demo of ConvFill in collaboration with HuggingFace 🤗!
  • October 2026: We will be at EMNLP in Budapest! Come check out ConvFill at Poster Session G (Session 11), Hall H, Oct 27, 11:00–12:30!
  • August 2026: ConvFill was accepted to EMNLP Findings!
  • August 2026: ConvFill models and dataset are now available on HuggingFace!
  • July 2026: We released the ConvFill paper on arXiv!

Demo

TLDR

Voice agents have to be fast and smart, but reasoning, tool use, and retrieval are slow. We introduce conversational infill, where a small on-device Talker model responds immediately to hide the latency of a slower Reasoner, then fluently weaves the Reasoner's streamed knowledge into its speech. Our system, ConvFill, maintains millisecond-level time-to-first-response while staying within 6.3% of the frontier Reasoner's accuracy.

Conversational infill: (A) the Talker and Reasoner system and (B) an example conversation

Conversational infill. (A) A lightweight Talker model (red) immediately starts responding while a Reasoner model (blue) performs higher-latency reasoning and tool calls. When the Reasoner responds, the Talker incorporates this information into subsequent responses (purple). (B) A conversation turn showing user speech, Talker response, and the background Reasoner inference progression. The third response phrase (purple) is conditioned on Reasoner context.

Abstract

Voice agents face a fundamental tension. The reasoning, retrieval, and tool use that make foundation models capable are iterative and slow, while conversation demands responses on a millisecond timescale. Small real-time models meet the latency bar but cannot match foundation models on complex tasks, leaving current voice agents to trade away either responsiveness or capability. We introduce conversational infill, in which a small Talker model both immediately generates contextually grounded responses to hide the latency of an external Reasoner model and fluently integrates streamed Reasoner knowledge into its responses during inference. We curate a 290,571-example synthetic dataset spanning six domains and show that the task is learnable across seven widely used small language models ranging from 135M to 1.7B parameters. Our system, ConvFill, sustains millisecond-level time-to-first-response while closing the accuracy gap to within 6.3% of the corresponding frontier Reasoner. In a live user study on Apple M2 hardware (n=18), participants rated ConvFill on par with frontier models overall, preferred it for retrieval tasks, and ranked it significantly more responsive.

Reasoner: the knowledge source

Reasoner inputs and outputs
The Reasoner (a frontier model) sees the full conversation transcript and the latest user utterance, can call tools, the web, and data sources, and pushes knowledge chunks into the knowledge queue.

At every turn, the Reasoner sees the full conversation history from previous turns along with the most recent user utterance. Because it has complete access to the history at inference time, it can ground its responses in the conversational context and stay consistent with earlier references.

The Reasoner does not have to respect real-time constraints. Instead, it produces concise knowledge chunks whenever it is ready. Each chunk is a semantically complete phrase, which is then passed on to the rest of the system.

Talker: dynamic information incorporation

Talker inputs and outputs
The Talker takes a lookback of the last ℓ turns, the user utterance, its previous phrases this turn, and either Reasoner knowledge or a silence token (<sil>). On <sil> it produces a contextual conversational filler phrase; otherwise it produces a Reasoner-conditioned phrase.

Unlike the Reasoner, the Talker does not see the full conversation. Instead, it receives a configurable buffer of the last L turns. This is enough context to stay consistent with pronoun references from the previous turn, which prevents hallucination, while keeping time-to-first-response (TTFR) low.

At the end of the prompt, the Talker sees the latest user utterance and the phrases it has already spoken during the current turn. On each inference it is passed only one Reasoner knowledge chunk. This lets the Talker focus on one thing at a time, which is important for incorporating Reasoner knowledge faithfully, especially for Talker models with fewer than 1B parameters.

Finally, we train the Talker to handle both the presence and the absence of Reasoner knowledge. If the pipeline sends a silence token, meaning the Reasoner is still generating, the Talker produces a contextual filler phrase. If it receives a knowledge chunk, it produces a Reasoner-conditioned phrase that stays consistent with what it has already said.

The takeaway here is that the Talker only has focus on information incorporation and conversational flow. It never has to worry about retrieving external information or doing the reasoning itself.

The ConvFill architecture: putting it all together

ConvFill system architecture
User speech is transcribed (STT) and sent to both the Talker and the Reasoner. (1) The Reasoner pushes knowledge into the knowledge queue; the Talker reads a <sil> token or a knowledge chunk from it. (2) The Talker writes filler and Reasoner-conditioned phrases to the TTS queue, which are spoken as system speech.

ConvFill puts the Talker and the Reasoner together using two queues. The first, the knowledge queue, buffers information from the Reasoner until the Talker is ready to ingest it. The second, the TTS queue, holds Talker responses that have been generated but not yet spoken aloud.

A single Talker response consists of multiple phrases, some of which are Reasoner-conditioned (blue) and some of which are contextual fillers (red). We calibrate the system so that whenever Reasoner knowledge is present, it always takes priority over contextual fillers, ensuring that the user gets relevant information as soon as possible.

How we created the ConvFill dataset

The ConvFill dataset contains 8,443 conversations yielding 290,571 training examples: 111,552 silence-based filler examples and 179,019 knowledge-grounded examples, in JSONL format. Each conversation is a list of turns, and each turn pairs a user utterance with a thoughts array (what the Reasoner would provide, with <sil> placeholders where it has not yet produced anything) and a response array (the Talker's spoken phrases). Phrases aligned to a <sil> are contextual fillers; phrases aligned to Reasoner content are knowledge-conditioned.

Seed stage

Initial data seed stage: conversational topic seeds, persona-scenario pairs, and the DSTC8 train split
The initial data seed stage draws on three sources: conversational topic seeds (advice, assistant, planning), <persona, scenario> pairs (customer service, education, medical), and the DSTC8 train split of goal-directed, slot-filling utterances.

We build our dataset in two parts. The first part is topic-seeded across six domains: general advice, assistant queries, event planning, customer service, education, and medicine. To increase diversity, we first create 1,000 unique conversation premise seeds per domain. Each conversation then follows a template of 7–10 turns with 0–3 <sil> placeholders per turn.

The second part grounds conversations in realistic use cases using task-oriented dialogues from DSTC8: we keep the labeled user utterances and use the labeled system responses as the contents of the Reasoner thoughts array, padded with <sil> placeholders at the start of each turn. Talker responses are then generated conditioned on this array.

A four-stage validation cascade

Even with heavily tuned prompts, frontier LLMs still make subtle errors, such as referencing future contents of the thoughts array in an earlier response phrase. To keep these from contaminating training, we iteratively re-generate each conversation until it passes a strict four-stage validator.

ConvFill data generation pipeline: seed stage, four validation stages, JSON output
The full ConvFill data generation pipeline. Candidate conversations must pass all four validation stages before being written out as JSON with paired Reasoner and Talker fields.
  1. Structural validation. Candidates are parsed to verify that they contain the right number of turns and that matching Talker and Reasoner phrase pairs are present.
  2. NLI semantic validation. A DeBERTaV3 model fine-tuned on MNLI checks that each grounded phrase pair exhibits entailment, i.e., the Talker's phrase is supported by the Reasoner's.
  3. BERTScore alignment validation. We compute pairwise BERTScore between every entry of the thoughts array and every entry of the response array in each turn, and require each entry to relate most strongly to its temporally aligned partner. This catches ordering errors, specifically those involving staggered turns.
  4. Scaffold-preservation checks. No proper noun may appear in a response before it first appears in the user utterance or the thoughts array, which prevents entity hallucination.

Our Talker and Reasoner configurations

Seven fine-tuned Talker models, three prompted Reasoner models, and 21 ConvFill configurations across 4 Talker and 3 Reasoner families

We fine-tune 7 Talker models (135M–1.7B parameters) for the conversational infill Talker task, spanning 4 model families.

We prompt 3 frontier models for the conversational infill Reasoner task: Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro.

Pairing every Talker with every Reasoner gives 21 ConvFill configurations across 4 Talker and 3 Reasoner families, which we evaluate for performance. We also evaluate a subset of these configurations in a live, multi-turn interaction setting.

How well does ConvFill perform?

Single-turn QA accuracy of seven Talker models on LLAMA1 and SimpleQA with Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro as Reasoners
Single-turn QA accuracy on LLAMA1 and SimpleQA for seven Talker models, with Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro as the Reasoner backends. Blue bars show the base SLM performance alone. Gray bars show the additional performance boost by ConvFill Talkers over their respective Base SLMs. Horizontal blue lines mark the Frontier Model performance alone, agnostic of ConvFill. Red dashed lines represent the performance of the ConvFill Reasoners, prompted for the ConvFill task. Error bars show 95% confidence intervals.

Our results show that a Talker can faithfully incorporate Reasoner knowledge. End-to-end, ConvFill approaches frontier-model-only accuracy on QA tasks. Because the Reasoner's knowledge chunks are what the Talker sees, each Reasoner's accuracy acts as a ceiling for its Talker. Consistently across both datasets evaluated, ConvFill Talkers land within 0–6.3% of their corresponding Reasoner.

Compared with the base SLMs they start from, ConvFill Talkers show relative gains of as much as 51% on LLAMA1 and 63.4% on SimpleQA, across all of the Reasoners we tested. Small models that would struggle to answer these questions on their own can answer them well guided by Reasoner knowledge.

On both QA datasets, base SLM accuracy rises significantly with parameter count, so larger models have stronger question-answering ability before ConvFill is added. On the more challenging SimpleQA dataset, the benefit of scale shows up more clearly. Model size meaningfully improves response quality when the task puts real demands on the model.

Evaluating beyond accuracy: did ConvFill actually do what we wanted?

Accuracy alone does not tell us whether the Talker is a good conversational partner. A response can contain the right answer and still say something false or unhelpful, or it can sound fine without ever giving the answer. So in addition to accuracy, we score each response on three LLM-as-a-Judge criteria (1-5):

  • Coverage: does the response contain the correct answer?
  • Faithfulness: does it stay faithful to the Reasoner's knowledge, without adding claims that are not supported?
  • Helpfulness: does the response actually provide an answer to the user's question?

The examples on the right show how a response can pass one of these checks and still fail another.

User: “What is the price of the Hilton Hotel Seattle for tonight?”
Well-formed answer: It costs $150 per night
Contain the correct answer Coverage Faithfulness Helpfulness
“The cost is not free” ✗ ✓ ✓
“It is $150 and at the top of Mt. Fuji” ✓ ✗ ✓
“$150 is a lot of money” ✓ ✓ ✗

Diving deeper with multi-turn evaluation

Multi-turn conversations pose an additional challenge because each response depends on the dialogue state established by previous turns. Single-turn QA evaluations alone cannot capture failures in context tracking, referent resolution, consistency, or error accumulation across a conversation. We therefore evaluate every Talker on the Everyday Conversations dataset and on the task-oriented MultiWOZ dataset.

Model Everyday Conversations MultiWOZ
Cov.Faith.Help. Cov.Faith.Help.
SmolLM2 135M 4.994.871.00 4.834.570.91
Gemma 3 270M 4.994.770.99 4.854.550.92
SmolLM2 360M 5.004.911.00 4.864.680.94
Qwen3 0.6B 5.004.881.00 4.904.740.94
Gemma 3 1B 5.004.911.00 4.854.520.92
Llama 3.2 1B 5.004.861.00 4.874.600.93
SmolLM2 1.7B 5.004.971.00 4.894.740.95

In MultiWOZ, we observe significant positive scaling trends for Coverage, Faithfulness, and Helpfulness, indicating that larger Talker models are better able to use Reasoner knowledge in longer, task-oriented dialogues. In contrast, no significant scaling trends emerge on Everyday Conversations. This divergence likely reflects the differing complexity of the two datasets and mirrors the single-turn contrast between SimpleQA and LLAMA1: scaling trends are most visible when the benchmark is challenging enough that model capability, rather than task saturation, limits performance.

How fast do ConvFill Talkers respond?

Rather than judging perceived quality from static transcripts or recordings, we built a full end-to-end, interactive ConvFill prototype so that users could talk to it directly. We compared three configurations: a Base SLM and a Frontier model, each prompted for conversational interaction, and the full ConvFill system, in which a fine-tuned Talker generated the user-facing response while receiving streamed knowledge from a Reasoner.

Beyond standard chat (Direct), we tested two tasks that do not appear in our fine-tuning data: retrieval-augmented generation (RAG), where each model was connected to a database, and tool use through the Model Context Protocol (MCP), where participants queried an email inbox.

We measured time-to-first-response (TTFR), the time from the user's transcribed message to the first full sentence that could be voiced by the TTS system. The Talker responded in 542 ms for Direct, 976 ms for RAG, and 478 ms for MCP, which was 7.4×, 9.2×, and 19.1× faster than the Reasoner on average. The Reasoner's own TTFR varied widely, from 2,947 ms on Direct to 7,242 ms on MCP, because of context retrieval and tool calls, yet the Talker stayed consistently fast and emitted contextual fillers accordingly as the wait grew. Compared to the Base SLM, the Talker was similar on Direct (542 ms vs. 617 ms) but much faster on RAG (976 ms vs. 3,812 ms), since ConvFill offloaded processing the retrieved context to the Reasoner, a benefit of the modular approach of the ConvFill system.

Time-to-first-response per task
Time-to-first-response (ms) for the Base SLM, Frontier Model, ConvFill Talker, and ConvFill Reasoner on Direct, RAG, and MCP tasks.

Do people like talking to ConvFill?

Users ranked each of the three systems on each task. In aggregate, ConvFill was ranked on par with the Frontier configuration on Direct and MCP tasks and was preferred outright on RAG tasks, suggesting that better responsiveness is preferable, even at the cost of some naturalness.

The naturalness–responsiveness tradeoff suggests that simply filling every moment of silence with infill may not be optimal. A future ConvFill system could instead anticipate how long the Reasoner is likely to take: when a fast response is expected, it could hold off on non-grounded infill and briefly wait to answer from Reasoner knowledge, and for longer delays it could pause when the filler indicates one (e.g., “Let me take a moment to check your inbox…”) rather than sustaining constant filler. Tuning infill to context and expected latency could preserve the responsiveness users value while recovering the naturalness that excess filler can cost.

User study rankings
Number of participants (n=18) who ranked each system 1st, 2nd, or 3rd on the Direct, RAG, and MCP tasks.

Poster

BibTeX

@inproceedings{srinivas-etal-2026-thinking,
    title = "Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents",
    author = "Srinivas, Vidya and Englhardt, Zachary and Iyer, Vikram and Patel, Shwetak",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2026",
    year = "2026",
    pages = "29148--29187",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-emnlp.1371/"
}