Corpus & audio samples

Training Intelligent Voice Assistant Wakeup with Controllable Synthetic Conversations

Marcin Sowański Kacper Leszczyński Kacper Krzywicki Krzysztof Wodnicki

TCL Research Europe · University of Warsaw

Paper - coming soon Code Dataset

The problem

A wake word tells you someone spoke. It does not tell you they were speaking to you.

A conventional wake-up model listens for its own name and hands everything after it to a speech understanding model. That works for a single command and breaks immediately afterwards. In a follow-up, people do not say the name again - they just keep talking, and expect the assistant to know which parts were meant for it.

We define intelligent wakeup as a module that decides, from the whole session and the current utterance, whether the assistant should act at all. Training that decision needs conversations where most speech is not addressed to the assistant, with the moments that are addressed to it marked. No corpus of that shape existed, so we generate one.

Every sample on this page is synthetic, produced by the pipeline below. The one exception is clearly labelled: the real recordings in the naturalness comparison.

How a scene is made

Three stages: dialogue, voices, room

One LLM agent per speaker - including one for the assistant - writes turns against a phase script. A review pass rewrites the draft for naturalness. Each turn is then synthesised separately in its character's own voice and placed at a position in a simulated room, over generated ambience.

The three-stage generation pipeline Scenario configuration and character backstories feed per-speaker LLM agents, whose draft dialogue passes through a review model, then per-turn text-to-speech, then a simulated room that mixes speech with generated ambience into one scene. scenario + phases character backstories review rules STAGE 1 agent per speaker + assistant REVIEW rewrite for naturalness 2nd model STAGE 2 TTS per turn, per character voice per role STAGE 3 simulated room + ambience speaker positions generated ambience scene wav + json

Full scenes

Where the assistant wakes, and where it stays out of it

Each bar below is one conversation, turn by turn, proportional to how long each turn takes. Click any block to jump the audio there. The outlined block is the turn that names the assistant; solid orange is the assistant replying. Everything grey is people talking to each other.

Positive and negative

The same evening, with and without an assistant in the room

For a large share of the corpus, every scenario has a twin in which the assistant is never addressed. Same characters, same evening, same kind of talk — but nothing in the second one should trigger a wake-up. These twins are what a false-accept rate is measured against.

Speakers

Over 150 designed voices

Characters are written as people first - a background, a job, a way of speaking - and each one is given a designed voice that matches. Fifteen of them introduce themselves below. The identifier under each card is the voice used throughout the corpus for that character.

Naturalness

Against real recorded meetings

The fair question about a synthetic corpus is how far it sits from the real thing. Each row pairs one of our generated meetings with a real recorded meeting from NOTSOFAR. Both sides are mono 44.1 kHz and neither is labelled until you ask - listen first, then reveal.

Our scenes here are all assistant-absent variants, so both sides are ordinary meetings between people. Real recordings are from the NOTSOFAR-1 meeting corpus.

Citation

Citing this work

Details will be updated once the preprint is available.

@inproceedings{intelligent_wakeup,
  title     = {Training Intelligent Voice Assistant Wakeup with
               Controllable Synthetic Conversations},
  author    = {Sowa{\'n}ski, Marcin and Leszczy{\'n}ski, Kacper and
               Krzywicki, Kacper and Wodnicki, Krzysztof},
  booktitle = {},
  year      = {}
}