Beyond the Benchmark

Abstract

Qwen3-TTS reports state-of-the-art zero-shot voice cloning, but its technical report leaves open how the model should be conditioned, how it behaves outside read speech, and how it fails.

We characterise its two cloning interfaces, in-context learning (ICL) and X-Vector, against Chatterbox, XTTS-v2 and F5-TTS on read, meeting, emotional and air-traffic-control speech. We use eighteen objective measures, an LLM judge, a spoofing countermeasure and a listening test. Neither interface dominates: ICL clones timbre more closely from correctly transcribed references, X-Vector clones timing, and ICL degrades sharply when a short reference transcript is wrong. Domain explains four times more variance than system choice. Qwen3-TTS leads on intelligibility and pitch register but trails on prosody, and the objective measures rank systems differently from listeners. An off-the-shelf countermeasure flags its clones no more often than genuine recordings, and on ATC radio it fails for every system.

Listen like the annotators

Each round is a two-alternative forced choice. One clip is the real AMI recording. The other is a clone of the same words. Play both, then pick the one that sounds real.

This is the protocol behind the human fooling rates in the paper. Listeners replayed freely and named four kinds of giveaway: room and channel sound, breathing and mic noise, speaking-rate variation, and flatter emotion. See whether those cues are enough for you.

Could not load data/catalog.json. Serve the folder over HTTP rather than opening the file directly.

What we measured

Four English domains, each subsampled to one hour: LibriSpeech (clean read speech), AMI (far-field spontaneous meetings), IEMOCAP (acted emotional dialogue) and ATCO2 (noisy, jargon-dense air-traffic-control radio). Every system gets the same reference clips and public checkpoints, with no fine-tuning.

Qwen3-TTS (Qwen3-TTS-12Hz-1.7B-Base) exposes two cloning interfaces. ICL, the default, receives the reference audio and its transcript and continues in that voice. X-Vector conditions only on a speaker embedding and needs no transcript. We also vary pairing. In the aligned condition the model hears and speaks the same utterance; in the shifted condition — the one that matches deployment — the voice comes from utterance i+1 and the text from i, so prosody cannot be copied. Results below use the shifted condition unless stated otherwise.

One utterance, six voices

Pick an AMI utterance and listen across the row. Same words. Different generators.

ICL should hug the timbre of a clean reference. X-Vector should keep closer timing. The baselines are here so the gap is audible, not just tabulated.

How often do clones fool a listener?

Qwen3-TTS is the hardest of the five systems to tell from real speech, and it remains detectable. On AMI, listeners pick its clones as real in 35.6% (X-Vector) and 27.0% (ICL) of rounds, against at most 9.0% for the baselines.

They reproduce the voice, but not the sounds around it, nor the rate variation of natural speech.

Human rates are AMI only. Gemini 3.5 Flash ran the same 100-pair protocol on all four corpora, twice, once in each order. The first ordering is reported; the second is a robustness check. The judge is stricter than people on Qwen3 and much less stable.

Human fooling rate on AMI

Table 1 in the paper. Shifted condition, 100 pairs per system. Gemini rates on AMI are 12% / 11% / 11% / 3% / 5%.

On LibriSpeech the judge is fooled most by XTTS-v2 (26%). Outside read speech that advantage disappears: XTTS-v2 collapses on ATCO2 and IEMOCAP and hallucinates on 18.5% of its samples. Chatterbox is the most robust alternative, close to Qwen3 on the objective score, yet it fools our listeners only 9% of the time.

ICL or X-Vector?

Humans on AMI are fooled more by X-Vector (35.6% vs 27.0%). The LLM judge prefers ICL on all four corpora. The objective profiles explain the split.

ICL clones timbre better from clean references (speaker score 80.2 vs 61.5 on LibriSpeech, 52.2 vs 43.6 on AMI). X-Vector produces closer prosody on every corpus, by 2.6 to 11.3 points, and matches or beats ICL on speaker similarity for noisy ATCO2 and short IEMOCAP references. Overall scores are nearly tied: 33.4 vs 34.3.

X-Vector is also safer. ICL hallucinates 1.6 times as often (4.1% vs 2.5%) because it must reconcile the reference transcript with the audio.

Domain decides more than the model

In a two-way decomposition of the overall score, corpus explains 63% of the variance and system only 15%. LibriSpeech is easiest (mean S = 43.4). ATCO2 is hardest (18.5): intelligibility collapses for every system and the judge is fooled in only 3.8% of rounds. IEMOCAP has the lowest F0 and prosody scores, because expressive pitch and timing are hard to transfer from a short reference.

A short in-domain validation is worth more than switching models.

Four dimension scores

Shifted condition. Each panel is scaled to its own best score, not to 100. In a corpus, the five stacked bars are ICL, X-Vector, Chatterbox, XTTS-v2 and F5-TTS. Hover a bar for the number. Averages hide the swing: F5-TTS leads speaker on AMI and is last on LibriSpeech.

• Qwen3 ICL
• Qwen3 X-Vector
• Chatterbox
• XTTS-v2
• F5-TTS

The clearest ICL failure is a wrong short transcript. Corrupt one middle word and the hallucination rate on IEMOCAP jumps from 0.8% to 21.2%, on ATCO2 from 1.6% to 15.2%. Long AMI and LibriSpeech references stay below 1.5%. The distractor leaks into the synthesis in 7.0% of utterances, almost always as one of the first words — an attention leak before the model commits to the target.

Inject an error, then generate

ICL conditions on audio_ref plus text_ref, then speaks text_target. The experiment keeps the audio and the target fixed and touches one middle word of the reference transcript: a three-letter ending such as -ing (PC) or a static distractor (WC).

Qwen3 leads on intelligibility and pitch register. Prosody is the weakness: Chatterbox and XTTS-v2 stay closer to real timing on three corpora. F0 is the weakest dimension for almost every system. Population pitch matches; the specific speaker often does not.

Machines do not catch what people miss

We ran the released AASIST checkpoint, uncalibrated, on the same shifted pairs. A system is detectable only where the spoof rate clearly exceeds the false-alarm rate on genuine audio.

By that rule Qwen3 is detectable in one of eight cells: ICL on LibriSpeech (39% vs 19% FA). In the other seven, the detector flags real recordings at least as often as clones. X-Vector on AMI is the extreme case: caught on one clip in a hundred, while a sixth of the genuine meetings are rejected.

Detectability does not track perceived quality. XTTS-v2 is the only system the detector separates convincingly (86–91%) and among the least convincing for listeners (6% on AMI). F5-TTS fools people 1% of the time and is still only weakly detected.

On ATC radio the failure is complete and system-independent.

AASIST flags 99% of genuine ATCO2 recordings as spoof — more often than the clones of any system. The decision is driven by the narrow-band VHF channel, not by the clone. Off-the-shelf anti-spoofing offers no protection in the domain where a spoofed transmission matters most.

Automatic scores do not replace a listener either. Averaged over corpora, S ranks systems as humans do on AMI. On AMI alone it puts XTTS-v2 first (40.9) even though that system fools people in only 6% of rounds. Speaker similarity even anti-correlates with fooling (−0.70): F5-TTS has the most similar voice and the least convincing clone. Gemini agrees with humans on easy baselines and misses most of the Qwen3 clips that fooled people. Two presentation orders disagree on up to 43% of pairs.

Takeaways

Neither cloning mode dominates. Use ICL for clean, verified transcripts longer than about ten words. Use X-Vector for short, noisy, or untranscribed references, and wherever timing matters more than timbre.

Domain drives quality far more than the choice of system. Chatterbox is a robust second. XTTS-v2 should stay on read speech. Qwen3 leads on intelligibility and pitch register and is the hardest of the five to tell from real speech, yet it still lags on prosody and per-speaker pitch.

No proxy settles the choice. Neither the objective score nor an LLM judge reproduces the listener ranking. A small forced-choice test in the target domain is still the only arbiter. Disclosure is a more dependable safeguard than post-hoc detection: an off-the-shelf countermeasure flags Qwen3 clones no more often than genuine recordings outside read speech, and is unusable on ATC radio at any threshold.

Future work should target ATC, where a cloned controller voice on a live frequency is a safety risk that no detector we tested can flag.

Hear the gap on ATCO2

Same landing clearance, two Qwen3-TTS ICL pairings. Aligned clones from this utterance, so timing can be copied. Shifted takes the voice from the next clip and the text from this one, which is closer to deployment. The channel noise and radio beeps sit only on the genuine recording — the cues listeners named, and the ones a studio-trained detector does not use.