The “Poor Yorick” Challenge: How to Make Local Speech Synthesis Transition to Whispering, Pause, and Shift Emotional Registers / Habr

How to read this chart: horizontally, all three recordings are aligned by the words in the monologue’s four phrases (so that the same words appear directly above one another, even though the models read the text at different speeds—from 21 to 44 seconds).The vertical axis plots the volume envelope and emotional intensity: at the bottom (the turquoise zone, −12…−18 dB) — a quiet, muffled whisper and an exhalation, in the middle (0 dB) — ordinary narrator’s speech, at the top (the red zone, +6…+9 dB) — a loud emotional outburst and anguish.

Test Setup: Why Specifically the Monologue with Yorick’s Skull?

To avoid testing models on random everyday phrases, I chose a classic passage from Act V of “Hamlet” as a common stress test. Over four sentences, the delivery and functioning of the vocal apparatus need to be completely changed four times. First comes warm nostalgia with a soft smile (“Alas, poor Yorick! I knew him, Horatio: a man of infinite wit, a most marvelous contriver; he carried me on his back a thousand times”). Then comes an abrupt change of tone, physical revulsion, and a spasm (“And now—how disgusting this is to me! My throat tightens when I look at it”). After that, the voice must drop into a quiet, muffled theatrical whisper right at the microphone (“These were the lips I kissed, I don’t know how many times”). And finally, after a heavy exhalation, comes a series of tragic questions with mounting intensity (“Where are your jokes now? Your tricks? Your songs? Your flashes of merriment that made the entire company at the table burst into laughter?”).

I am a teacher and researcher, and this task itself grew out of developing an interactive training course as an alternative to the usual recording of lectures on the university campus. To keep students’ attention and maintain their immersion in the subject even in distance learning, we wanted to move away from monotonous, presenter-style reading to a lively, conversational, and theatrical delivery.The pipeline for this is built in two stages: first, the base TTS model generates speech with the required rhythm, breathing, and intonation pattern, and then, if necessary, the track is passed through RVC v2 (Retrieval-based Voice Conversion, 40 kHz—a neural network for replacing voice timbre from an audio source), transferring the finished performance to a single target voice.

Part 1. Why Compact Models and Manual Audio Refinement Were Not Enough

Approach 1. Base Generation with Qwen3-TTS 1.7B

As a starting point, I took a model fine-tuned for Russian based on siriusfreak/qwen3-tts-12hz-1.7b-ru-stress-cf, which understands U+0301 stress marks (остроу́мный, вы́думщик, засто́лица пока́тывалась со́ смеху). The input was a clean studio reference lasting 9.1 seconds.

The output was a 21.0-second file (24 kHz) with phonetically clean, intelligible speech, without an accent or digital artifacts. Emotionally, however, the result was completely flat: the model delivered the line “My throat tightens when I look at him” and the whispering scene in the same even, narrator-like tone as the first line (on the graph, this is the gray dashed line, which runs almost horizontally like a shelf across the entire text).

Approach 2. Splitting into Phrases and Manually Adjusting Voice Pitch in Praat

The next idea was that, since the model does not change intonation within a long passage, the monologue can be split into individual phrases, each generated with its own reference (calm, grim, emotional), while the intonation is fine-tuned on top through a script using the parselmouth library. This is a Python wrapper around Praat, the classic program for phonetic analysis and speech modification.Using the TD-PSOLA algorithm (pitch shifting without changing speech rate), I tried to forcibly lower the fundamental frequency of the voice (F0, the base pitch frequency in hertz) on a grim remark, add slight trembling, and raise the pitch on the final questions.

In practice, this approach ran into two problems:

1. Audible seams between phrases. When you feed a compact TTS model different samples of the same person from phrase to phrase, it copies not only the intonation but also the slightest differences in distance from the microphone, loudness, and room acoustics. At the joins, it feels as though the lines were recorded in different rooms.

2. Whispering and a throat spasm cannot be reduced to pitch (F0). When a living person switches to whispering or speaks with a spasm, their vocal folds stop closing as usual: the clear fundamental tone disappears, air and breath noise appear in the voice, and the spectrum shifts in frequency. If you simply stretch the pitch of ordinary speech up and down in Praat, you get mechanical modulation rather than a whisper. That said, for minor adjustments—for example, correcting the stress or slightly raising the intonation of a question in a calm lecture—Praat remains a perfectly workable tool.

Approach 3.Recordings of professional audiobook narrators (ToneBooks)

The third hypothesis is to test whether the result will change if we use recordings of professional voice actors from the open ToneBooks dataset, with pronounced artistic delivery and a semi-whisper, as the reference.

The recording began to sound like a high-quality commercial audiobook, but this revealed a genre difference: an audiobook narrator and a dramatic actor perform different tasks. A narrator reads a book that runs for hours over the long haul—he takes care of his vocal cords and maintains an even delivery without sharp changes in volume or breathing. He does not act out spasms, broken breathing, or a switch to a muffled whisper.

Part 2. How Fish Audio Drama 3 and S2 Pro Work

In my search for models capable of changing vocal register and breathing directly within a single generation, I decided to test the lineup from the Fish Audio team (the developers of the open Fish Speech family).In late October 2026, they released the weights of the flagship fishaudio/drama-3 model, and in early November, its quantized port, mlx-community/fish-audio-drama-3-4bit, appeared in the mlx-audio library.

In terms of its internal architecture, Drama 3 (model_type: fish_drama3) differs markedly from the usual lightweight TTS solutions:

• Large base language model 35B-A3B MoE (FishOmniModel): Instead of a small network with 0.5B–1.7B parameters, it uses a 40-layer hybrid transformer (with one full-attention layer for every three linear-attention layers) with a Mixture-of-Experts architecture: 256 routed experts in each layer, of which 8 are activated per token (about 3 billion active parameters out of 35 billion total), with a vocabulary of 252,288 tokens.

• Temporal audio tower (Tower) and 44.1 kHz codec: Unlike the junior S2 Pro, which uses a simple 4-layer Dual-AR head with no memory between adjacent frames, Drama 3 uses a 13-block audio tower (codebook_mtp_shared) that preserves its own KV cache over time.It predicts 10 codebooks (layers of discrete audio tokens with a vocabulary of 4096 that encode timbre and sound details), which are decoded by the Fish S1 DAC neural audio codec at a sample rate of 44.1 kHz.

• Multi-reference template (reference-bracket-v1): If calm speech, a whisper, and a scream are concatenated into a single reference file for a standard zero-shot model, it will either average them or begin randomly jumping in pitch in the middle of ordinary words. The Drama 3 system prompt accepts an array of separate [Reference 1], [Reference 2], [Reference 3] pairs (text + audio codes), binding each voice state to its corresponding transcript.

• Stage-direction tags in the text: The model is trained to interpret textual stage directions in square brackets (in my tests, I tried [sighs], [sigh], [sad], [whisper], [low voice]) and paired XML tags <whisper>...</whisper>. These tags themselves are not separate special tokens in the Drama 3 vocabulary and are split by BPE like ordinary text; however, the chunking preprocessor (chunking.py, a limit of 300 UTF-8 bytes per chunk) specifically protects <tag>...</tag> and [...] constructs using the UNCUTTABLE regular expression so that a stage direction is not split at a segment boundary.The tokenizer also includes 16 special tokens for multi-voice dialogues (<|speaker:0|>..<|speaker:15|>), but I did not test this mode—I only tested the model with a single voice.

Memory consumption and hardware for local deployment

Even with mixed quantization using group_size=64 (4-bit for routed MoE experts and 8-bit for attention layers, shared experts, and the audio tower), the model weights occupy 22.15 GiB (23.8 GB) on disk. On desktop GPUs with 24 GB of VRAM, running such a model with SGLang-Omni or vLLM-Omni hits the VRAM limit.

Local deployment settings (M4 Pro, 48 GB, mlx-audio backend): the model loads into memory in 8.3 seconds, peaks at 28.6–28.8 GB, and generates 44.1 kHz audio at approximately 1:1 real-time speed (44.0 seconds of monologue takes 44.8 seconds).

Part 3. Preparing emotional references and launch code

To enable Drama 3 to demonstrate the capabilities of the multi-reference template, it needs clean studio recordings in which the same person sounds in different emotional states without background music or noise.

For a lab test, the easiest option was to use clean studio tracks from a game localization. Ideally, for such a task, one should turn to theatrical archives of actor audition recordings, radio plays, or monodrama recordings, where the emotional range of a single performer has already been clearly laid out.But at that point, all I had the presence of mind for was the unpacked game source code.

I compiled three test voice profiles (a low dramatic baritone, an emotional tenor, and a restrained chamber baritone) and selected for each three clips lasting from 5.6 to 12.5 seconds in different states: calm narration, muffled intermittent whispering, and a heavy exhale with a spasm.

The main technique for annotating reference clips

When submitting the list ref_audio=[wav_1, wav_2, wav_3] along with their text transcriptions ref_text=[txt_1, txt_2, txt_3], I placed the very same tags [sighs] and <whisper>...</whisper> exactly where the actor sighs or switches to whispering in the actual recording (it is important that the text in ref_text match word for word the words spoken in the recording, otherwise the alignment between the text and audio codes breaks down). This way, the model directly associates the tag in the prompt with how this particular person’s sigh and whisper actually sound:

```python # pip install -U "mlx-audio>=0.6.2" soundfile numpy import numpy as np import soundfile as sf from mlx_audio.tts.utils import load_model # Loading the 4-bit model (peak memory during generation: ~28.8 GB) model = load_model("mlx-community/fish-audio-drama-3-4bit") hamlet_text = ( "[sighs] Alas, poor Yorick! I knew him, Horatio: a fellow of infinite jest, of most excellent fancy; he bore me on his back a thousand times. " "[sighs] And now—how abhorrent this is! My throat tightens when I look at him. " "<whisper>Here were those lips that I kissed who knows how many times.</whisper> " "[sighs] Where are your jokes now? Your antics? Your songs? " "Your flashes of merriment, which had everyone at the table rolling with laughter?" ) ```Three reference samples of the same voice (calm speech, whisper, heavy sigh) ref_audio = ["refs/profile_a_calm.wav", "refs/profile_a_whisper.wav", "refs/profile_a_sigh.wav"] ref_text = [ "I, too, had dreamed about this child. I used to say: he’ll be born into the world, and everything will change for the better. But he died before he was born.", "<whisper>She was lying there... tiny... defenseless... on bloodstained sheets... dead... because of me.</whisper>", "Thrown away? Oh, gods... I was hoping she still might... [sighs] Aah...", ] pieces = [ np.array(res.audio, dtype=np.float32) for res in model.generate( text=hamlet_text, ref_audio=ref_audio, ref_text=ref_text, temperature=0.8, top_p=0.8, top_k=30, ) ] audio = np.concatenate(pieces) sf.write("hamlet_drama3_44k.wav", audio / np.max(np.abs(audio)) * 0.89, model.sample_rate)

Part 4.Synthesis results and comparison with the smaller Fish Audio S2 Pro (4B)

On all three voice profiles, Drama 3 delivered the transitions exactly as required by the script:

• On the [sighs] tags, a lifelike inhale and exhale can be heard, after which the model adjusts the pace and pitch of the voice rather than merely inserting silence (the graph shows the orange line dropping down into the turquoise zone in the [sighs] column);

• The phrase inside <whisper>...</whisper> (the third column of the graph) sounds like a genuine muffled whisper with air noise at the microphone—and both Drama 3 (the orange line) and S2 Pro (the blue line) drop to the very bottom of the graph here, unlike the gray dashed line of Qwen3-TTS;

• In the final series of rhetorical questions (the fourth column), Drama 3 independently constructs an intonational staircase that rises into the upper red zone (voice break/shriek on the final questions). Stability is also worth noting separately: at temperature=0.8, the model varies the pause duration slightly from run to run (40.5–44.0 s), but it never breaks into hallucinations or swallows any words.

TAG220@@After generation, I ran the resulting tracks through RVC v2 (40 kHz) to adapt them to the target timbre of our project. Yes, resampling through 40 kHz RVC cuts some of the high frequencies compared with the native 44.1 kHz Drama 3, but this is a deliberate practical compromise: a high-quality RVC model for the desired voice can be trained on 10–15 minutes of clean audio, whereas fully fine-tuning a 35-billion-parameter model on one’s own voice is vastly more difficult.To prevent RVC from trying to detect the fundamental pitch in a muffled whisper and artificially “singing” it with the vocal folds during conversion, I locked in protect=0.35 (protecting voiceless consonants and breathing from timbral substitution) and rms_mix_rate=0.25 (preserving the original loudness envelope so the converter does not boost a quiet whisper to the level of loud speech).

Comparison with the smaller Fish Audio S2 Pro model (8-bit, 6.7 GB)

Since we are talking about the difference between 1.7B and 35B, and it is not always sensible to allocate nearly 29 GB of memory to voice synthesis, I tested an intermediate option on the same monologue: the smaller model from the same family, mlx-community/fish-audio-s2-pro-8bit (4B parameters, 6.27 GiB / 6.73 GB on disk). Since S2 Pro in mlx-audio accepts a single reference array and uses prefixed single-word tags, the three reference clips were supplied concatenated with a 0.25 s pause between them, and the markup was adapted to its native syntax ([sigh], [sad], [whisper] [low voice]).

A direct comparison of S2 Pro (4B) and Drama 3 (35B MoE) revealed three main differences. First, in terms of resources, S2 Pro initializes in just 2.4 s and peaks at 12.45 GB of memory (2.3 times less than the larger model), using the same 44.1 kHz Fish S1 DAC audio codec. Second, the smaller model handles the [whisper] [low voice] tags well—the transition to a whisper in the middle of the monologue is preserved—but it tends to skip the sigh tags [sigh], speaking the entire monologue in 29.8 s versus 44.0 s for Drama 3.Meanwhile, the 29.8-second audio computation on S2 Pro took 54.4 seconds because the Dual-AR head was recalculated without an inter-frame cache.

Third, with raw text and no stress marks, the smaller 4B model made a mistake with a rare name—it pronounced “Góratio” instead of “Gorátio”—whereas the 35B model Drama 3 read the classic text without a single error on the first pass. Naturally, this problem can easily be worked around with the smaller model by using a cleaner prompt (adding stress marks and employing other tricks that everyone who works with voice generation knows about, even without my help), but the fact itself illustrates the difference in the size of the language model within TTS.

Summary table of the tested models

Such an analysis back in my day would have saved me several days of experimentation. So I’ll be glad if it proves useful to someone.