Fixing emotion tags by editing the weights, not retraining

Listen

All clips come from the held-out test set: lines, seeds and two of the three voices were never used while building the edit. Same text, same voice, same seed, only the weights differ. Clips are loudness-normalized the same way generate() does by default. Every take, failures included, is in the explorer.

<whisper> the biggest change

Not every whisper works. The worst of the 24 test takes, voice 260 / "You kept the letter all these years." / seed 11, comes out as barely audible mumbling. Hear it in the explorer.

Other emotions

What was wrong

1. The tag embeddings barely moved. Each tag row was initialized from a word (<sad> from "sad", <whisper> from "whisper") and still has 60–80% cosine similarity to it. Making the rows longer would not help: a LayerNorm sits before attention, so only the direction counts.
2. The audio does not read the tags. An attention probe showed that audio frames attend to <happy> 100–3000× more than to an average word. For the other tags it is only 1–10×.
3. The information is already there. Boosting attention to <whisper> directly in the activations (×30, all heads) produced a whisper in 62% of takes with clean words. ×100 broke speech. Boosting only the three heads that read tags the most did almost nothing.

The edit

Each of the 8 layers gets one rank-1 term added to its key projection:

• z_t is a linear detector for tag t, fit by least squares on the layer's input: 1.00 on the tag token, 0 ± 0.03 everywhere else.
• D shifts the key in the slowest RoPE pairs, which barely rotate, along the mean query of the audio frames. So it raises the tag's attention score by about ln g_t wherever the tag sits in the text.
• Gains g_t: sad, fear, surprised ×5; angry ×3; whisper ×10. <happy> and the sound tags are left untouched.

That changes 524k weights (3.4% of the transformer, all inside in_proj), but the edit has only about 5k free parameters. Every variant was tried on a dev split and kept only if it improved. The final one was checked once on the test split.

Results

How these were measured. WER comes from Whisper base. A take counts as a whisper when less than 40% of its loud frames are voiced (pyin). Separability is a nearest-centroid classifier on Whisper-base encoder features, after subtracting the mean of each (voice, line, seed) group, holding out whole groups. Consistency is how much the tag moves takes in the same direction (|mean change|² / mean |change|²). The dev split shows the same picture: whisper 0/24 → 24/24, separability 0.35 → 0.61.

What didn't work

• Pushing tag embeddings away from their seed words: no change beyond noise.
• Applying the boost to audio-frame queries only, keeping text tokens out of it: the effect mostly disappears. The tag works mainly through the text tokens that read it, not through a direct audio → tag path.
• Whisper at ×14 or more: it whispers more reliably, but words smear (×20 → WER 0.41).

Caveats

• Some whispers still slur words (whisper WER 0.18), and 1 of 24 test whispers collapses into mumbling.
• Whispers come out about 20 dB quieter than speech. generate() raises quiet takes to speech level by default, so keep normalize=True.
• <surprised> went up about 4 semitones and can sound over the top.
• Untagged takes are not bit-identical to the release. The detector leaks about 0.001, which acts like a seed change. Quality is the same.
• All metrics are proxies. There was no emotion classifier in the loop, so in the end your ears decide.

Every test take

3 voices × 4 lines × 2 seeds × 7 classes. "No tag" is the same line through the edited model, to check that plain speech did not change.

Try it

pip install -r requirements.txt   # from the Hugging Face repo

from silencio2 import Silencio2TTS
tts = Silencio2TTS.from_pretrained("BlueRoseLabs/Silencio-2", revision="weightedit")
tts.save("out.wav", "<whisper> Keep your voice down.", voice="260")