← All research

Is Prosody Lost in Translation?
Fine-Grained Cross-Lingual Prosody Similarity Across Languages

Haopeng Xie, Ismail Rasim Ulgen, Sofia Son, Berrak Sisman, Philipp Koehn

Center for Language and Speech Processing, Johns Hopkins University

Findings of EMNLP 2026

🇭🇺 I'll be presenting this work at EMNLP 2026 in Budapest. Hope to meet everyone there!

arXiv PDF HTML BibTeX Follow-up: Tone & Word Order

One sentence, two languages

The paper's running example, from a German show dubbed into English. Explore the real pitch contours from the paper, then see how alignment lines up the words before measuring correlation.

DE Ich habe immer gewusst, dass ich weiter das Mittelmeer erkunden muss.

EN I always knew I had to return to explore the Mediterranean Sea.

German (source) English (dubbed) interpolated (unvoiced)

Data extracted from Figure 2 of the paper.

Try it with your own voiceSpeak a sentence and watch your pitch contour drawn live, like the German and English contours above. It uses a simpler real-time method than the paper, so values will differ. Nothing is recorded or stored. Open the pitch tracker →
3language pairs
300Kmined utterance pairs
49Kpairs after filtering
110 hof dubbed speech
p<10−80vs. shuffled baseline

Overview

Prosody (pitch, loudness, and timing) carries emphasis, emotion, and intent beyond the words themselves. Expressive speech-to-speech translation systems try to carry it across languages, but little is known about how prosody actually corresponds between languages.

We present the first fine-grained, word-level analysis of cross-lingual prosody, using professionally dubbed TV speech in English–German, English–Spanish, and English–French. Dubbing actors aim to preserve the expressiveness of the original, which makes dubbing a natural parallel corpus for prosody.

How it works

Mine parallel speech and keep only clean pairs

Source and dubbed clips are mined with SpeechVecAlign, then filtered for audio quality, single speakers, and semantic match. About 1 in 6 pairs survives (see the funnel below).

SpeechVecAlignSQUIM (SI-SDR, STOI, PESQ)pyannoteSONAR
Data

From 100K mined pairs per language to about 16K clean ones

Utterance pairs remaining

Findings

Finding 1

Prosody partly survives translation

Source and dubbed contours are positively correlated in every language pair, far above the shuffled baseline. Pitch corresponds more strongly than energy.

Mean Spearman ρ95% CIShuffled baseline

How the correlations are distributed

Each utterance pair gets its own correlation. The whole distribution shifts right of zero, so the effect isn't driven by a few outliers. Drag the threshold to see what share of pairs clear it.

Utterance pairs per binMeanShuffled-baseline meanρ = 0

Show data table

Data extracted from Figure 3 of the paper; bin counts sum to each language pair's n.

Finding 2

Nouns carry the shared prosody

Removing a word class from the contour shows how much it contributes. Nouns help most in every language pair. Function words make energy correspondence worse.

Helps correspondenceHurts correspondence
Finding 3

Timing is almost fully preserved

At the utterance level, duration and phoneme count are strongly correlated across languages (ρ ≈ 0.87–0.89). Dubbing actors rarely need to change their speaking rate much to fit the timing.

DE–EN
0.891duration0.885phonemes
EN–ES
0.890duration0.875phonemes
EN–FR
0.884duration0.871phonemes

My role

Follow-up work

Under review

Lexical Tone and Word Order Matter in Translation: Prosodic Correspondence Across Languages

Extends this analysis to English paired with Mandarin, Thai, Vietnamese, Japanese, and Korean, testing how lexical tone and word reordering affect prosodic correspondence. The European results from this paper serve as its reference.

View project →

Citation

@article{xie2026prosody,
  title   = {Is Prosody Lost in Translation? Fine-Grained Cross-Lingual
             Prosody Similarity Across Languages},
  author  = {Xie, Haopeng and Ulgen, Ismail Rasim and Son, Sofia and
             Sisman, Berrak and Koehn, Philipp},
  journal = {arXiv preprint arXiv:2608.27848},
  note    = {Accepted to Findings of EMNLP 2026},
  year    = {2026}
}