Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
MCML Authors
Felicia Körner
Florian Eichin
Gitta Kutyniok
Prof. Dr.
Core PI
Barbara Plank
Prof. Dr.
Core PI
Michael Hedderich
Dr.
JRG Leader Human-Centered NLP
Abstract
Felicia Körner
Florian Eichin
Gitta Kutyniok
Prof. Dr.
Core PI
Barbara Plank
Prof. Dr.
Core PI
Michael Hedderich
Dr.
JRG Leader Human-Centered NLP
Abstract
Large language models exhibit impressive cross-lingual capabilities. However, prior work analyzes this phenomenon through isolated factors and at sparse points during training, limiting our understanding of how cross-lingual generalization emerges--particularly in the early phases of learning. To study the early trajectory of linguistic and translation capabilities, we pretrain a multilingual 1.7B model on nine diverse languages, capturing checkpoints at a much finer granularity. We further introduce a novel word-level translation dataset and trace how translation develops over training through behavioral analyses, model-component analysis, and parameter-based ablations. We find that the model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which more generalizing translation mechanisms are developed while copying is refined. Together, these findings provide a fine-grained view of how cross-lingual generalization develops during multilingual pretraining.
inproceedings KME+26
EMNLP 2026
Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026. To be published. Preprint available.
Authors
F. Körner • M. Matveev • F. Eichin • G. Kutyniok • B. Plank • M. A. HedderichLinks
arXivResearch Areas
BibTeXKey: KME+26