Tokenization and sequence modelling
Vocabulary allocation determines effective capacity across scripts.
Multilingual and low-resource NLP
Language count is not language coverage. This chapter audits scripts, data, tokenization, resource tiers, native evaluation, and per-language uncertainty.
14 maintained papers · literature reviewed through 2 October 2026
A multilingual score hides script, tokenizer, data, domain, dialect, and annotation inequalities. Cross-lingual transfer must distinguish language coverage, parallel versus monolingual supervision, unseen-language transfer, retrieval or generation quality, and human evaluation by fluent speakers.
Prerequisite contract
Task and metric contracts
Translation, retrieval, and NLI require different language-aware evidence.
How the field developed
2016–2018
Cross-lingual annotation and inference
Universal Dependencies and XNLI establish comparable structure and inference tasks across many languages.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 01.01 Universal Dependencies v1: A Multilingual Treebank Collection Universal Dependencies (UD) v1 is an open, dependency-based annotation framework designed to make morphology and syntax comparable across languages. The paper fixes a shared lexicalist analysis, universal inventories, and a common serialized representation while permitting language-specific extensions. Its accompanying v1.2 snapshot comprised 37 treebanks in 33 languages; this is a historical resource paper, not the specification of today’s larger UD release. Universal Dependencies 01.02 XNLI: Evaluating Cross-lingual Sentence Representations XNLI turns natural-language inference into a controlled cross-lingual evaluation: 7,500 English MultiNLI development/test pairs were professionally translated into 14 additional languages, producing 112,500 labeled premise–hypothesis pairs across 15 languages. The three-way entailment, contradiction, or neutral label is inherited only after a separate validation process checks whether translation preserved the relation. Official dataset Work: Audit translation, annotation, and genre differences before comparing language scores.
Exit evidence: Produce per-language data and uncertainty cards rather than one macro average.
2020–2022
Multilingual pretraining, generation, and adapters
XLM-R, mBART, mT5, ByT5, and MAD-X compare shared subwords, multilingual denoising, text-to-text transfer, bytes, and modular language/task adapters.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 02.01 Unsupervised Cross-lingual Representation Learning at Scale XLM-R scales one multilingual masked-language-model encoder to 100 languages using 2.5 TB of cleaned CommonCrawl text, a shared SentencePiece vocabulary, smoothed language sampling, and substantially increased model capacity. Without parallel sentences or language-specific supervision during pretraining, it learns representations that transfer from labeled English—or combined multilingual labels—to classification, sequence labeling, and question answering in many languages. Models and code 02.02 Multilingual Denoising Pre-training for Neural Machine Translation mBART pretrains a complete multilingual encoder-decoder Transformer as a denoising autoencoder. It corrupts monolingual documents in 25 languages with span masking and sentence permutation, then reconstructs the original text autoregressively. Because both encoder and decoder are pretrained, the same checkpoint can initialize supervised sentence translation, document translation, unsupervised translation, and transfer to language pairs absent from bitext. Official fairseq implementation and checkpoints 02.03 MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer MAD-X adapts a frozen multilingual Transformer by composing small language adapters with a task adapter. A language adapter learns unsupervised masked-language modeling for one language; a task adapter learns labeled supervision in a source language; inference swaps only the language component for the target. Invertible adapters around the embedding interface reduce mismatch between a shared multilingual vocabulary and languages that were rare or absent in pretraining. Official AdapterHub resource · Official adapters implementation 02.04 mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer mT5 applies the T5 text-to-text encoder-decoder recipe to 101 languages using the newly constructed mC4 CommonCrawl corpus and model sizes from roughly 300M to 13B parameters. Span-corruption pretraining and one shared vocabulary let every downstream task produce text, enabling both multilingual classification and generation—but also allowing unconstrained outputs in the wrong language. Code and checkpoints 02.05 ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models ByT5 removes the learned SentencePiece vocabulary from multilingual T5 and maps UTF-8 bytes directly through an encoder-decoder Transformer. It spends parameters freed from the large embedding/softmax matrices on wider dense layers, uses an encoder three times as deep as its decoder, and pretrains by reconstructing masked byte spans. The result is one byte-to-byte text-to-text interface for classification and generation, with no learned subword segmentation. Code and checkpoints Work: Measure tokenizer fertility, sequence length, adapter capacity, compute, and task quality by language and script.
Exit evidence: Attribute gains to data, tokenization, objective, modular transfer, or scale rather than “multilinguality.”
2022–2024
Comparable translation and retrieval coverage
FLORES-101 supplies many-to-many translation evaluation; MIRACL evaluates native multilingual retrieval; NLLB scales translation toward 200 languages with explicit low-resource data work.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 03.01 The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation FLORES-101 is a professionally translated, fully aligned machine-translation evaluation set spanning 101 languages. It starts from 3,001 English Wikipedia-derived sentences and provides a translation of every sentence in every language, enabling comparable English-centric and direct many-to-many evaluation across 10,100 directed language pairs. Its central scientific contribution is controlled coverage of low-resource languages, not a training corpus. Official FLORES repository 03.02 MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages MIRACL is a native-language, monolingual retrieval resource: a query in one of 18 languages searches Wikipedia passages in that same language. Native speakers produced natural questions and binary relevance judgments, yielding roughly 78.7k queries and more than 724k displayed split labels across languages that span 11 scripts, 10 families, and 13 subfamilies. It supports training and evaluation rather than only a small multilingual test set. Dataset and evaluation 03.03 Scaling neural machine translation to 200 languages No Language Left Behind (NLLB) is an end-to-end program for machine translation at 200-language scale, not just a larger Transformer. It combines community-informed requirements, language identification, cleaned monolingual data, LASER3 bitext mining, human seed data, FLORES-200, a 54.5-billion-parameter sparsely gated mixture-of-experts model, human quality assessment, and Toxicity-200. The system evaluates 40,602 directions among 204 language varieties. Official NLLB artifacts Work: Evaluate native queries and generation with fluent review, domain shift, and resource-tier slices.
Exit evidence: Report quality, toxicity, entities, dialect, direction, and compute by language.
2026
Morphology, culture, and fact-checking coverage
PolyGloss targets low-resource morphology, Afri-MCQA separates linguistic from cultural multimodal gaps, and VeriTaS normalizes multilingual claims over time; none represents every language or culture.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 04.01 Massively Multilingual Joint Segmentation and GlossingPeer-reviewed ACL 2026 long paper and Outstanding Paper award recipient. PolyGloss treats morphological segmentation and interlinear glossing as one structured generation problem instead of two isolated pipelines. A multilingual encoder-decoder emits aligned morpheme and gloss sequences in an interleaved representation, so every predicted boundary is trained together with its linguistic label. This joint contract lets segmentation evidence improve glossing and gives low-resource languages a shared model rather than one model per language. Author manuscript · Official code · Official models and data 04.02 Afri-MCQA: Multimodal Cultural Question Answering for African LanguagesPeer-reviewed ACL 2026 long paper and Best Social Impact Paper award recipient. Afri-MCQA is a culturally grounded multimodal question-answering resource created by native speakers rather than translated from a Western benchmark. It contains about 7.5K parallel question–answer pairs across 15 African languages and 12 countries, with culturally relevant images, English and local-language text, and speech. Parallel forms let evaluators separate cultural knowledge from language or speech access instead of collapsing every error into one VQA score. Author manuscript · Official dataset 04.03 VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-CheckingPeer-reviewed ACL 2026 long paper and Best Resource Paper award recipient. VeriTaS is a dynamic multimodal fact-checking benchmark rebuilt on a quarterly cadence from professional fact-check organizations. A seven-stage pipeline discovers reviews, scrapes articles, retrieves original media, normalizes claims, maps heterogeneous verdicts to a disentangled claim-integrity scale, and rectifies records. The moving time split reduces the value of memorizing a static benchmark and lets evaluators test whether search actually closes a model's knowledge-cutoff gap. Author manuscript · Official project and data portal · Official code Work: Evaluate morphology, culturally grounded multimodal QA, and dynamic multilingual fact checking with language-, culture-, source-, and time-held-out splits rather than one pooled multilingual score.
Exit evidence: Publish per-language denominators, annotation and translation provenance, cultural expert agreement, modality ablations, temporal leakage checks, access conditions, and update cadence.
2025
Embedding quality is a task–language profile, not one universal rank
MMTEB expands embedding evaluation across languages and task families while reducing retrieval cost through pooled candidates and task selection. Mean scores, Borda ranks and individual retrieval metrics answer different questions; language coverage and translated labels constrain interpretation.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 05.01 MMTEB: Massive Multilingual Text Embedding BenchmarkReview uses the proceedings PDF and Table 2 benchmark definitions. The landing abstract states 1,000+ languages while the PDF abstract states 250+; Section 5 attributes the broader count to bitext mining. Task-count inconsistencies in prose are retained as caveats, not silently combined. MMTEB expands text-embedding evaluation across languages, tasks and domains, then constructs smaller benchmark suites that preserve useful model discrimination. It evaluates embeddings, not free-form answer generation. Its strongest lesson is that a model’s English or retrieval ranking need not predict its performance across low-resource languages and other embedding tasks. Proceedings PDF · Benchmark implementation · Versioned evaluation results · arXiv Work: Does the selected model work on the intended language, task, corpus and relevance labels?
Exit evidence: Proceedings text and tables disagree on some counts; the review preserves those differences. The inspected results are a fixed paper snapshot, not the current leaderboard or a clinical retrieval validation.
Research branches
Representation and modular transfer
Which language receives positive transfer, and can adapters isolate interference?
Unsupervised Cross-lingual Representation Learning at Scale · MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer · mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer · ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models
Multilingual retrieval
Are query, corpus, relevance judgments, and negatives native to the language?
MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages
Low-resource translation
What supervision, benchmark coverage, filtering, back-translation, and human evidence support each direction?
Multilingual Denoising Pre-training for Neural Machine Translation · The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation · Scaling neural machine translation to 200 languages
Morphological, cultural, and temporal multilingual coverage
Is failure caused by language form, cultural knowledge, visual grounding, source normalization, or evidence that changed over time?
Massively Multilingual Joint Segmentation and Glossing · Afri-MCQA: Multimodal Cultural Question Answering for African Languages · VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking
Embedding quality is a task–language profile, not one universal rank
Does the selected model work on the intended language, task, corpus and relevance labels?
Comparison table
| Family or stage | Scientific distinction | Main risk | Anchors |
|---|---|---|---|
| Shared subwords | Shares parameters through a learned vocabulary. | High-resource languages dominate vocabulary and batches. | 2020 · 2021 |
| Bytes | Avoids unknown tokens and language-specific segmentation. | Longer sequences raise compute and context pressure. | 2022 |
| Task-scale coverage | Builds comparable retrieval or translation resources for broad language sets. | Nominal language count hides unequal domains and quality. | 2022 · 2023 · 2024 |
| Morphology, culture, and fact-checking coverage | PolyGloss targets low-resource morphology; Afri-MCQA separates linguistic and cultural multimodal gaps; VeriTaS normalizes professional fact checks into a periodically refreshed multilingual resource. | Language-family concentration, translated artifacts, Western-centric models, gated sources, and temporal overlap can be mistaken for broad multilingual or cultural competence. | 2026 · 2026 · 2026 |
| Embedding quality is a task–language profile, not one universal rank | MMTEB expands embedding evaluation across languages and task families while reducing retrieval cost through pooled candidates and task selection. Mean scores, Borda ranks and individual retrieval metrics answer different questions; language coverage and translated labels constrain interpretation. | Proceedings text and tables disagree on some counts; the review preserves those differences. The inspected results are a fixed paper snapshot, not the current leaderboard or a clinical retrieval validation. | 2025 |
Concept lineage
XNLI: Evaluating Cross-lingual Sentence RepresentationsUnsupervised Cross-lingual Representation Learning at Scale
A cross-lingual benchmark becomes a central test of massively multilingual pretraining.
mT5: A Massively Multilingual Pre-trained Text-to-Text TransformerScaling neural machine translation to 200 languages
General multilingual generation is complemented by translation-specific data and evaluation at far broader low-resource coverage.
Multilingual Denoising Pre-training for Neural Machine TranslationThe FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
Multilingual denoising generation is tested against a many-to-many translation benchmark with much broader language coverage.
Failure modes to audit
- Reporting only averages over languages.
- Treating translated test sets as native usage.
- Ignoring tokenizer fertility and script normalization.
- Claiming unseen-language transfer after data contamination.
- Using automatic metrics without fluent human review.
Questions for every paper
- Which languages, scripts, and domains are covered?
- What data exists per language?
- Is the evaluation translated or native?
- How does quality vary by resource tier?
Complete maintained paper route
Section titled “Complete maintained paper route”14 canonical readings · 2016–2026· route reviewed through October 2, 2026.
This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.
Show all 14 readings in maintained order
Universal Dependencies v1: A Multilingual Treebank Collection
Joakim Nivre et al.
Official artifacts: Universal Dependencies
Metadata checked
XNLI: Evaluating Cross-lingual Sentence Representations
Alexis Conneau et al.
Official artifacts: Official dataset
Metadata checked
Multilingual Denoising Pre-training for Neural Machine Translation
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer
Official artifacts: Official fairseq implementation and checkpoints
Metadata checked
MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder
Official artifacts: Official AdapterHub resource · Official adapters implementation
Metadata checked
Unsupervised Cross-lingual Representation Learning at Scale
Alexis Conneau et al.
Official artifacts: Models and code
Metadata checked
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Linting Xue et al.
Official artifacts: Code and checkpoints
Metadata checked
The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzmán, and Angela Fan
Official artifacts: Official FLORES repository
Metadata checked
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel
Official artifacts: Code and checkpoints
Metadata checked
MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages
Xinyu Zhang et al.
Official artifacts: Dataset and evaluation
Metadata checked
Scaling neural machine translation to 200 languages
Marta R. Costa-jussà et al.
Official artifacts: Official NLLB artifacts
Metadata checked
MMTEB: Massive Multilingual Text Embedding Benchmark
Kenneth Enevoldsen et al.
Review uses the proceedings PDF and Table 2 benchmark definitions. The landing abstract states 1,000+ languages while the PDF abstract states 250+; Section 5 attributes the broader count to bitext mining. Task-count inconsistencies in prose are retained as caveats, not silently combined.
Official artifacts: Proceedings PDF · Benchmark implementation · Versioned evaluation results · arXiv
Metadata checked
Massively Multilingual Joint Segmentation and Glossing
Michael Ginn, Lindia Tjuatja, Enora Rice, Ali Marashian, Maria Valentini, Jasmine Xu, Graham Neubig, and Alexis Palmer
Peer-reviewed ACL 2026 long paper and Outstanding Paper award recipient.
Official artifacts: Author manuscript · Official code · Official models and data
Metadata checked
Afri-MCQA: Multimodal Cultural Question Answering for African Languages
Atnafu Lambebo Tonja, Srija Anand, Emilio Villa-Cueva, Israel Abebe Azime, Jesujoba Oluwadara Alabi, Muhidin A. Mohamed, Debela Desalegn Yadeta, Negasi Haile Abadi, Abigail Oppong, Nnaemeka Casmir Obiefuna, Idris Abdulmumin, Naome A Etori, Eric Peter Wairagala, Kanda Patrick Tshinu, Imanigirimbabazi Emmanuel, Gabofetswe Malema, Alham Fikri Aji, David Ifeoluwa Adelani, and Thamar Solorio
Peer-reviewed ACL 2026 long paper and Best Social Impact Paper award recipient.
Official artifacts: Author manuscript · Official dataset
Metadata checked
VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking
Mark Rothermel, Marcus Kornmann, Marcus Rohrbach, and Anna Rohrbach
Peer-reviewed ACL 2026 long paper and Best Resource Paper award recipient.
Official artifacts: Author manuscript · Official project and data portal · Official code
Metadata checked