Attention and autoregressive models
Needed to trace teacher forcing, exposure, and decoding.
Text generation, translation, and evaluation
Training loss, decoding probability, reference overlap, learned evaluation, and human utility are related but non-equivalent forms of evidence.
For image-conditioned text, CIDEr (2015) weights one- through four-gram agreement with multiple references by TF–IDF, so phrases repeatedly supported for one image but rare across images carry more credit. SPICE (2016) instead parses candidate and references into object, attribute, and relation propositions and computes an F-score. CIDEr measures lexical consensus; SPICE measures parsed semantic overlap. Neither reads the image, so grounding, hallucination, reference coverage, parser failure, and blinded human validity remain separate checks.
26 maintained papers · literature reviewed through 2 October 2026
Autoregressive likelihood, sequence-to-sequence pretraining, distillation state distribution, and decoding define how text is produced; BLEU, ROUGE, learned metrics, and human judgments measure different aspects. Fixed teacher outputs and teacher feedback on student-visited prefixes train different distributions. Factuality, adequacy, fluency, diversity, calibration, and utility must remain separate claims.
Prerequisite contract
Measurement and uncertainty
Metric correlations and human preferences require study design.
How the field developed
2002–2004
Reference-overlap metrics
BLEU aggregates modified n-gram precision for translation; ROUGE emphasizes recall-oriented overlap for summaries.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 01.01 BLEU: a Method for Automatic Evaluation of Machine Translation BLEU is a corpus-level reference-overlap metric for machine translation. It geometrically averages clipped candidate n-gram precisions—normally orders one through four—and multiplies by a brevity penalty when the whole candidate corpus is shorter than its closest-length references. It was proposed as a fast understudy for frequent system development, not as a replacement for human judgment of every sentence. — 01.02 ROUGE: A Package for Automatic Evaluation of Summaries ROUGE is not one universal number but a reference-based family for summarization: ROUGE-N counts reference-side n-gram recall, ROUGE-L scores longest in-order subsequences, ROUGE-W rewards consecutive runs within an LCS, and ROUGE-S/SU count ordered skip-bigram pairs, optionally with unigrams. The paper evaluates many configurations against DUC system-level human content-coverage judgments. — Work: Construct examples where semantic quality and overlap disagree, then inspect corpus aggregation.
Exit evidence: State tokenization, references, smoothing, aggregation, and uncertainty.
2015–2016
Caption consensus and semantic propositions
CIDEr applies TF–IDF to one- through four-gram reference consensus; SPICE parses captions into object, attribute, and relation tuples and scores proposition F1.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 02.01 CIDEr: Consensus-Based Image Description Evaluation CIDEr evaluates an image caption by asking whether its one- to four-word n-grams agree with the consensus of multiple independent human descriptions. Candidate and reference sentences become TF–IDF vectors: term frequency rewards phrases present in the references, while inverse document frequency downweights phrases common across many images. For each n-gram order, cosine similarity to every reference is averaged, then the four orders are combined. The central contribution is therefore reference-consensus weighting, not a learned judge or direct verification against image pixels. Official CVF PDF 02.02 SPICE: Semantic Propositional Image Caption Evaluation SPICE evaluates a caption after converting candidate and references into textual scene graphs. The graphs encode objects, attributes, and relations; these become logical tuples such as object, object–attribute, or subject–relation–object. SPICE computes precision, recall, and F-score over matched candidate and reference propositions, with WordNet synonym matching. Compared with n-gram metrics, it attempts to score what the caption says about the scene rather than how closely its word sequence copies a reference. CIDEr and BLEU remain complementary lexical views rather than visual validators. Author manuscript · Authors' implementation Work: Score adversarial captions with CIDEr-D and SPICE, then inspect unsupported entities, wrong relations, parser errors, and missing valid paraphrases.
Exit evidence: Separate lexical consensus, semantic proposition overlap, visual grounding, fluency, and human validity.
2015–2020
Neural, denoising, and multilingual generation
Attention and subword NMT establish neural sequence generation; BART, mBART, and PEGASUS pretrain denoising or gap-sentence objectives for monolingual, multilingual, or summarization transfer.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 03.01 Neural Machine Translation by Jointly Learning to Align and Translate RNNsearch replaces one fixed source vector with a bank of bidirectional token annotations. Before predicting each target word, a learned alignment network scores every source annotation, softmax-normalizes the scores, and supplies their weighted sum as a step-specific context. The encoder and decoder therefore learn translation and soft alignment jointly by backpropagation. — 03.02 Neural Machine Translation of Rare Words with Subword Units This paper adapts byte-pair encoding from compression to deterministic word segmentation. Start with characters plus an end-of-word marker, repeatedly merge the most frequent adjacent symbol pair, and retain the ordered merge list. Frequent words become long units while rare or unseen words remain decomposable into shorter character sequences, letting one fixed NMT vocabulary generate open-ended word forms without a dictionary back-off. — 03.03 BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension BART is a sequence-to-sequence denoising autoencoder: a bidirectional encoder reads an arbitrarily corrupted document, and a left-to-right decoder reconstructs the complete original document. This unifies BERT-like full input context with GPT-like autoregressive generation, allowing the same pretrained model to support both comprehension heads and conditional text generation. — 03.04 Multilingual Denoising Pre-training for Neural Machine Translation mBART pretrains a complete multilingual encoder-decoder Transformer as a denoising autoencoder. It corrupts monolingual documents in 25 languages with span masking and sentence permutation, then reconstructs the original text autoregressively. Because both encoder and decoder are pretrained, the same checkpoint can initialize supervised sentence translation, document translation, unsupervised translation, and transfer to language pairs absent from bitext. Official fairseq implementation and checkpoints 03.05 PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization PEGASUS pre-trains an encoder–decoder specifically for abstractive summarization by removing whole sentences from a document and asking the decoder to generate them. Its gap-sentence generation (GSG) objective selects sentences that appear summary-like—especially by ROUGE overlap with the rest of the document—so pre-training rehearses document compression rather than generic token denoising alone. Official code Work: Trace encoder, decoder, corruption, language coverage, loss, tokenizer, and decoding policy.
Exit evidence: Separate model probability, pretraining transfer, decoding policy, and selected checkpoint.
2016–2025
Sequence, divergence, adaptive, and on-policy distillation
Sequence KD trains on teacher-decoded targets; MiniLLM uses reverse KL; DistiLLM combines skew-KL with an adaptive off-policy mixture; on-policy distillation queries student-visited prefixes; TAID adapts the teacher target over training. These policies expose different quality–coverage and stability trade-offs.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 04.01 Sequence-Level Knowledge Distillation Sequence-Level Knowledge Distillation replaces a multimodal reference target with a complete sequence generated by a strong teacher. A compact sequence model is then trained by ordinary maximum likelihood on these teacher outputs. The central object is not a softened probability at each token but a decoded global hypothesis that exposes one coherent path through the teacher's sequence distribution. Official seq2seq implementation · Official mobile demonstration 04.02 MiniLLM: Knowledge Distillation of Large Language Models MiniLLM distills an autoregressive language model by minimizing reverse KL from the student sequence distribution to the teacher. The student generates its own responses, the teacher scores those trajectories, and policy-gradient machinery moves probability toward teacher-supported sequences. Reverse KL is mode seeking: it discourages the student from assigning mass to low-teacher-probability outputs rather than forcing it to cover every teacher mode. Official code 04.03 DistiLLM: Towards Streamlined Distillation for Large Language ModelsPeer-reviewed ICML 2024 proceedings paper. DistiLLM targets two coupled failures in autoregressive language-model distillation: unstable token-distribution matching and train/test exposure mismatch. It replaces ordinary forward or reverse KL with skew KL divergences whose mixture reference controls estimator behavior, then adaptively trains on student-generated sequences and reuses recent generations through an off-policy buffer. The student therefore learns both from fixed instruction data and from states its own policy is likely to visit. Official code 04.04 On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes Generalized Knowledge Distillation changes which prefixes receive teacher supervision. Instead of learning only on ground-truth or teacher-generated sequences, an autoregressive student samples complete outputs from its current policy; the teacher then supplies token distributions at those student-visited prefixes. A student-data fraction mixes these on-policy trajectories with a fixed dataset, while a selectable forward KL, reverse KL, or generalized Jensen-Shannon divergence controls how a capacity-limited student approximates the teacher. — 04.05 TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models TAID bridges a large language-model capacity gap by moving the student's target gradually from its own current distribution toward the teacher. At early steps, the interpolated target remains close to a detached student and is easy to fit; as learning stabilizes, the interpolation coefficient increases and exposes more teacher knowledge. The schedule adapts to objective progress instead of following only wall-clock training. Official code 04.06 Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation The paper offers a minimal distribution-level explanation for why a small generative student can produce sharper samples after knowledge distillation. It separates three distributions: an underlying ground truth p-star, an imperfect teacher p-prime learned from it, and a lower-capacity student p-double-prime learned from teacher samples. Reducing teacher entropy concentrates its samples on high-density modes. Forward-KL or maximum-likelihood training then spends the student's limited capacity on those modes, increasing precision while decreasing recall. Gaussian mixtures expose the mechanism analytically; a SmolLM2 1.7B→360M→135M pipeline tests the same temperature-controlled trade-off in autoregressive language modeling. Official code Work: Compare fixed sequences, forward/reverse/skew divergence, adaptive targets, off-policy mixtures, and on-policy states from one initialization.
Exit evidence: Report rollout, divergence, proposal distribution, importance weighting, intervention schedule, free-running precision/recall, quality, and total sampling compute.
2024
Parallel categorical generation
Discrete Flow Matching defines continuous-time probability paths over categorical states, providing a non-autoregressive alternative whose schedule, transition parameterization, and sampling steps must be evaluated separately from left-to-right decoding.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 05.01 Discrete Flow Matching Discrete Flow Matching transports a source distribution such as an all-mask sequence into a data distribution through continuous time while each state remains categorical. A learned probability denoiser determines token-jump rates in a continuous-time Markov chain. With the appropriate discrete divergence and continuity equation, the resulting probability velocity has the same denoiser-minus-current-state form as continuous flow matching. — Work: Implement one masked or uniform categorical path and compare quality, diversity, wall-clock latency, and function evaluations with an autoregressive baseline.
Exit evidence: State the probability path, transition model, scheduler, number of denoising steps, and decoding constraints.
2020–2024
Contextual metrics, human meta-evaluation, and LLM judges
BERTScore and COMET learn semantic quality signals; SummEval audits summarization dimensions and metric correlations; FLORES-101 standardizes broad translation evaluation; G-Eval uses prompted LLMs as structured judges.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 06.01 BERTScore: Evaluating Text Generation with BERT BERTScore evaluates generated text by replacing exact n-gram matches with contextual token similarities. It embeds a reference and candidate with a frozen pretrained encoder, greedily finds each token's best cosine match in the other sentence, averages in both directions as recall and precision, and combines them as F1. The result is a reference-based semantic-overlap signal that tolerates many paraphrases while retaining token-level evidence. — 06.02 COMET: A Neural Framework for MT Evaluation COMET is a learned machine-translation evaluation framework that scores a hypothesis by jointly representing the source sentence, candidate translation, and human reference with a multilingual pretrained encoder. It supports regression to human error/quality labels and pairwise ranking. The essential change from BLEU is that evaluation becomes supervised semantic prediction rather than fixed surface-overlap counting. Official code 06.03 SummEval: Re-evaluating Summarization Evaluation SummEval is a meta-evaluation resource: it evaluates both modern summarization systems and the automatic metrics used to judge them. The study unifies outputs from 23 papers, collects detailed annotations for 16 representative systems on 100 CNN/DailyMail articles, and asks expert and crowd annotators to score coherence, factual consistency, fluency, and relevance. It then correlates 23 automatic metric variants with expert system rankings. Official toolkit and annotations 06.04 The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation FLORES-101 is a professionally translated, fully aligned machine-translation evaluation set spanning 101 languages. It starts from 3,001 English Wikipedia-derived sentences and provides a translation of every sentence in every language, enabling comparable English-centric and direct many-to-many evaluation across 10,100 directed language pairs. Its central scientific contribution is controlled coverage of low-resource languages, not a training corpus. Official FLORES repository 06.05 G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment G-Eval turns natural-language generation evaluation into a structured form-filling task for an instruction-following LLM. A prompt specifies the task, evaluation criterion, and numbered scale; the LLM first writes evaluation steps, then reasons over the source and candidate output, and finally produces a score. Instead of trusting one integer, the method uses the probability distribution over score tokens to compute a continuous weighted score. Official implementation 06.06 Scaling neural machine translation to 200 languages No Language Left Behind (NLLB) is an end-to-end program for machine translation at 200-language scale, not just a larger Transformer. It combines community-informed requirements, language identification, cleaned monolingual data, LASER3 bitext mining, human seed data, FLORES-200, a 54.5-billion-parameter sparsely gated mixture-of-experts model, human quality assessment, and Toxicity-200. The system evaluates 40,602 directions among 204 language varieties. Official NLLB artifacts Work: Validate every metric or judge against blinded humans, adversarial edits, target-domain errors, and system-order controls.
Exit evidence: Publish per-dimension failure categories, judge agreement, bias, and uncertainty—not only correlation.
2026
Iterative value-guided decoding
Evolutionary Guided Decoding alternates value exploration with on-policy self-refinement to reduce policy/value distribution mismatch, at the cost of repeated generation and fitting.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 07.01 Evolutionary Guided Decoding: Iterative Value Refinement for LLMsPeer-reviewed ACL 2026 long paper and Outstanding Paper award recipient. Evolutionary Guided Decoding makes the value function evolve with the policy it guides. It samples multiple continuations, uses a task reward to label every prefix with its trajectory outcome, fits a prefix-value model, then decodes a stronger policy by multiplying base next-token probabilities with exponentiated predicted value. The improved policy generates the next training generation, reducing the stale-distribution problem that limits one-shot value-guided search. Earlier author manuscript Work: Compare base decoding, static value guidance, evolutionary exploration, and on-policy self-refinement with the same model, prompts, candidate budget, value data, and final evaluator.
Exit evidence: Report quality by iteration, number of generated candidates, value-model refits, wall time, calibration, evaluator agreement, diversity, and regressions outside the optimized domain.
2026
Residual language transfer across teacher errors and tokenizers
Residual-learning LLM KD subtracts projected teacher state only where the teacher top prediction is wrong; ACTD maps top-k teacher tokens to student anchors and regularizes probability outside them.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 08.01 Knowledge Distillation for Large Language Models through Residual LearningPublished at ICLR 2026; no official implementation was located by 2026-09-02. This ICLR 2026 method distills LLMs by emphasizing tokens where the teacher errs and representing the student's hidden state relative to a compressed teacher state. A pretrained teacher autoencoder, adaptive scaling, cross-tokenizer alignment, and mixture-of-experts fusion let residual supervision operate even when teacher and student vocabularies differ. OpenReview record · First-party Amazon Science page 08.02 ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual RegularizationThe arXiv record reports acceptance to the EMNLP 2026 main conference. No final ACL Anthology proceedings record was available at the 2026-09-02 audit, so methods and results remain pinned to arXiv v1 submitted 30 August 2026. ACTD distills reasoning models across different tokenizers by mapping teacher top-k tokens into student-space anchors. An anchor L1 term matches probability on aligned tokens, while residual regularization constrains the student's probability mass outside those anchors. With multiple teachers, ACTD computes a complete teacher-specific loss for each teacher and averages those losses; it does not average anchor signals or teacher distributions first. Official implementation Work: Record vocabulary map, token alignment, teacher-error mask, projector, divergence, and free-running evaluation.
Exit evidence: Compare same- and cross-tokenizer teachers on generation, reasoning, calibration, and total teacher-state cost.
2026
Token-level direction and teacher consensus
BinaryOPD tests the sign of sampled-token teacher/student probability differences. C-MOPD resolves agreement and disagreement between task-specialist teachers rather than blindly averaging their signals.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 09.01 Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?Preprint checked 2026-10-02. Empirical results concern autoregressive math and code distillation; the title is not a general theorem eliminating KL from all distillation. BinaryOPD replaces the magnitude of a teacher-student log-probability reward with its sign. The paper then uses high-disagreement token analysis to motivate C-MOPD, which combines teachers while declining strongly conflicting updates. Versioned full text · Author implementation Work: Which disagreement carries useful information, and when should conflicting teachers produce no update?
Exit evidence: Tested math/code token objectives are not a theorem that KL magnitudes or continuous feature residuals can always be discarded. Multiple teacher queries remain a training cost.
Research branches
Machine translation
Are adequacy, terminology, style, and named entities preserved in both directions?
Neural Machine Translation by Jointly Learning to Align and Translate · Neural Machine Translation of Rare Words with Subword Units · COMET: A Neural Framework for MT Evaluation · Scaling neural machine translation to 200 languages
Summarization
Is compression faithful, salient, non-redundant, useful, and consistently judged?
ROUGE: A Package for Automatic Evaluation of Summaries · BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension · PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization · SummEval: Re-evaluating Summarization Evaluation · G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Autoregressive distillation
Are targets sequences, forward/reverse/skew divergences, adaptive off-policy mixtures, or teacher distributions on student prefixes?
Sequence-Level Knowledge Distillation · MiniLLM: Knowledge Distillation of Large Language Models · DistiLLM: Towards Streamlined Distillation for Large Language Models · On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes · TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models · Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation
Metric and judge validity
Which human construct and error distribution does the metric or judge represent?
BLEU: a Method for Automatic Evaluation of Machine Translation · CIDEr: Consensus-Based Image Description Evaluation · SPICE: Semantic Propositional Image Caption Evaluation · BERTScore: Evaluating Text Generation with BERT · COMET: A Neural Framework for MT Evaluation · SummEval: Re-evaluating Summarization Evaluation · G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Iterative value-guided decoding
Does iterative exploration reduce policy–value distribution mismatch enough to justify repeated generation and fitting?
Evolutionary Guided Decoding: Iterative Value Refinement for LLMs
Cross-tokenizer residual transfer
How is teacher information aligned when token boundaries and vocabularies differ?
Knowledge Distillation for Large Language Models through Residual Learning · ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
Token-level direction and teacher consensus
Which disagreement carries useful information, and when should conflicting teachers produce no update?
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Comparison table
| Family or stage | Scientific distinction | Main risk | Anchors |
|---|---|---|---|
| Training objective | Scores reference tokens under teacher forcing or corruption reconstruction. | Lower loss need not improve factual or human utility. | 2020 · 2020 |
| Distillation distribution and state policy | Matches sequences or teacher distributions under fixed, adaptive, reverse/skew-divergence, off-policy, or student-sampled states. | State policy, divergence, and importance weighting can trade sample quality against coverage while adding teacher/student compute or estimator variance. | 2016 · 2024 · 2024 · 2024 · 2025 · 2025 |
| Surface overlap | Compares generated and reference token sequences. | Valid paraphrases score poorly and copied errors can score well. | 2002 · 2004 |
| Caption consensus / propositions | Weights reference consensus with TF–IDF or compares parsed object–attribute–relation tuples. | Neither metric sees the image; reference omissions, parser errors, and unsupported but overlapping text remain. | 2015 · 2016 |
| Learned metric / LLM judge | Uses contextual representations, human-rated data, or prompted model judgments. | Domain bias, position bias, verbosity preference, and adversarial sensitivity transfer into the score. | 2020 · 2020 · 2021 · 2023 |
| Iterative value-guided decoding | Evolutionary Guided Decoding alternates value-guided exploration with on-policy self-refinement rather than treating a fixed value model as a static reranker. | Extra sampling and evaluator overfitting can be reported as an algorithmic gain while compute, diversity loss, and off-domain regressions remain hidden. | 2026 |
| Residual language transfer across teacher errors and tokenizers | Contrasts hidden-state cross-model attention with anchor-probability mapping. | Approximate alignment and benchmark-specific teacher errors can produce brittle gains. | 2026 · 2026 |
| Token-level direction and teacher consensus | BinaryOPD tests the sign of sampled-token teacher/student probability differences. C-MOPD resolves agreement and disagreement between task-specialist teachers rather than blindly averaging their signals. | Tested math/code token objectives are not a theorem that KL magnitudes or continuous feature residuals can always be discarded. Multiple teacher queries remain a training cost. | 2026 |
Concept lineage
Neural Machine Translation by Jointly Learning to Align and TranslateBART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Task-trained encoder–decoder attention becomes a general denoising pretraining architecture for generation.
Sequence-Level Knowledge DistillationOn-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
A fixed set of teacher-decoded targets becomes teacher distribution matching on prefixes sampled by the current student.
MiniLLM: Knowledge Distillation of Large Language ModelsDistiLLM: Towards Streamlined Distillation for Large Language Models
Reverse-divergence generative distillation is complemented by skew-KL and an adaptive off-policy proposal that balances teacher and student samples.
BLEU: a Method for Automatic Evaluation of Machine TranslationCOMET: A Neural Framework for MT Evaluation
Fixed overlap scoring is complemented by a learned evaluator trained against human translation judgments.
BLEU: a Method for Automatic Evaluation of Machine TranslationCIDEr: Consensus-Based Image Description Evaluation
Uniform n-gram overlap becomes image-specific consensus weighting through TF–IDF over multiple descriptions.
CIDEr: Consensus-Based Image Description EvaluationSPICE: Semantic Propositional Image Caption Evaluation
Consensus-weighted phrase overlap is complemented by parsed object, attribute, and relation propositions with diagnostic F-scores.
SummEval: Re-evaluating Summarization EvaluationG-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Human meta-evaluation of summarization dimensions becomes a test bed for prompted LLM judges using explicit criteria and structured scoring.
Failure modes to audit
- Selecting decoding settings on the test set.
- Training only on fixed prefixes and evaluating only with teacher forcing.
- Using one reference as complete semantic truth.
- Reporting metric gains below human-judgment noise.
- Ignoring hallucination and source attribution.
- Letting an LLM judge know system identity.
Questions for every paper
- What distribution does decoding sample?
- Which quality dimensions are measured?
- How were references and judges produced?
- Which errors are hidden by averages?
Complete maintained paper route
Section titled “Complete maintained paper route”26 canonical readings · 2002–2026· route reviewed through October 2, 2026.
This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.
Show all 26 readings in maintained order
BLEU: a Method for Automatic Evaluation of Machine Translation
Kishore Papineni et al.
Metadata checked
ROUGE: A Package for Automatic Evaluation of Summaries
Chin-Yew Lin
Metadata checked
CIDEr: Consensus-Based Image Description Evaluation
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh
Official artifacts: Official CVF PDF
Metadata checked
Neural Machine Translation by Jointly Learning to Align and Translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio
Metadata checked
Sequence-Level Knowledge Distillation
Yoon Kim and Alexander M. Rush
Official artifacts: Official seq2seq implementation · Official mobile demonstration
Metadata checked
SPICE: Semantic Propositional Image Caption Evaluation
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould
Official artifacts: Author manuscript · Authors' implementation
Metadata checked
Neural Machine Translation of Rare Words with Subword Units
Rico Sennrich, Barry Haddow, and Alexandra Birch
Metadata checked
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu
Official artifacts: Official code
Metadata checked
COMET: A Neural Framework for MT Evaluation
Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie
Official artifacts: Official code
Metadata checked
Multilingual Denoising Pre-training for Neural Machine Translation
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer
Official artifacts: Official fairseq implementation and checkpoints
Metadata checked
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Mike Lewis et al.
Metadata checked
BERTScore: Evaluating Text Generation with BERT
Tianyi Zhang et al.
Metadata checked
SummEval: Re-evaluating Summarization Evaluation
Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev
Official artifacts: Official toolkit and annotations
Metadata checked
The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzmán, and Angela Fan
Official artifacts: Official FLORES repository
Metadata checked
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu
Official artifacts: Official implementation
Metadata checked
Scaling neural machine translation to 200 languages
Marta R. Costa-jussà et al.
Official artifacts: Official NLLB artifacts
Metadata checked
Discrete Flow Matching
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman
Metadata checked
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem
Metadata checked
MiniLLM: Knowledge Distillation of Large Language Models
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang
Official artifacts: Official code
Metadata checked
DistiLLM: Towards Streamlined Distillation for Large Language Models
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun
Peer-reviewed ICML 2024 proceedings paper.
Official artifacts: Official code
Metadata checked
TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
Makoto Shing, Kou Misaki, Han Bao, Sho Yokoi, and Takuya Akiba
Official artifacts: Official code
Metadata checked
Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation
Sungmin Cha and Kyunghyun Cho
Official artifacts: Official code
Metadata checked
Evolutionary Guided Decoding: Iterative Value Refinement for LLMs
Zhenhua Liu, Lijun Li, Ruizhe Chen, Yuxian Jiang, Tong Zhu, Zhaochen Su, Wenliang Chen, and Jing Shao
Peer-reviewed ACL 2026 long paper and Outstanding Paper award recipient.
Official artifacts: Earlier author manuscript
Metadata checked
Knowledge Distillation for Large Language Models through Residual Learning
Thinh On, Hengzhi Pei, Leonard Lausen, and George Karypis
Published at ICLR 2026; no official implementation was located by 2026-09-02.
Official artifacts: OpenReview record · First-party Amazon Science page
Metadata checked
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
Huiyi Zhang, Zijian Li, Xiaocheng Feng, Weitao Ma, Xiaoliang Yang, Yichong Huang, and Bing Qin
The arXiv record reports acceptance to the EMNLP 2026 main conference. No final ACL Anthology proceedings record was available at the 2026-09-02 audit, so methods and results remain pinned to arXiv v1 submitted 30 August 2026.
Official artifacts: Official implementation
Metadata checked
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Wenze Lin, Jiyuan Long, Jiale Zhao, Shenzhi Wang, Xitai Jiang, Ce Luo, Rui Lan, Qianli Ma, Fukang Wen, Hui Wu, Liyuan Chen, Shuoling Liu, Jiangpeng Yan, and Gao Huang
Preprint checked 2026-10-02. Empirical results concern autoregressive math and code distillation; the title is not a general theorem eliminating KL from all distillation.
Official artifacts: Versioned full text · Author implementation
Metadata checked