Transformers and language-model pretraining
Masked, autoregressive, replaced-token, and infilling objectives create different code capabilities.
Code generation and software engineering
Code retrieval, function completion, competition problem solving, repository repair, and multi-turn dependency reuse are different scientific tasks. Their scores are not comparable without the execution harness, test coverage, sample budget, model cutoff, benchmark release, context policy, and failure rules.
The route now makes two often-collapsed benchmark steps explicit: APPS broadens execution-based evaluation across introductory, interview, and competition problems, while BigCodeBench tests longer instructions and composition across standard and external Python libraries. Neither partial test success nor dense branch coverage turns unit tests into a complete specification of correctness, safety, or maintainability.
12 maintained papers · literature reviewed through 2 October 2026
Code intelligence is not one task. Representation learning asks whether code and language share useful structure; synthesis asks whether a completion executes; competitive programming adds search and selection; code foundation models add infilling, instruction following, and context; repository repair adds localization, environment reconstruction, and regression testing; dynamic and multi-turn benchmarks probe contamination and dependency reuse. Read metrics only together with task unit, sampling budget, execution harness, time split, and allowed context.
Prerequisite contract
Evaluation, leakage, and uncertainty
Execution tests, sampling estimators, temporal splits, and repository environments define what a score means.
How the field developed
2020
Joint code–language representation
CodeBERT combines masked-language modeling with replaced-token detection over paired documentation and code plus unimodal code. It supports retrieval and understanding, but its token-sequence encoder does not explicitly model program syntax or execution.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 01.01 CodeBERT: A Pre-Trained Model for Programming and Natural Languages CodeBERT learns one bidirectional representation space for natural-language documentation and source code. It keeps RoBERTa's masked-language objective on paired text–code examples and adds replaced-token detection that can also exploit unimodal code: small generators propose plausible replacements, while the main encoder decides which tokens are original. The result is an understanding-oriented encoder, not an executable program generator. Official implementation and checkpoints Work: Reproduce code search and distinguish representation quality from generated-code correctness.
Exit evidence: Explain the two pretraining objectives and the role of bimodal versus unimodal data.
2021
Broad executable programming benchmark
APPS turns unrestricted natural-language programming specifications into a 10,000-problem execution benchmark. Its balanced train/test split, introductory/interview/competition strata, many human solutions, and per-problem test cases separate partial test-case success from strict all-tests correctness.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 02.01 Measuring Coding Challenge Competence With APPS APPS is a 10,000-problem execution benchmark for translating unrestricted natural-language specifications into Python programs. It spans introductory, interview, and competition difficulty, supplies human solutions and problem-specific tests, and evaluates behavior rather than surface overlap. Its key distinction is between passing some tests and strict correctness on every available test; neither should be confused with formal verification. Official implementation, evaluation harness, and data access · Author manuscript Work: Reproduce test-case average and strict accuracy on one frozen APPS split, with sandboxing and difficulty-stratified errors.
Exit evidence: Explain why partial test success, strict accuracy, and best-of-five selection answer different questions.
2021
Functional generation and pass@k
Codex scales autoregressive code modeling and introduces HumanEval: 164 handwritten Python tasks scored by hidden unit tests. The pass@k estimator measures whether at least one of k sampled programs is correct, so sampling budget and sandboxing are part of the claim.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 03.01 Evaluating Large Language Models Trained on Code Codex adapts a large autoregressive language model to public source code, then evaluates sampled Python completions by executing unit tests instead of comparing surface text. HumanEval contributes 164 handwritten function problems with signatures, docstrings and hidden tests; pass@k asks whether at least one of k sampled programs is functionally correct. The key conceptual shift is that code generation is judged as executable behavior, while sampling diversity becomes part of the measured system. Official HumanEval harness Work: Implement the unbiased pass@k estimator and execute untrusted generations in an isolated environment.
Exit evidence: Separate pass@1 from oracle selection among many samples and from security or formal correctness.
2022
Large-scale sampling, filtering, and clustering
AlphaCode turns competition solutions into a search-and-selection pipeline: sample many programs, reject failures on example tests, cluster surviving programs by behavior on generated inputs, and submit a small diverse set. The gain depends on both the model and a very large inference budget.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 04.01 Competition-Level Code Generation with AlphaCode AlphaCode treats competitive programming as a large-search problem around a learned generator. Transformer models produce an enormous, diverse pool of complete programs; example tests eliminate obvious failures; behavior on additional generated inputs clusters programs that appear to implement similar algorithms; and at most ten representatives are submitted. Its defining contribution is therefore not merely a larger code model, but a generate–filter–cluster–submit system that spends inference compute to turn a broad candidate distribution into a small portfolio. Author manuscript · Official CodeContests dataset Work: Hold the generator fixed and ablate example filtering, behavioral clustering, and sample count.
Exit evidence: Report candidates sampled, candidates executed, submissions allowed, compute, and temporal cutoff.
2023
Code foundation models, infilling, and long context
Code Llama continues training Llama 2 on code-heavy data, adds fill-in-the-middle training to selected sizes, Python-specialized and instruction-tuned variants, and long-context fine-tuning with a changed RoPE base. Model size, variant, prompt format, context regime, and license remain distinct choices.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 05.01 Code Llama: Open Foundation Models for Code Code Llama continues Llama 2 checkpoints on a predominantly code corpus, then branches the family by capability: general code completion, Python specialization, instruction following, fill-in-the-middle editing and longer-context inference. The work's central design is staged specialization rather than training an isolated architecture from scratch. Causal next-token learning remains the backbone, while data mixture, infilling serialization, long-context fine-tuning and instruction data determine which interface each checkpoint supports. Official implementation and model documentation Work: Compare base, Python, and Instruct variants on completion and infilling with identical prompts and decoding.
Exit evidence: Trace FIM serialization and state what long-context evidence does—and does not—show.
2024
Repository-scale issue resolution
SWE-bench restores a real repository before a human fix, gives the model an issue, applies its patch, and requires issue-revealing plus regression tests to pass. Localization, dependency installation, patch application, and test coverage become inseparable from generation.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 06.01 SWE-bench: Can Language Models Resolve Real-World GitHub Issues? SWE-bench turns a real GitHub issue, its repository at the pre-fix commit, and tests introduced by the resolving pull request into an executable repair task. A model must produce an applicable patch that makes every issue-specific fail-to-pass test succeed while preserving every previously passing regression test. Benchmark Work: Run one task end to end and audit retrieved context, environment failures, patch applicability, and both test groups.
Exit evidence: Distinguish a fully resolved issue from a patch that merely applies or passes incomplete tests.
2025
Complex instructions and library composition
BigCodeBench evaluates 1,140 function-level Python tasks that compose 723 calls from 139 libraries across seven domains. Complete supplies structured docstrings; Instruct removes non-essential code context to test natural-language instruction following. Dense tests and branch coverage strengthen execution evidence without making tests a complete specification.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 07.01 BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions BigCodeBench is an execution benchmark of 1,140 function-level Python tasks that require composing 723 function calls from 139 libraries across seven domains. BigCodeBench-Complete supplies structured docstrings and setup; BigCodeBench-Instruct removes non-essential detail to approximate natural-language requests. The paired variants expose whether a model can both use diverse libraries and recover precise constraints from a shorter instruction. Official benchmark implementation and evaluation harness · Official benchmark dataset · Author manuscript Work: Run the same checkpoint on Complete and Instruct with pinned benchmark, dependency, prompt, and calibration versions.
Exit evidence: Trace task construction, tool-call breadth, pass@1 calibration, and the interpretation of the Complete–Instruct gap.
2025
Dynamic, time-aware evaluation
LiveCodeBench continually collects newly released contest problems, attaches release dates, and evaluates generation, self-repair, code execution, and output prediction. Post-cutoff windows reduce likely exposure but do not prove absence of contamination.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 08.01 LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code LiveCodeBench makes release time a first-class benchmark variable. It continuously collects newly published contest problems, records each problem's date, and evaluates four code capabilities—generation, self-repair, code execution, and test-output prediction—so a model can be scored on problems released after its declared training cutoff. This design reduces likely benchmark exposure and reveals broader coding behavior; it cannot certify that every problem or semantic pattern was absent from training. Official benchmark implementation Work: Freeze one benchmark release, assign each model a declared cutoff, and reproduce scenario-specific metrics.
Exit evidence: Record release, date window, model version, prompt, execution policy, and sample count together.
2026
Iterative, dependency-aware codeflow
CodeFlowBench decomposes competitive and repository code into ordered function subproblems. At each turn the model must implement the next function while reusing prior implementations; fail-stop evaluation exposes error propagation across turns and dependency depth.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 09.01 CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation CodeFlowBench evaluates codeflow: implementing a complex program as an ordered sequence of functions while reusing implementations produced in earlier turns. Each turn exposes the next function signature and description plus already implemented dependencies. A fail-stop harness ends an instance after the first failed subproblem, and depth-aware metrics reveal how far correct reuse survives. The benchmark therefore measures iterative dependency management, not merely whether one monolithic completion passes. Official implementation and benchmark data Work: Compare the same model in single-turn and multi-turn modes while stratifying by turn count and dependency depth.
Exit evidence: Explain how provided functions, unit tests, fail-stop scoring, and dataset transformation constrain the result.
2026
Evaluate the controller around a coding worker
LoopArena decomposes agentic software work into a fixed worker and a controller that assigns, checks, continues, or stops.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 10.01 LoopArena: Benchmarking Models as Runtime Controllers for Loop EngineeringarXiv v1 submitted 28 August 2026. The exact title follows the current arXiv record, superseding an earlier discovery-feed paraphrase. Benchmark data and evaluation code were released; no peer-reviewed record was verified. Core idea: evaluate an LLM as a runtime Controller while holding the coding Worker fixed. After each round the Controller reads a structured progress packet, selects the next contract—implement, inspect, test, recover, or stop—and is scored at three execution depths. This separates loop guidance from the code-producing model's raw ability and exposes failures hidden by a single end-to-end success rate. Official benchmark and evaluation code Work: Replay identical repository states and worker capabilities while varying controller instructions only.
Exit evidence: Measure verified repository outcomes, not conversational plausibility, under a fixed execution budget.
2025–2026
Reasoning budgets, executable outputs and scientific edits
Test-time scaling studies how extra reasoning changes perception and mathematics; EmbedCopilot checks whether plausible embedded-system answers actually execute; GRADE requires domain-correct and readable image edits. Together they motivate measuring compute, implementation success and constraint satisfaction rather than judging prose alone.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 11.01 EmbedCopilot-Bench: Evaluating Vision-Language Models for Hardware-Aware Embedded System DevelopmentECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #9127 checked on 2026-09-12. Publisher version of record was not independently verified. The PDF uses EmbedCopilot-Bench; the program shortens this to EmbedCopilot. EmbedCopilot-Bench evaluates whether a vision-language assistant can turn hardware and software screenshots into useful embedded-development steps and functioning code. It combines rubric-based response quality with execution success, explicitly separating exact pin-grounded operation from behavior that works only after pin correction. Fluent instructions alone are insufficient. ECCV Spotlight program · Conference manuscript · Official project or implementation Work: What external test can falsify a persuasive explanation: a perception answer, a running circuit, or a scientifically valid image?
Exit evidence: More tokens can reduce perceptual accuracy. Judge scores and physical execution use different protocols; GRADE strict accuracy is a conjunction of three maxima, not its mean reasoning score.
2026
Token-level direction and teacher consensus
BinaryOPD tests the sign of sampled-token teacher/student probability differences. C-MOPD resolves agreement and disagreement between task-specialist teachers rather than blindly averaging their signals.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 12.01 Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?Preprint checked 2026-10-02. Empirical results concern autoregressive math and code distillation; the title is not a general theorem eliminating KL from all distillation. BinaryOPD replaces the magnitude of a teacher-student log-probability reward with its sign. The paper then uses high-disagreement token analysis to motivate C-MOPD, which combines teachers while declining strongly conflicting updates. Versioned full text · Author implementation Work: Which disagreement carries useful information, and when should conflicting teachers produce no update?
Exit evidence: Tested math/code token objectives are not a theorem that KL magnitudes or continuous feature residuals can always be discarded. Multiple teacher queries remain a training cost.
Research branches
Code representation
Does a representation retrieve and align code with language without claiming execution correctness?
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Program synthesis and foundation models
Which model, prompt, sampling, infilling, and selection regime produces executable programs?
Measuring Coding Challenge Competence With APPS · Evaluating Large Language Models Trained on Code · Competition-Level Code Generation with AlphaCode · Code Llama: Open Foundation Models for Code · BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Software-engineering workflows
Can the system navigate a repository, preserve regressions, and reuse dependencies across turns?
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? · CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation
Evaluation validity
How do execution, time, contamination, sample budget, and test coverage bound the score?
Measuring Coding Challenge Competence With APPS · Evaluating Large Language Models Trained on Code · SWE-bench: Can Language Models Resolve Real-World GitHub Issues? · BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions · LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code · CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation
Long-horizon software control
Does the controller request the verification needed before declaring completion?
LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
Reasoning budgets, executable outputs and scientific edits
What external test can falsify a persuasive explanation: a perception answer, a running circuit, or a scientifically valid image?
EmbedCopilot-Bench: Evaluating Vision-Language Models for Hardware-Aware Embedded System Development
Token-level direction and teacher consensus
Which disagreement carries useful information, and when should conflicting teachers produce no update?
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Comparison table
| Family or stage | Scientific distinction | Main risk | Anchors |
|---|---|---|---|
| Representation / retrieval | Ranks code or language pairs without executing a generated program. | A high retrieval score does not imply synthesis or repair ability. | 2020 |
| Function synthesis | Generates a self-contained solution checked by unit tests, from compact specifications through multi-library tasks. | pass@k changes with sampling; prompts and dependency environments differ; tests are incomplete specifications. | 2021 · 2021 · 2023 · 2025 |
| Competition search | Spans difficulty-stratified challenge suites and systems that use many candidates, filtering, and selection. | Difficulty mix, inference compute, hidden tests, and candidate selection can dominate apparent model quality. | 2021 · 2022 · 2025 |
| Repository engineering | Requires context localization, environment execution, patching, and regression preservation. | Environment and harness failures can be mistaken for reasoning failures, while limited tests can accept unsafe code. | 2024 · 2026 |
| Evaluate the controller around a coding worker | Attributes loop-level decisions separately from code-writing capability. | A fixed worker and harness define the result; success need not transfer to another tool stack. | 2026 |
| Reasoning budgets, executable outputs and scientific edits | Test-time scaling studies how extra reasoning changes perception and mathematics; EmbedCopilot checks whether plausible embedded-system answers actually execute; GRADE requires domain-correct and readable image edits. Together they motivate measuring compute, implementation success and constraint satisfaction rather than judging prose alone. | More tokens can reduce perceptual accuracy. Judge scores and physical execution use different protocols; GRADE strict accuracy is a conjunction of three maxima, not its mean reasoning score. | 2026 |
| Token-level direction and teacher consensus | BinaryOPD tests the sign of sampled-token teacher/student probability differences. C-MOPD resolves agreement and disagreement between task-specialist teachers rather than blindly averaging their signals. | Tested math/code token objectives are not a theorem that KL magnitudes or continuous feature residuals can always be discarded. Multiple teacher queries remain a training cost. | 2026 |
Concept lineage
CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesMeasuring Coding Challenge Competence With APPS
Code moves from representation transfer to a broad executable benchmark with explicit difficulty strata.
Measuring Coding Challenge Competence With APPSEvaluating Large Language Models Trained on Code
Both require execution, but APPS emphasizes scale and difficulty strata while HumanEval isolates handwritten function synthesis and pass@k.
CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesEvaluating Large Language Models Trained on Code
Code moves from a bidirectional representation target to autoregressive executable synthesis.
Evaluating Large Language Models Trained on CodeCompetition-Level Code Generation with AlphaCode
Functional testing is embedded in a much larger sample–filter–cluster search pipeline.
Evaluating Large Language Models Trained on CodeLiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
A fixed handwritten suite becomes an evolving, release-dated multi-scenario benchmark.
Code Llama: Open Foundation Models for CodeSWE-bench: Can Language Models Resolve Real-World GitHub Issues?
A code model becomes only one component inside localization, patch, environment, and regression workflows.
Evaluating Large Language Models Trained on CodeBigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
A compact single-function suite expands to longer instructions and explicit composition across standard and external libraries.
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex InstructionsLiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Richer tool-composition tasks complement, rather than replace, release-dated dynamic contamination control.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation
Repository execution expands from one issue patch to constructive multi-turn reuse of an explicit dependency chain.
Failure modes to audit
- Comparing pass@1, pass@10, and filtered multi-sample systems as if they used the same inference budget.
- Calling unit-test success formal correctness, security, maintainability, or deployment readiness.
- Ignoring model and benchmark dates when discussing contamination.
- Collapsing function synthesis, repository repair, and iterative dependency reuse into one capability.
- Mixing evolving leaderboard values with the frozen protocol described in a paper.
Questions for every paper
- What is the task unit: token, function, contest problem, issue patch, or dependency chain?
- What context, tools, tests, retries, and samples may the system use?
- Which execution environment and failure states define success?
- What model cutoff and benchmark release define the contamination claim?
- Which risks remain after every available test passes?
Complete maintained paper route
Section titled “Complete maintained paper route”12 canonical readings · 2020–2026· route reviewed through October 2, 2026.
This is the complete structured list for this chapter, ordered by an explicit prerequisite-aware teaching sequence. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.
Show all 12 readings in maintained order
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou
Official artifacts: Official implementation and checkpoints
Metadata checked
Measuring Coding Challenge Competence With APPS
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt
Official artifacts: Official implementation, evaluation harness, and data access · Author manuscript
Metadata checked
Evaluating Large Language Models Trained on Code
Mark Chen et al.
Official artifacts: Official HumanEval harness
Metadata checked
Competition-Level Code Generation with AlphaCode
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals
Official artifacts: Author manuscript · Official CodeContests dataset
Metadata checked
Code Llama: Open Foundation Models for Code
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve
Official artifacts: Official implementation and model documentation
Metadata checked
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez et al.
Official artifacts: Benchmark
Metadata checked
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro von Werra
Official artifacts: Official benchmark implementation and evaluation harness · Official benchmark dataset · Author manuscript
Metadata checked
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica
Official artifacts: Official benchmark implementation
Metadata checked
CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation
Sizhe Wang, Zhengren Wang, Dongsheng Ma, Yongan Yu, Rui Ling, Zhiyu Li, Feiyu Xiong, and Wentao Zhang
Official artifacts: Official implementation and benchmark data
Metadata checked
EmbedCopilot-Bench: Evaluating Vision-Language Models for Hardware-Aware Embedded System Development
Dongsheng Yuan, Yimo Deng, and Huangxun Chen
ECCV 2026 Spotlight verified in the official Vision-Language Models & Visual Foundation Models session; manuscript #9127 checked on 2026-09-12. Publisher version of record was not independently verified. The PDF uses EmbedCopilot-Bench; the program shortens this to EmbedCopilot.
Official artifacts: ECCV Spotlight program · Conference manuscript · Official project or implementation
Metadata checked
LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
Yi Wang, Haopeng Zhang, Chengxiang Huang, Rui Dai, Kaikui Liu, Piotr Koniusz, and Xiangxiang Chu
arXiv v1 submitted 28 August 2026. The exact title follows the current arXiv record, superseding an earlier discovery-feed paraphrase. Benchmark data and evaluation code were released; no peer-reviewed record was verified.
Official artifacts: Official benchmark and evaluation code
Metadata checked
Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Wenze Lin, Jiyuan Long, Jiale Zhao, Shenzhi Wang, Xitai Jiang, Ce Luo, Rui Lan, Qianli Ma, Fukang Wen, Hui Wu, Liyuan Chen, Shuoling Liu, Jiangpeng Yan, and Gao Huang
Preprint checked 2026-10-02. Empirical results concern autoregressive math and code distillation; the title is not a general theorem eliminating KL from all distillation.
Official artifacts: Versioned full text · Author implementation
Metadata checked