Writing study: In-Place Test-Time Training

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Wenhao Huang, Di He, Tianle Cai · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Learning & optimization algorithms (primary) · Architecture & representation design.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

21 physical pages

Before references / appendix

≈ 10.43 page-equivalents

Main figures / tables

4 / 3

Attached appendix figures / tables

0 / 5

Attached appendix span

≈ 5.89 page-equivalents

Before references / appendix: ≈ 49.67%References / intervening material: ≈ 22.29%Attached appendix: ≈ 28.05%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.17 pp.	5.57%
2 PRELIMINARY: TEST-TIME TRAINING	2	≈ 1.08 pp.	5.14%
3 IN-PLACE TEST-TIME TRAINING	3	≈ 3.5 pp.	16.67%
4 EXPERIMENTS	7	≈ 3.42 pp.	16.29%
5 CONCLUSION	10	≈ 0.32 pp.	1.52%
ETHICS STATEMENT	11	≈ 0.11 pp.	0.52%
REPRODUCIBILITY STATEMENT	11	≈ 0.22 pp.	1.05%
REFERENCES	11	≈ 4.68 pp.	22.29%
A RELATED WORK	16	≈ 0.75 pp.	3.57%
B PROOF OF THEOREM 1	16	≈ 1.47 pp.	7%
C CONTEXT PARALLEL ALGORITHM FOR IN-PLACE TTT	18	≈ 0.4 pp.	1.9%
D EXPERIMENT DETAILS	18	≈ 2.66 pp.	12.67%
E USAGE OF LLMS	21	≈ 0.62 pp.	2.95%

All detected figure and table locations

Main text

• Figure 1 · p. 6
• Table 1 · p. 7
• Table 2 · p. 8
• Figure 2 · p. 8
• Table 3 · p. 9
• Figure 3 · p. 9
• Figure 4 · p. 10

Attached appendix

• Table 4 · p. 19
• Table 5 · p. 19
• Table 6 · p. 19
• Table 7 · p. 20
• Table 8 · p. 20

Argument and organization

1. Three barriers and matching design choices

   Maps architectural compatibility, parallelism, and objective mismatch to in-place MLP updates, chunks, and predictive targets.

2. Three experimental questions

   Separates adaptation of pretrained models, training from scratch, and component ablations.

Problem, claim and contribution

The introduction and preliminary desiderata use the same three-part organization, making each proposed component answer a named limitation.

Sections 1–2 · Main paper

Argument and section order

The theory is placed after the objective definition and before implementation, and its assumptions isolate an induction-style retrieval setting rather than claiming a universal learning guarantee.

Sections 3.2–3.4; Appendix B · Main paper

Figures, captions and visual explanation

The architecture diagram preserves attention and residual paths while highlighting the apply-then-update fast-weight cycle and its separate target branch.

Figure 1, p. 6 · Main paper

Tables, comparisons and evidence

Research questions directly label the experimental subsections. The RULER table distinguishes trained context lengths from extrapolation and includes a matched continually trained baseline.

Section 4; Table 1, p. 7 · Main paper

State size, chunk size, objective components, throughput, and memory are reported separately, allowing the reader to inspect accuracy and implementation cost rather than conflating them.

Figures 3–4, pp. 9–10 · Main paper

Appendix and reproducibility

The appendix documents important non-headline details: hidden-state targets for continual pretraining, document resets, near-zero initialization, and inference-time update clipping.

Appendices C and D.1–D.3 · Main paper

What to reuse—and what not to infer

Transferable lesson: Use an explicit barrier-to-design-to-experiment mapping, while keeping theoretical assumptions and practical stabilization rules visible in the same narrative.

Boundary: Drop-in means compatible with a pretrained architecture, not training-free: the 4B study uses about 35B additional tokens. Attention remains present and fast weights reset between documents, so lifelong memory is not demonstrated. Future-target wording, causal padding, and the exclusive prefix-sum convention require implementation-level clarification.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

PDF pp. 1–11, Sections 1–5 plus ethics and reproducibility statements.

Appendix / supplement

PDF pp. 16–21, Appendices A–E, including proof, context-parallel pseudocode, datasets, training, inference clipping, and initialization.

Visually inspected PDF pages

6, 7, 10

• ICLR 2026 official proceedings PDF, retrieved 2026-09-19 · 21 pages