Writing study: ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Federico Danieli, Pau Rodriguez, Miguel Sarabia, Xavier Suau, Luca Zappella · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Inference, algorithms & systems efficiency (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

32 physical pages

Before references / appendix

≈ 10.54 page-equivalents

Main figures / tables

3 / 2

Attached appendix figures / tables

10 / 4

Attached appendix span

≈ 17.89 page-equivalents

Before references / appendix: ≈ 32.94%References / intervening material: ≈ 11.16%Attached appendix: ≈ 55.91%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Source-bound correction: ParaRNN: Table 3 is a real throughput table on physical page 20. Its side-wrapped caption is split between a line ending Table and a separate line beginning 3:. Visually checked page 20 and restored the caption at the first caption-label baseline.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.96 pp.	6.13%
2 PARALLEL APPLICATION OF NONLINEAR RNNS	3	≈ 2.28 pp.	7.12%
3 ADAPTING GRU AND LSTM FOR PARARNN	5	≈ 1.45 pp.	4.53%
4 EFFICIENT IMPLEMENTATION OF PARALLEL REDUCTION	7	≈ 0.51 pp.	1.59%
5 RESULTS	7	≈ 2.68 pp.	8.38%
6 CONCLUSION AND FUTURE WORK	10	≈ 0.65 pp.	2.03%
ACKNOWLEDGMENTS	11	≈ 0.12 pp.	0.38%
REPRODUCIBILITY STATEMENT	11	≈ 0.14 pp.	0.44%
ETHICS STATEMENT	11	≈ 0.18 pp.	0.56%
REFERENCES	11	≈ 3.57 pp.	11.16%
A NEWTON CONVERGENCE FOR PARAGRU AND PARALSTM	15	≈ 2 pp.	6.25%
B DETAILS ON PROFILING RESULTS	17	≈ 4 pp.	12.5%
C DETAILS ON LANGUAGE MODELING RESULTS	21	≈ 4 pp.	12.5%
D PARARNN SOFTWARE PACKAGE OVERVIEW	25	≈ 7 pp.	21.88%
E USE OF AI WRITING ASSISTANCE	32	≈ 0.89 pp.	2.78%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Figure 2 · p. 8
• Figure 3 · p. 9
• Table 1 · p. 9
• Table 2 · p. 10

Attached appendix

• Figure 4 · p. 15
• Figure 5 · p. 18
• Figure 6 · p. 18
• Figure 7 · p. 19
• Figure 8 · p. 19
• Table 3 · p. 20
• Figure 9 · p. 20
• Figure 10 · p. 21
• Table 4 · p. 22
• Table 5 · p. 22
• Figure 11 · p. 23
• Table 6 · p. 23
• Figure 12 · p. 27
• Figure 13 · p. 27

Argument and organization

1. Introduction / Prior work

   Positions scalable implementation as the contribution rather than inventing Newton-based recurrence solving.

2. Parallel application / Limitations / Cell adaptations

   Derives the solver and immediately states convergence and Jacobian constraints.

3. Implementation / Results

   Separates software abstraction, kernel performance and language-model quality.

4. Appendices A–E

   Turns proofs-of-feasibility into reproducible systems details and extensible interfaces.

Problem, claim and contribution

The introduction explicitly credits earlier Newton-based parallelization and defines the new contribution as scalable kernels, seven-billion-parameter experiments and a reusable library. This is a persuasive systems claim without requiring a newly invented mathematical primitive.

§1 contributions and prior work, pp.2–3 · Main paper

Argument and section order

Derivation proceeds from recurrence to nonlinear system, linearization and parallel reduction. A limitations subsection appears before adapted cells, making the diagonal/block-diagonal restriction a motivated tradeoff rather than a hidden implementation footnote.

§§2–3, Eqs.1–9, pp.3–7 · Main paper

Figures, captions and visual explanation

Figure 10 isolates the new recurrent cell inside a familiar projected, gated and residual block. Table 4 places width, depth, tuning, step time and memory together so readers can check whether apparently similar model sizes really use comparable configurations.

Appendix C.1, Figure 10 and Table 4, pp.21–22 · Main paper

Tables, comparisons and evidence

The evaluation deliberately separates synthetic expressivity, kernel timing, generation throughput and language quality. It acknowledges failure on the A5 task and Mamba2's lower perplexity, supporting feasibility rather than uniform state-of-the-art superiority.

§5, Tables 1–2, pp.8–10 · Main paper

Appendix and reproducibility

Profiling details disclose hardware, warm-up, compiled baselines and minimum rather than mean timings. The implementation appendix maps each mathematical operation to extensible classes, then follows thread, warp, block and grid levels with diagrams and examples.

Appendix B, p.17; Appendix D, pp.25–31 · Main paper

Clarity and writing conventions

Boxed appendix summaries orient readers through technical material. The standalone AI-writing disclosure specifies language assistance and human responsibility; its placement is an observed author choice, not a conference-mandated section template.

Appendices A–C, pp.15, 17, 21; Appendix E, p.32 · Main paper

What to reuse—and what not to infer

Transferable lesson: For systems papers, connect mathematical reformulation to hardware implementation, then evaluate correctness, runtime and task quality as distinct claims with explicit comparison units.

Boundary: The 665-times headline compares a cell implementation with naive sequential application, not complete Transformer training. Three Newton iterations are empirically adequate for the tested cells, not guaranteed for arbitrary recurrences. Synthetic results report best-of-three seeds; language-model hyperparameters are selected on the named SPJ test set. Do not turn these protocols into universal performance claims.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

PDF pp.1–10, §§1–6; statements and references pp.11–14

Appendix / supplement

PDF pp.15–32, Appendices A–E, including convergence, profiling, language-model training, implementation code and AI-writing disclosure

Visually inspected PDF pages

21, 22

• official ICLR 2026 proceedings PDF · 32 pages