Writing study: Efficient Resource-Constrained Training of Transformers via Subspace Optimization

ICLR overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICLR 2026 oral writing corpus

Le-Trung Nguyen, Enzo Tartaglione, Van-Tam Nguyen · Official ICLR 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Learning & optimization algorithms (primary) · Inference, algorithms & systems efficiency.

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

21 physical pages

Before references / appendix

≈ 9.36 page-equivalents

Main figures / tables

8 / 0

Attached appendix figures / tables

4 / 4

Attached appendix span

≈ 8.89 page-equivalents

Before references / appendix: ≈ 44.57%References / intervening material: ≈ 13.1%Attached appendix: ≈ 42.33%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1 INTRODUCTION	1	≈ 1.19 pp.	5.67%
2 RELATED WORKS	2	≈ 0.95 pp.	4.52%
3 METHOD	3	≈ 2.94 pp.	14%
4 EXPERIMENTS	6	≈ 3.14 pp.	14.95%
5 CONCLUSION	9	≈ 0.51 pp.	2.43%
REPRODUCIBILITY STATEMENT	10	≈ 0.11 pp.	0.52%
REFERENCES	10	≈ 2.74 pp.	13.05%
A ADDITIONAL THEORETICAL DETAILS	13	≈ 4.52 pp.	21.52%
B ADDITIONAL EXPERIMENTAL DETAILS	17	≈ 2.91 pp.	13.86%
C LIMITATIONS	20	≈ 0.12 pp.	0.57%
D LLM USAGE	20	≈ 1.35 pp.	6.43%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Figure 2 · p. 5
• Figure 3 · p. 7
• Figure 4 · p. 7
• Figure 5 · p. 8
• Figure 6 · p. 8
• Figure 7 · p. 9
• Figure 8 · p. 9

Attached appendix

• Figure 10 · p. 18
• Figure 11 · p. 18
• Figure 9 · p. 18
• Table 1 · p. 19
• Figure 12 · p. 19
• Table 2 · p. 20
• Table 3 · p. 20
• Table 4 · p. 21

Argument and organization

1. Memory bottlenecks and decomposition precedents

   Separate weight, activation and deployment costs.

2. Stable subspaces and joint iteration

   Introduce an empirical stability hypothesis, weight subspace iteration and activation compression.

3. Hypothesis tests, tradeoffs and devices

   Move from rank diagnostics to downstream curves and measured edge-device latency.

Problem, claim and contribution

The introduction distinguishes reducing trainable parameters from reducing saved activations or inference work. This taxonomy clarifies the specific gap claimed for joint weight-activation compression.

Sections 1–2, pages 1–3 · Main paper

Argument and section order

The experiments first inspect singular spectra and compare repeated SVD with subspace reuse, then evaluate task accuracy. This separates evidence for the motivating hypothesis from evidence for practical utility.

Sections 3.3 and 4.2, pages 5–7 · Main paper

Figures, captions and visual explanation

Figure 1 distinguishes forward, backward and compressed-activation paths by arrow style and color. Figures 5–6 repeat four aligned resource axes, making tradeoffs easier to compare without reducing efficiency to one headline ratio.

Figure 1, page 1; Figures 5–6, page 8 · Main paper

Tables, comparisons and evidence

Raspberry Pi latency and Jetson energy measurements complement FLOP accounting. Main savings chiefly concern selected linear layers; TinyLlama measurements explicitly cover only the layers being fine-tuned.

Sections 4.1–4.4, pages 6–9; Tables 2–4, pages 20–21 · Main paper

Appendix and reproducibility

The appendix provides tensor contractions, rank selection, cost terms and hyperparameters, but the written WASI rank objective omits the stated perplexity constraint. Reproduction therefore requires reconciling equations and implementation.

Appendix A.1–A.3, pages 13–16; Equation 32, page 15 · Main paper

Clarity and writing conventions

A negative convolutional-model result is retained: high-rank factors can increase memory. This usefully qualifies the broad portability claim; the reported seed variance is also limited to one Pets experiment.

Appendix B.2–B.3, pages 18–19 · Main paper

What to reuse—and what not to infer

Transferable lesson: Build an efficiency argument in layers: identify the stored tensors, test the structural assumption, show accuracy-resource frontiers, then measure real hardware and disclose where compression fails.

Boundary: Do not interpret selected-layer analytical memory ratios as whole-device peak-memory savings or stable rank as proof of stable subspace orientation. The formulas contain implementation ambiguities, and most experiments use one seed.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-09-19

Reading status

Full text + selected visuals inspected

Main text

Pages 1–10, Sections 1–5 and reproducibility

Appendix / supplement

Pages 13–21, Appendices A–D and all four supplementary numerical tables

Visually inspected PDF pages

1, 8, 20

• ICLR 2026 final proceedings · 21 pages