Writing study: DTO-KD: Dynamic Trade-off Optimization for Effective Knowledge Distillation
← ICLR 2026 oral writing corpus
Zeeshan Hayder, Ali Cheraghian, Lars Petersson, Mehrtash Harandi, Richard Hartley · Official ICLR 2026 oral record
Figure, table and section ledger
Editorial contribution type: Learning & optimization algorithms (primary).
Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.
- Canonical PDF
- 13 physical pages
- Before references / appendix
- ≈ 10.11 page-equivalents
- Main figures / tables
- 4 / 5
- Attached appendix figures / tables
- 0 / 0
- Attached appendix span
- Separate file recorded below; not in this PDF total
Extraction notes: No attached appendix boundary detected. Zero means no recognized caption, not independently verified absence.
Section-by-section lengths and source pages
The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.
All detected figure and table locations
Main text
Attached appendix
No recognized caption labels.
Separate source files—excluded from canonical-PDF totals
A separate author version may repeat main text and references. Whole-file lengths and counts are not extra appendix material and are never added to the canonical PDF totals. Read the study's source scope for which sections received substantive reading versus layout inspection.
Additional source 1
Separate PDF (or ZIP containing it): whole-file layout of 3 physical pages; 0 detected figures and 0 tables. These may include overlapping main-text material.
Located appendix only: starts on physical p.1; ≈ 2.69 page-equivalents, 0 figure labels and 0 table labels. This interval is distinct from the whole file.
| Detected heading | PDF page | Estimated span |
|---|---|---|
| 1 APPENDIX | 1 | ≈ 2.69 pp. |
Argument and organization
- Two optimization failures
The introduction names gradient conflict and dominance, then promises a mechanism for both.
- Vector objective and implementation
Formalizes relative improvement, derives a two-objective direction, then introduces an amortized approximation.
- Tasks, ablations and limitations
Classification and detection tables precede component/teacher analyses and a data-availability limitation.
Problem, claim and contribution
The introduction gives two distinct bottlenecks memorable names, then reuses them in the method and gradient diagnostics. This makes the proposed optimization change easier to locate than a generic compression motivation alone.
Sections 1 and 3; Figure 1 · Main paper · Additional source 2
Argument and section order
A conventional weighted loss appears before the vector-objective formulation. However, the transition from the analytical two-gradient solution to an amortized weight update needs especially explicit signposting: these are not the same computational procedure.
Section 3.1, Equations 9–18 and Algorithm 1 · Main paper · Additional source 2
Figures, captions and visual explanation
Figure 2 separates the teacher/student feature path from the optimization inset and supplies a visual legend for features, projectors and heads. The graph is useful, but an inference-only inset would clarify what survives deployment.
Figure 2, main-paper page 4 · Main paper · Additional source 2
Tables, comparisons and evidence
Tables group student scales and distinguish homogeneous from heterogeneous teacher–student pairs. The two compact ablation tables answer component and teacher-choice questions separately. Their very dense page layout is a reminder to inspect tables at actual reading size.
Tables 2–5, main-paper page 9 · Main paper · Additional source 2
Appendix and reproducibility
The separate supplement expands the closed-form proof rather than providing the promised classification/detection loss definitions. A reproducibility map should name where such implementation details actually appear, and not promise appendix content that is absent from the inspected supplement.
Main-paper page 4; entire separate supplement, pages 1–3 · Main paper · Additional source 2
Clarity and writing conventions
The paper's general Pareto and non-collapse language is stronger than the printed derivation warrants. As an editorial check, two unit gradients at 120 degrees produce a half-length combined direction, contradicting the stated 1/√2 lower bound. Accepted status is not a substitute for checking mathematical qualifications.
Main Corollary 3.4; supplement Corollary 1.4 and its proof · Main paper · Additional source 2
What to reuse—and what not to infer
Transferable lesson: Name each optimization failure, show the corresponding mechanism, and give it a separate diagnostic or ablation. Keep an exact derivation, a practical approximation and their respective guarantees visibly distinct.
Boundary: Do not inherit an oral paper's guarantee language uncritically. The closed-form equality-constrained solution omits the simplex boundary case, the stated lower bound has a counterexample, and the amortized implementation needs its own evidence. Figure/table placement in an accepted PDF is also not an override of the official template.
Apply this lesson using the argument and evidence worksheet.
Inspection record and sources
- Checked
- 2026-09-19
- Reading status
- Full text + selected visuals inspected
- Main text
- All main-paper pages 1–13: Sections 1–6 and bibliography.
- Appendix / supplement
- Entire separate three-page supplement: closed-form derivation and Corollaries 1.2–1.5.
- Visually inspected PDF pages
- 4, 6, 9
Visually inspected supplement pages: 3.