Writing study: Faster Activation Functions at the Edge for Post-Training Speedups

ICML overviewCorpus atlasBy contribution typeVisual galleryRules & templateWriting patternsPaper studies

← ICML 2026 oral writing corpus

Anton Lydike, Jun Bi, Jackson Woodruff · Official ICML 2026 oral record

Observed writing choices, not acceptance rules. This study examines the inspected manuscript version. It does not establish why the paper was accepted or that its presentation choices should be copied unchanged.

Figure, table and section ledger

Editorial contribution type: Efficiency, inference & systems (primary).

Layout extraction, not a required paper template. Counts are distinct caption labels; a multi-panel figure counts once. Section spans are estimated from heading positions, including floats and whitespace. Method and limitations.

Canonical PDF

13 physical pages

Before references / appendix

≈ 9.04 page-equivalents

Main figures / tables

8 / 4

Attached appendix figures / tables

0 / 0

Attached appendix span

No boundary detected; not assumed zero

Before references / appendix: ≈ 69.54%References / intervening material: ≈ 30.46%Estimated layout envelopes, not text/artwork area. Unresolved appendix boundaries remain within the undivided post-body material.

Extraction notes: No attached appendix boundary detected. Zero means no recognized caption, not independently verified absence.

Source-bound correction: Join wrapped printed heading text using adjacent source lines; original heading start coordinates are unchanged.

Section-by-section lengths and source pages

The title/abstract/front matter precede the first numbered heading. A span ends at the next detected top-level heading, not at the last sentence; end-page whitespace is included. References and statements are listed separately. These are not measurements of text-only or artwork area.

Top-level heading	Starts on PDF page	Estimated span	Share of entire PDF
1. Introduction	1	≈ 1.38 pp.	10.62%
2. Background and Motivation	2	≈ 0.4 pp.	3.08%
3. Approximations Based On Bit-Representations	3	≈ 1.7 pp.	13.08%
4. Implementation	4	≈ 1.29 pp.	9.92%
5. Evaluation	6	≈ 2.98 pp.	22.92%
6. Related Work	9	≈ 0.55 pp.	4.23%
7. Conclusion	9	≈ 0.19 pp.	1.46%
Impact Statement	9	≈ 0.07 pp.	0.54%
Acknowledgements	9	≈ 0.15 pp.	1.15%
References	10	≈ 3.96 pp.	30.46%

All detected figure and table locations

Main text

• Figure 1 · p. 1
• Table 1 · p. 2
• Figure 2 · p. 3
• Figure 3 · p. 4
• Figure 4 · p. 5
• Table 2 · p. 5
• Figure 5 · p. 5
• Figure 6 · p. 6
• Figure 7 · p. 7
• Figure 8 · p. 8
• Table 3 · p. 8
• Table 4 · p. 8

Attached appendix

No recognized caption labels.

Argument and organization

1. 1 Introduction; 2 Background

   Frames activation cost specifically on constrained hardware.

2. 3 Bit-Representation Approximations; 4 Implementation

   Derives primitives before compiler normalization, replacement, constant tuning and lowering.

3. 5 Evaluation; 6 Related Work; 7 Conclusion

   Separates activation microbenchmarks, model-stage runtime and post-training accuracy.

Problem, claim and contribution

The hardware argument explains why activation overhead matters more for small models without fast math instructions, rather than assuming a uniform bottleneck.

Introduction and Figure 1, p.1 · Main paper

Argument and section order

Bit-level derivations feed explicit rewrite rules and a constant-tuning stage before generated code and benchmarks.

Sections 3-4, pp.3-6 · Main paper

Figures, captions and visual explanation

The pipeline distinguishes training from inference substitution, and the accuracy page pairs function overlays with logarithmic error plots and task-result tables.

Figure 4, p.5; Figure 8 and Tables 3-4, p.8 · Main paper

Tables, comparisons and evidence

Microbenchmark gains are separated from smaller ConvNeXt-stage gains, and the post-training comparison reuses weights rather than retraining the approximate activation.

Sections 5.2-5.5, pp.6-7 · Main paper

Appendix and reproducibility

There is no appended supplement in this PDF; generated modules, tuning settings and the numerical failure case are placed in the main text.

Listing 1, p.6; Section 5.4, p.7; failure discussion, p.8 · Main paper

Clarity and writing conventions

The table retains a NaN anomaly-detection failure, and the accompanying text labels range-safe guard generation as future work rather than a solved limitation.

Table 3 and Section 5.5, p.8 · Main paper

What to reuse—and what not to infer

Transferable lesson: Separate primitive speed, model-stage speed and task accuracy, and leave failed deployment ranges visible beside favorable drop-in results.

Boundary: An approximation tuned near zero is not uniformly safe over all activation values; post-training replacement avoids weight retraining but still requires compiler constant tuning and range validation.

Apply this lesson using the argument and evidence worksheet.

Inspection record and sources

Checked

2026-10-07

Reading status

Writing-focused manuscript scan + selected visuals inspected

Main text

Writing-oriented scan of the main argument through physical p.9, with focused physical-page text inspection on 1, 3, 5, 6, 7, 8, 9, 13. Framing, section progression, mechanism/proof explanation, displays, comparison conditions and conclusions were inspected. This is not exhaustive technical reading, proof verification or reproduction.

Appendix / supplement

The published 13-page PDF has no appendix: physical pp.10-13 contain references. No separately linked supplement inspected.

Visually inspected PDF pages

5, 8

This is a writing-focused scan of the manuscript and the stated supplementary scope: framing, organization, visual explanation, evidence presentation and wording. It is not a line-by-line proof audit, reproduction of results or exhaustive scientific review.

• Published proceedings PDF, PMLR volume 306, retrieved 2026-10-07 · 13 pages