SlackSwap combines data transfers and recomputation in GPU-limited training
Researchers at UC Merced and the University of Texas at Austin introduced SlackSwap, a planner for deep-learning training. It recomputes some activations when the transfer budget between a GPU and host memory is exceeded; reported results come from simulation, not hardware measurements.
Source review period: 2026-10-09
Published: · Updated:
The source does not state the event date; the period shown is the source review period.
Intermediate activations saved for backpropagation can exceed GPU memory during neural-network training. SlackSwap profiles a training step and treats the transfer time that can be hidden during computation as a budget: it moves activations that are costly to rebuild within that budget and recomputes the rest.
In a discrete-event simulation parameterized for an A100 and PCIe 4.0, with the activation-memory budget set to half the no-memory-limit GPU’s peak, SlackSwap retained 96.3% of ideal throughput for BERT-Large, 97.6% for GPT-2 XL and 99.6% for a Llama-7B-style model. The authors say all results are simulated and that effects such as allocator fragmentation are not modeled; these are not measurements from physical GPU training.
Data
Simulated training throughput
Results are from simulation, not a run on a physical GPU.
| BERT-Large | 96.3 |
| GPT-2 XL | 97.6 |
| Llama-7B-style | 99.6 |
Methodology
The study evaluates three transformer workloads at a 50% activation-memory budget in a discrete-event simulation parameterized for an A100 and PCIe 4.0.
Updated:
Subjects and places
Subject proposals
- Science and technologySuggested
- Preserved editorial category/topic; broad candidate onlyarxiv.org ↗
Geographic references
Only source-provided geographic references are shown. Opening an external map is optional. Optional story maps use OpenStreetMap. Opening a map sends your IP address and this site’s address to the map provider; location data remains readable as text without loading the map.
- United States · Place role: Coverage · Place type: Country or area · Location precision: RegionalEvidence
- Explicit preserved editorial country/region; no city or national centroid inferredarxiv.org ↗