A GPU cluster can have a healthy training fabric and still miss its checkpoint window. The useful buying question is not “How fast is this storage appliance?” It is “Can this complete training state reach recoverable storage before the deadline, while the other jobs keep running?” This guide provides a reusable sizing worksheet, a worked 256-GPU example, a bottleneck matrix and a deployment acceptance checklist.
Scope: Everything in the worked design is hypothetical. Bandwidth values are planning assumptions, not benchmark results or vendor performance promises. Calculations use decimal GB/TB and single-direction network rates. No GPU, storage or failure-injection tests were executed for this article; the sizing arithmetic was executed locally.
1. Define the checkpoint before sizing the network
Start with the object you must restore, not the GPU count. A weights-only export and a resumable training checkpoint are different procurement inputs. Ask the application owner to enumerate model tensors, optimizer state, any master-weight copies, scheduler and progress state, random-number state, and data-loader progress required by their recovery procedure. This is a proposed inventory, not a claim that every framework saves every item automatically.
PyTorch Distributed Checkpoint (DCP) supports saving and loading across multiple ranks, produces multiple files per checkpoint, and supports load-time resharding between cluster topologies.[5] Consequently, ask for both total bytes across unique shards and the file/object count. Do not multiply a global checkpoint estimate by every GPU unless the actual implementation writes that many complete replicas.
For a first estimate, define:
- P: parameter count.
- b: bytes of persisted tensor state per parameter, based on the chosen checkpoint format.
- E: additional saved state and serialization allowance in bytes.
- S = P × b + E: total bytes in one complete, logical checkpoint.
In this article's illustrative ledger, a 70-billion-parameter job saves 2 bytes of model weights, 4 bytes of master weights, and two 4-byte optimizer-state values per parameter: 14 bytes per parameter. This is an explicit modeling assumption, not a universal mixed-precision layout. Some implementations omit, reconstruct, transform, or replicate state. Inspect a representative saved checkpoint before approving capacity.
That hypothetical tensor state occupies 980 GB. A chosen 10% allowance makes the planning checkpoint 1,078 GB. If the real serializer writes materially more or less, replace the assumption rather than explaining the difference away. Count temporary files separately when they coexist with the committed generation.
2. Four clocks: pause, staging, completion and recovery
Do not put one “checkpoint time” column in the acceptance spreadsheet. Record four measurements: training-visible pause; completion of a consistent staged snapshot; completion of storage writing; and successful recovery from the committed checkpoint. Ask the storage supplier what its write acknowledgement actually protects against.
PyTorch's asynchronous-checkpoint recipe describes copying state into CPU buffers before background saving, with additional host-memory requirements related to checkpoint size per rank and rank count.[2] Its pinned-memory discussion also warns that holding persistent staging buffers increases memory pressure.[2] An asynchronous call is therefore not evidence that checkpointing consumes no CPU memory or that a recoverable copy already exists.
The documented asynchronous response distinguishes staging completion from upload completion.[5] Treat those as different observability points. Neither label by itself substitutes for the storage platform's documented power-loss, replication, and failure-domain guarantees. Record the writer, filesystem or object-store backend, and durability contract alongside the timestamp.
If the deployed release supports fully asynchronous staging, use that release's synchronization requirements before model state can be modified; the PyTorch recipe explicitly discusses waiting for staging before the optimizer changes parameters.[2] Do not copy a newer example into an older runtime without checking the installed API. A short pause obtained by saving inconsistent state is not a successful optimization.
3. Worked example: four jobs on 256 GPUs
Assume 32 servers with eight GPUs each. Four concurrent training jobs each occupy eight servers, or 64 GPUs. The server and GPU counts describe placement only; they are not specifications for a particular NVIDIA system, GPU generation, PCIe layout or NVLink domain.
Each job creates one 1,078 GB logical checkpoint, evenly sharded across its eight servers for this example. Every job has a 60-second storage-completion target and starts a checkpoint every 900 seconds. The worst planned burst is all four jobs starting together.
| Planning input or result | Value | What it means |
|---|---|---|
| Servers / GPUs per server | 32 / 8 | Hypothetical placement |
| Concurrent jobs / servers per job | 4 / 8 | Separate job state, not replicas |
| State per job, including allowance | 1,078 GB | Unique logical checkpoint bytes |
| Checkpoint bytes per server | 134.75 GB | Assumes balanced sharding |
| Storage-completion target | 60 s | Not merely the API return time |
| Payload throughput per job | 17.97 GB/s | Checkpoint bytes divided by deadline |
| Payload throughput per server | 2.25 GB/s | Balanced lower-level target |
| Four-job burst throughput | 71.87 GB/s | Simultaneous checkpoint writes |
| Average checkpoint write rate | 4.79 GB/s | Four checkpoints per 900 s |
| Design utilization ceiling | 70% | Chosen planning headroom, not measured efficiency |
| Required sustainable service rate | 102.67 GB/s | Burst target divided by 0.70 |
The difference between 4.79 GB/s average and 71.87 GB/s during the deadline window is the buying decision. A system sized only for the interval average might eventually drain the queue while consistently missing the recovery-point objective. The 102.67 GB/s design figure reserves headroom against a chosen service-rate ceiling; it is not a claim that all systems have 30% protocol overhead.
The 60-second arithmetic allocates the whole window to transfer. If measurement shows 10 seconds of non-overlapping planning, metadata and commit work, only 50 seconds remain: the four-job payload target becomes 86.24 GB/s, and the same headroom rule requires 123.20 GB/s. Measure the critical path before deciding whether a faster network will solve the deadline.
Network translation without double-counting
Suppose each server has a dedicated 100 Gb/s storage link. Its theoretical single-direction bit-to-byte ceiling is 12.5 GB/s before protocol and implementation effects. That exceeds the hypothetical 2.25 GB/s per-server payload requirement, but it says nothing about shared uplinks, storage servers, filesystem layout or competing reads.
For illustration, a 400 Gb/s aggregate-path unit corresponds to 50 GB/s theoretical single-direction capacity. The 102.67 GB/s service target requires at least three such units by simple division and rounding up. This is a link-capacity lower bound, not a three-port or three-switch bill of materials: you still need verified endpoint throughput, path distribution, fabric topology and a failure budget.
With three ideal 400 Gb/s units, losing one leaves 100 GB/s theoretical capacity. That remains above the burst payload target but below the chosen headroom-adjusted service target. Decide explicitly whether a degraded system must meet the original deadline, a relaxed deadline, or only preserve correctness. No network calculation here compares NVLink aggregate bandwidth with a NIC's single-direction rate.
4. Reusable sizing worksheet
Use these equations with measured checkpoint bytes. Keep all rates in the same units:
checkpoint_GB = parameters * saved_bytes_per_parameter / 1e9
checkpoint_GB *= 1 + extra_fraction
burst_payload_GBps = concurrent_jobs * checkpoint_GB / transfer_seconds
service_GBps = burst_payload_GBps / utilization_ceiling
average_GBps = concurrent_jobs * checkpoint_GB / interval_seconds
per_server_GBps = checkpoint_GB / servers_per_job / transfer_seconds
retained_usable_TB = concurrent_jobs * retained_generations * checkpoint_GB / 1000
Runnable arithmetic example, executed locally for this article: this uses only Python's standard library. It performs no network calls and writes no storage test data.
from math import ceil
parameters = 70_000_000_000
saved_bytes_per_parameter = 14 # hypothetical checkpoint layout
extra_fraction = 0.10
jobs = 4
servers_per_job = 8
transfer_seconds = 60
interval_seconds = 900
utilization_ceiling = 0.70
retained_generations = 3
size_gb = parameters * saved_bytes_per_parameter / 1e9
size_gb *= 1 + extra_fraction
burst = jobs * size_gb / transfer_seconds
service = burst / utilization_ceiling
print(f"Checkpoint per job: {size_gb:.2f} GB")
print(f"Burst payload: {burst:.2f} GB/s")
print(f"Service target: {service:.2f} GB/s")
print(f"Average: {jobs * size_gb / interval_seconds:.2f} GB/s")
print(f"400 Gb/s theoretical units: {ceil(service / (400 / 8))}")
print(f"Retained usable: {jobs * retained_generations * size_gb / 1000:.3f} TB")
Reject zero or negative deadlines when adapting this into a calculator, and restrict the utilization ceiling to greater than zero and at most one. If checkpoints have different sizes or deadlines, replace the equal-job multiplication with a time-window schedule. The sum of independent per-job requirements is a conservative simultaneous-burst model, not a simulation of actual arrival times.
5. Capacity, staging memory and retention are separate budgets
Keeping three complete generations for each of the four jobs requires 12.936 TB usable for retained checkpoints. Keeping those generations while all four next checkpoints are in progress raises the peak logical footprint to 17.248 TB. Leaving 20% of the provisioned usable space free raises that particular capacity target to 21.56 TB.
These numbers exclude datasets, logs, scratch files, filesystem metadata, snapshots and other tenants. “Usable” also does not mean raw drive capacity. Ask the supplier to translate the agreed usable requirement into raw media after its protection scheme, spares and operating constraints. Avoid applying an invented generic redundancy multiplier to every storage product.
The balanced example stages 134.75 GB per server for one full checkpoint image. If the implementation holds two equivalent generations in host memory simultaneously, that becomes 269.5 GB per server before application and operating-system memory. This is a planning bound, not a measured PyTorch allocation: actual buffers depend on the saved state and implementation. PyTorch recommends limiting concurrent checkpoint requests to control memory usage.[2]
If a burst buffer uses local NVMe, budget capacity per node, not merely cluster-wide free space. A failed node holding a unique shard can make the aggregate checkpoint unusable unless the chosen scheme provides protection elsewhere. For this design, declare local staging expendable and count a generation as recoverable only after its protected destination is committed and validated.
6. Choose a storage path, not a feature label
Shared filesystem as the first acceptance target
A shared filesystem is a reasonable candidate when the application writer expects files and operators want a common restore namespace. Evaluate it with the real shard count and concurrent jobs, not just a single large sequential stream. DCP's documented multiple-file output makes metadata behavior part of the checkpoint design.[5]
Ask how clients distribute writes across storage targets, how completion is coordinated, and what happens while a target rebuilds. These are procurement questions, not assumptions that every parallel filesystem uses the same layout. Obtain both per-client results and end-to-end checkpoint results under the agreed concurrency.
Local NVMe staging plus protected storage
Choose this alternative when a measured local write path reduces the training-visible disruption and the system can safely drain to a protected tier. Track the oldest undrained checkpoint and queue depth. For any sustained schedule, the long-run arrival rate must remain below the effective drain rate; otherwise a finite buffer eventually fills.
Do not present local staging completion as fulfillment of a rack-loss recovery objective. Test a node loss during transfer and confirm that the previous protected generation remains loadable. A tiering system needs explicit cleanup ownership, backpressure, retry behavior and a generation manifest.
Object storage as a backend or archive
Object storage is an alternative when the selected checkpoint writer supports it and its consistency, commit and restore behavior meet the application contract. Do not assume a filesystem path can simply be replaced with an object-store URI. Request a supported integration and demonstrate the complete save/load cycle, including interrupted uploads and retention cleanup.
Where GPUDirect Storage fits
NVIDIA GPUDirect Storage (GDS) enables a direct DMA path between GPU memory and storage that avoids a CPU bounce buffer; its documentation describes explicit cuFile APIs for this path.[4] That makes it relevant to an application whose actual storage data path supports GDS, not an automatic acceleration flag for every training checkpoint.
The PyTorch asynchronous recipe described above stages checkpoint state into CPU memory.[2] Therefore, do not assume installing GDS converts that writer into direct GPU-to-storage I/O. Validate the framework integration, filesystem support, driver compatibility and observed path for the exact deployment. NVIDIA also documents compatibility paths and cases where intermediate buffers are used.[4] Benchmark the implemented path against the same workload without it; a feature checklist cannot supply the missing measurements.
7. Why checkpoints are slow: bottleneck isolation matrix
This is a proposed diagnostic sequence. Change one factor at a time and keep the checkpoint contents, concurrency and cache conditions recorded.
| Symptom | First evidence to collect | Controlled next test |
|---|---|---|
| Training pauses before network traffic rises | Stage/planning timestamps, host memory and CPU activity | Compare staged bytes and pause with one job |
| One server finishes much later | Per-rank bytes, per-server transfer rate, locality | Check shard imbalance before replacing optics |
| Single job is fast, four jobs miss the window | Shared uplinks, target throughput, queue depth | Repeat synchronized and staggered starts |
| Fast payload write, slow final completion | File count, metadata/commit timestamps | Compare writer layouts supported by the application |
| Async saves become progressively slower | In-flight generations, drain time, buffer occupancy | Limit outstanding saves and observe queue recovery |
| Save succeeds but restart fails | Manifest, missing shards, logs, software versions | Load the committed generation in an isolated job |
| Training slows only during storage writes | Training step latency plus storage traffic timeline | Compare dedicated and shared paths under equal load |
Correlate application timestamps with NIC counters and storage telemetry. A busy interface is evidence of utilization, not proof that it is the limiting resource. Likewise, low average utilization can hide short saturation bursts. Keep the checkpoint window visible in the monitoring dashboard rather than averaging it over the whole training run.
8. Deployment and acceptance checklist
Use the following as a contractual test plan with thresholds filled in before purchase. The example deadline is a design target; no row below reports an executed cluster test.
- Inventory: record server layout, storage NICs, link rates, switch uplinks, storage endpoints, software versions and checkpoint writer configuration. Confirm separate training, storage and management paths where the design calls for them.
- Known checkpoint: produce a representative state and record its unique bytes, file count, shard distribution and restore requirements. Compare measured size with the 14-byte assumption before committing capacity.
- Single-job baseline: record pause, staging, write completion, commit and actual restore. Note whether reads hit client caches, server caches or backing media.
- Four-job burst: trigger the hypothetical concurrent workload and require every job to meet the agreed 60-second deadline. Report individual completion times, not only aggregate GB/s.
- Mixed workload: repeat with expected dataset reads and training communication active. Record application step-time impact as well as checkpoint throughput.
- Repeatability: run enough successive intervals to reveal accumulating backlog, memory growth or retention leakage. Define the run duration and acceptable tail latency with the application owner.
- Degraded path: in a maintenance lab or approved window, remove one agreed storage-network path or endpoint. Verify correctness and the pre-agreed degraded deadline; restore redundancy before continuing.
- Interrupted save: stop a disposable test job mid-checkpoint. Ensure recovery selects a complete generation and cleanup does not remove the last recoverable one.
- Recovery drill: restore on the intended replacement allocation. DCP supports load-time resharding, but the specific application state and deployment still need an actual restore test.[5]
- Handover: retain logs, plots, manifests, retention policy, alert thresholds, recovery commands and ownership. Do not accept screenshots of peak bandwidth in place of a recoverable checkpoint.
For operating alerts, prioritize checkpoint age, completion deadline violations, outstanding saves, free usable capacity and restore failures. Alert on the application objective before the device reaches a red utilization threshold. Preserve the previous known-good checkpoint until the new generation has passed the agreed commit validation; schedule periodic restore drills rather than equating file existence with recoverability.
Practical takeaways and next reading
Size the checkpoint's unique persisted state first. Separate burst bandwidth from interval average, logical usable capacity from raw drives, and training pause from recoverable completion. In the hypothetical design, the meaningful target is 71.87 GB/s of simultaneous payload writes, not the 4.79 GB/s interval average. Headroom, metadata time and degraded operation change the purchase requirement materially.
Use the worksheet to request evidence from suppliers, then replace assumptions with measurements from the actual writer. More NIC bandwidth is useful only when the end-to-end path and storage service can consume it. A short asynchronous API call and a successful file listing are not a recovery test.
Continue with the earlier AI GPU Cluster Network Calculator, NVIDIA InfiniBand design and acceptance guide, and NCCL Slow AllReduce test matrix. Browse the AI Infrastructure hub and Data Center hub for the surrounding network design.
Post a Comment