A model that fits on paper can still fail when real users arrive. The missing line is often not the model weights: it is the key-value cache, runtime workspace, or a parallel layout that cannot distribute memory as evenly as the spreadsheet assumes.
This GPU memory sizing guide answers a practical procurement question: how many H100 or H200 GPUs should you budget for a 70-billion-parameter inference service? It includes a reusable calculation, a concurrency table, two original diagrams, and an acceptance checklist. All designs are hypothetical; the arithmetic was executed in Python, but no GPU benchmarks or deployment tests were run for this article.
1. Define the service before choosing the GPU
Write a one-page workload contract before requesting server quotes. Record the exact model revision, weight representation, attention architecture, maximum input and output lengths, active concurrent sequences, latency objectives, availability requirement, and serving software version. Separate inference, full-parameter training, and adapter tuning: they need different memory budgets.
Use resident tokens, not the advertised context window alone. A service allowing long prompts but generating short responses has a different load profile from one generating long answers. For conservative admission planning, account for prompt tokens plus the allowed generated tokens. Distinguish active sequences from connections waiting in a queue; only the former belong in this simple cache calculation.
Hugging Face documents that dynamic caches grow during generation, while sliding-window or chunked-attention layers can stop growing at their configured limits.[2] Consequently, the full-attention calculation below is deliberately scoped: it is not a universal estimator for every hybrid, recurrent, multimodal, or mixture-of-experts model.
Worksheet inputs: model parameter count; weight bytes per parameter; attention layers; KV heads per layer; head dimension; cache bytes per element; resident tokens; active sequences; usable memory per GPU; and the engine's supported parallel layouts. Obtain architecture inputs from the actual model configuration, not a similarly named model on a comparison chart.
2. H100 versus H200: compare the exact form factor
For the H100 SXM, NVIDIA lists 80GB of GPU memory and 3.35TB/s of GPU memory bandwidth; its H100 NVL column instead lists 94GB and 3.9TB/s.[4] Do not attach SXM numbers to an unspecified PCIe purchase order.
For H200 SXM and H200 NVL, NVIDIA lists 141GB and 4.8TB/s; the form factors and server connectivity options differ even though those memory figures match.[5] The H200 page describes the product as available, but still includes a preliminary-specification footnote, so confirm the shipped server bill of materials and contractual specifications.[5] Neither that wording nor this article establishes local stock or delivery dates.
| Candidate used in this worksheet | Vendor-listed memory per GPU | Vendor-listed memory bandwidth | Procurement boundary |
|---|---|---|---|
| H100 SXM | 80GB | 3.35TB/s | SXM server configuration, not an arbitrary H100 card [4] |
| H200 SXM | 141GB | 4.8TB/s | Verify the complete HGX/server configuration [5] |
| H200 NVL | 141GB | 4.8TB/s | PCIe product; verify the actual bridged GPU grouping [5] |
These are vendor figures, not measured application throughput. For transparent first-pass arithmetic, the worksheet treats the advertised GB values as decimal billions of bytes. Before acceptance, replace them with the bytes your installed platform actually exposes and the memory available to the serving process.
The comparison intentionally stays within these Hopper products rather than declaring a “latest GPU.” If a supplier proposes a Blackwell or newer system, rerun the same worksheet with its exact delivered specifications, topology, software support, and quote. A newer name does not answer the service-sizing question.
3. Derive the weight and KV-cache budget
The weight-only calculation is straightforward bookkeeping:
weight_bytes = parameter_count × stored_bytes_per_parameter
Our hypothetical dense model has exactly 70,000,000,000 parameters stored at two bytes each. Its raw weight payload is 140.00GB, or approximately 130.39GiB. GB and GiB are different units; keep one unit convention throughout the spreadsheet. This is not a promise that an actual model marketed as “70B” has exactly that parameter count.
A nominal four-bit representation would reduce the raw payload to 35.00GB before quantization metadata, scales, unquantized tensors, alignment, and runtime costs. Do not present that lower bound as the complete installed footprint. Record the actual checkpoint representation and the serving engine's loaded memory instead.
A KV cache preserves attention keys and values so generation can reuse earlier work.[2] For our explicitly assumed full-attention layout, derive its raw storage by counting both tensors:
KV_bytes = 2 × layers × KV_heads × head_dimension
× bytes_per_cache_element × total_resident_tokens
total_resident_tokens = active_sequences × tokens_per_sequence
The leading two represents keys and values. This is our tensor-element calculation, not a vendor benchmark. With different sequence lengths, replace the multiplication by the sum of actual resident tokens. Use KV heads, not query heads, for grouped-query attention. The worksheet excludes prefix sharing, padding, block rounding, extra cache metadata, and speculative-decoding state; validate those separately in your engine.
For the hypothetical model, assume 80 attention layers, 8 KV heads, a head dimension of 128, and two-byte cache elements. The result is 327,680 bytes per resident token across the model. These architecture values are design assumptions, not a claim about a named commercial model.
Source: original Network freak diagram, generated for this hypothetical sizing example. Segment widths are illustrative, not measured allocations.
4. Worked concurrency table: why the weight-only answer fails
For the first pass, reserve 15% of nominal device capacity outside weights and raw KV storage. This is an explicit planning assumption, not a vendor recommendation or a guarantee against out-of-memory errors. It must accommodate runtime buffers and other allocations; real profiling may require more.
Calculate the pooled-capacity lower bound as:
minimum_devices = ceiling(
(weight_bytes + KV_bytes) / (device_bytes × 0.85)
)
| Active sequences | Resident tokens per sequence | Raw KV cache, GB | Weights plus KV, GB | H100 SXM capacity lower bound | H200 capacity lower bound |
|---|---|---|---|---|---|
| 1 | 8,192 | 2.68 | 142.68 | 3 | 2 |
| 8 | 8,192 | 21.47 | 161.47 | 3 | 2 |
| 16 | 8,192 | 42.95 | 182.95 | 3 | 2 |
| 8 | 32,768 | 85.90 | 225.90 | 4 | 2 |
| 16 | 32,768 | 171.80 | 311.80 | 5 | 3 |
Every number above comes from the stated assumptions and executed arithmetic. These are pooled-memory lower bounds, not recommended tensor-parallel sizes. They assume ideal distribution of both weights and cache and say nothing about throughput. A result of three or five devices is a warning to inspect supported layouts, not permission to configure that size blindly.
For this example's eight KV heads, evaluate supported layouts such as two, four, or eight GPUs rather than assuming a three-way split is efficient or available. A serving engine may replicate KV heads or other tensors instead of dividing them ideally. Layer placement, embeddings, temporary allocations, and uneven shards can also invalidate the pooled total. Inspect the busiest rank, not just average GPU memory.
At eight active sequences and 32,768 resident tokens each, a two-H200 candidate passes this preliminary capacity screen. A four-H100 candidate also passes. Neither has passed a latency or production-readiness test. At sixteen such sequences, both candidates must be reconsidered because their earlier capacity calculation no longer covers the workload.
5. Reusable offline calculator
This standard-library Python example reproduces the table's eight-sequence, long-context row. It calculates raw payload and an idealized capacity lower bound only. Change the assumptions to match your workload, then select a supported topology separately.
from math import ceil
def estimate(parameters, weight_bytes, layers, kv_heads, head_dim,
cache_bytes, sequences, tokens, gpu_gb, reserve):
positive = (parameters, weight_bytes, layers, kv_heads, head_dim,
cache_bytes, sequences, tokens, gpu_gb)
if any(x <= 0 for x in positive) or not 0 <= reserve < 1:
raise ValueError("Positive inputs and 0 <= reserve < 1 required")
weights = parameters * weight_bytes
kv = 2 * layers * kv_heads * head_dim * cache_bytes * sequences * tokens
usable = gpu_gb * 1_000_000_000 * (1 - reserve)
return weights / 1e9, kv / 1e9, ceil((weights + kv) / usable)
for gpu_gb in (80, 141):
weights, kv, minimum = estimate(
70_000_000_000, 2, 80, 8, 128, 2,
8, 32768, gpu_gb, 0.15
)
print(f"{gpu_gb} GB: weights={weights:.2f} GB, "
f"KV={kv:.2f} GB, capacity lower bound={minimum}")
Executed CPU-only calculation output:
80 GB: weights=140.00 GB, KV=85.90 GB, capacity lower bound=4
141 GB: weights=140.00 GB, KV=85.90 GB, capacity lower bound=2
The code does not interrogate a GPU, select kernels, or predict tokens per second. Keep a copy alongside your quotation and acceptance results so changes to context length or concurrency have a visible capacity consequence.
6. Fit is not speed: map memory onto the server
NVIDIA lists H200 NVL with two- or four-way NVLink bridge options, while H200 SXM is offered in HGX-based four- or eight-GPU server configurations.[5] Accordingly, “eight GPUs installed” must not be treated as proof that every pair has the same communication path.
For procurement, request the actual GPU-to-GPU and GPU-to-NIC topology, including bridge or switch domains, PCIe placement, and supported serving layouts. Keep tightly communicating shards within a validated scale-up group where practical. Treat crossing servers as a separate network-design decision, not a free extension of one memory pool.
HBM bandwidth, NVLink bandwidth, PCIe bandwidth, and NIC line rate describe different paths. This worksheet does not compare an aggregate bidirectional NVLink figure with a single-direction NIC rate or derive application performance from either. Ask suppliers to state directionality and scope for every quoted bandwidth.
Source: original Network freak logical diagram. GPU groups and separate hosts illustrate failure-domain planning; this is not an NVIDIA reference architecture.
For network design after memory sizing, use the earlier GPU cluster network calculator. If a model fits but distributed inference is slow, the NCCL troubleshooting matrix provides a separate investigation path. Neither replaces testing the actual serving engine.
7. Alternatives when the candidate does not fit
Reduce admitted concurrency or resident context first. These are explicit service changes, so get application-owner approval rather than silently truncating requests. Use the worksheet to expose the trade-off and publish queueing and rejection behavior in the service contract.
Evaluate weight quantization and cache quantization separately. Smaller weights do not automatically imply a smaller KV cache. Hugging Face documents quantized caches as a memory-saving option and warns they can harm latency when context is short and sufficient GPU memory is already available.[2] Require model-quality checks, supported-kernel confirmation, and end-to-end latency measurements for the exact combination.
Consider CPU cache offload deliberately. Hugging Face describes moving most layer caches to CPU memory, prefetching the next layer and returning the current layer's cache after attention computation; throughput can degrade depending on generation choices.[2] Budget host memory and test the actual transfer path. Do not buy smaller GPUs on the assumption that host RAM behaves like additional HBM.
Compare more shards with higher-memory devices. Ask for complete configurations and price them in EUR or USD using current supplier quotes. Include server, support, networking, operational constraints, and spare capacity. This article supplies no invented hardware prices and no claim that one option has lower total cost without workload measurements.
Distinguish a shard from a replica. In the proposed design, shards cooperate to serve one model instance; a replica is an independently serviceable instance. Capacity distributed across a shard group is not the same as surviving a failed member. Document which requests are lost and where new requests go after failure.
8. Training needs a different worksheet
Do not reuse the inference table for full training. Hugging Face's versioned training-anatomy guide describes a conventional mixed-precision AdamW example requiring 18 bytes per parameter plus activation memory, including weights, optimizer states, and gradients.[3] This is an implementation-specific baseline, not a universal constant for every optimizer or precision policy.
Applied to our assumed parameter count, that baseline is 1.26TB before activations and additional runtime costs. This calculated illustration explains why fitting inference weights does not establish training feasibility. Sharding strategy, offload, activation checkpointing, sequence length, and optimizer configuration need their own recorded assumptions and profiling.
Adapter tuning also deserves a separate estimate: identify frozen versus trainable tensors and what the framework retains during backward execution. Do not substitute the inference cache budget for training activations. For checkpoint planning, continue with the AI checkpoint storage sizing guide, which covers another part of the infrastructure budget.
9. Deployment and acceptance checklist
The following is a proposed test plan, not a report of executed GPU tests. Set numeric latency, error-rate, and recovery thresholds with the application owner before testing. Preserve the model revision, image digest, configuration, workload generator, and results together.
| Test | Evidence to capture | Acceptance question |
|---|---|---|
| Inventory and topology | Exact GPU SKU, exposed bytes, driver, runtime, GPU/NIC map | Does the delivered system match the quote and intended shard group? |
| Cold start | Load time and peak per-rank memory | Does loading succeed without relying on undocumented spare capacity? |
| Long-context prefill | Peak memory, time to first token, errors | Does the largest supported input meet the agreed objective? |
| Sustained decode | Inter-token latency, output rate, memory growth | Is the service stable at the agreed resident-token budget? |
| Mixed-length traffic | Tail latency, queue depth, admission decisions | Do short requests remain usable during long requests? |
| Quantization comparison | Task-quality checks and latency under identical inputs | Is the memory saving acceptable to the application owner? |
| Replica failure | Failed requests, rerouting, remaining capacity | Does the surviving deployment meet the declared degraded-service objective? |
| Soak and restart | Memory trend, error log, restart recovery | Is behavior repeatable rather than a single successful request? |
For resilience, put the proposed independent replicas on separate host failure domains and test the actual scheduler or load balancer. Size the surviving capacity against the promised degraded workload. Two replicas that are both required for normal peak demand do not automatically provide full peak service after one fails.
10. Troubleshooting: the estimate says yes, production says no
Failure while loading: compare the real checkpoint tensors and loading peak with the weight-only assumption. Look for temporary copies, unexpected precision, and extra processes. Record the rank that fails.
Failure during long prompts: isolate prefill peak from steady-state decode. Revisit workspace reserve and resident-token limits; a raw KV calculation does not include every temporary allocation.
Failure only under concurrency: check active sequences, generated-token growth, cache allocation policy, and whether the engine reserves more cache than the worksheet assumed. Reduce one variable at a time and retain the failing case.
One GPU fills while others have space: inspect shard placement and replicated state before buying more aggregate memory. An uneven layout needs a per-rank budget.
Memory fits but latency is unacceptable: investigate the serving engine, batching policy, compute phase, interconnect path, and offload behavior. A memory-capacity pass is not a throughput benchmark.
Practical takeaways
Start with the service contract, count weights and resident KV tensors separately, and treat the result as a lower-bound screen. Then validate supported shard layouts, per-rank peaks, latency, and failure behavior before committing to a server order. The useful procurement deliverable is not “this model needs two GPUs”; it is a reproducible worksheet plus an acceptance test for a specific model, workload, and topology.
Continue through the AI Infrastructure hub and Data Center networking hub for the remaining network and deployment decisions.
Post a Comment