Source-backed deployment guide; calculations are labeled. This refresh uses Meta’s Llama architecture paper and Llama 3.3 model card, current inference-runtime documentation, HostFleet’s September 10 GPU dataset audit, and selected public GPU prices rechecked on September 16, 2026. HostFleet did not benchmark tokens per second, latency, model quality, cloud capacity, or cold starts. Read the HostFleet methodology and affiliate policy for how sourced and measured claims are separated.

Selected GPU prices verified: September 16, 2026
Full GPU dataset verified: September 10, 2026
Model and runtime sources checked: August 28, 2026

What GPU do you need to run Llama 70B? VRAM, context, and KV-cache guide

For Llama 3.3 70B, use a 48 GB GPU only as the entry point for a 4-bit, short-context, single-user test. Use 80–96 GB when the 4-bit deployment needs longer contexts or concurrency, or when you want to attempt 8-bit on one GPU. For BF16/FP16, 180 GB is the first credible single-GPU tier for modest contexts, while a full 128K context or several active sequences can justify a 288 GB tier or multiple GPUs.

The common shortcut—70 billion parameters times four bits equals about 35 GB—counts only raw weights. It does not count quantization metadata, CUDA graphs, runtime workspace, temporary tensors, or the key-value cache that grows with context and concurrent sequences.

This guide makes the missing cache term explicit. It is still a planning model, not a promise that a particular quantized artifact or inference engine will fit.

The September refresh also makes the 48 GB cost boundary less abstract. Massed Compute’s public one-GPU L40S VM moved from $0.88 to $0.97 per hour on September 16, a 10.2% increase. That does not change the memory recommendation, but it raises the 720-hour planning estimate from $633.60 to $698.40 before any other cost. Capacity and lifecycle still matter more than a nine-cent rate change.

The buying answer

Llama 70B deployment goalCapacity to test firstWhy
4-bit, one short sequence48 GBAbout 35 GB of raw weights leaves limited room for quantization overhead, runtime allocations, and cache
4-bit, 32K context or several active short sequences80–96 GBA BF16 KV cache is about 10 GiB per full 32K sequence before runtime overhead
8-bit, one short sequence80 GB lab floor; 96 GB saferRaw 8-bit weights are about 70 GB, so an 80 GB card has little operating headroom
BF16/FP16, short-to-moderate context180 GBRaw weights are about 140 GB; a 141 GB H200 is not a practical fit
BF16/FP16 near 128K, or more concurrency288 GB or multiple GPUsOne full BF16 128K KV cache is about 40 GiB before runtime allocations

These are capacity starting points, not speed recommendations. A newer GPU can be faster than an older card with the same VRAM, and an engine may not support every quantization or accelerator equally well. Confirm software compatibility and measure the exact deployment before buying reserved capacity.

Turn the VRAM answer into a deployment decision

This page answers the capacity question first. The next useful question is whether the billing model fits the way the deployment will actually run:

  • For a 48 GB first test, use the serverless GPU pricing matrix to separate request-priced products from VM-like capacity before comparing rates.
  • For an 80–96 GB endpoint, use the H100 rental price comparison to compare current 80 GB options and their provider-specific lifecycle boundaries.
  • Once the target product is clear, use the GPU cloud cost calculator to model allocated time, retained resources, and missed-cleanup exposure instead of multiplying a headline rate by 720 hours.

These links do not change the memory recommendation. They keep the next decision in the right order: fit the model, select a launchable product, then price its actual billable state.

Llama 70B memory has three main buckets

The planning equation is:

required GPU memory ≈ model weights + KV cache + runtime headroom

1. Model weights

A deliberately simple lower-bound estimate is:

raw weight memory in decimal GB ≈ parameters in billions × bits per weight ÷ 8

For the public 70B parameter label in Meta’s Llama 3.3 model card:

Weight formatRaw-weight estimateWhat the number omits
4-bitabout 35 GBQuantization scales, metadata, runtime workspace, and KV cache
8-bitabout 70 GBRuntime workspace and KV cache; some methods retain selected values at higher precision
BF16 / FP16about 140 GBRuntime workspace and KV cache

The formula uses decimal GB and the rounded 70B label. It is not a file-size claim for a specific GGUF, AWQ, GPTQ, bitsandbytes, or other artifact. Hugging Face’s bitsandbytes documentation says 8-bit quantization halves model memory usage, but the actual footprint must still be read from the loaded model.

2. KV cache

During autoregressive inference, the server retains key and value tensors for tokens already processed. That cache is why context length and concurrent sequences are memory decisions.

Meta’s Llama 3 architecture paper lists the 70B design with 80 layers, model dimension 8,192, 64 attention heads, and 8 key/value heads. The attention-head dimension is 8,192 ÷ 64 = 128. For a BF16 or FP16 cache at two bytes per value, the transparent estimate is:

KV bytes per token per sequence =
  2 for key and value
  × 80 layers
  × 8 KV heads
  × 128 values per head
  × 2 bytes
  = 327,680 bytes

That is 0.3125 MiB per token per active sequence. It produces the following cache-only estimates:

Filled context per active sequenceBF16/FP16 KV cache, one sequenceFour full sequences
8,192 tokens2.5 GiB10 GiB
32,768 tokens10 GiB40 GiB
65,536 tokens20 GiB80 GiB
131,072 tokens40 GiB160 GiB

Estimate assumptions: the Llama 3 70B architecture in Meta’s paper; every sequence filled to the stated length; keys and values stored at two bytes each; no prefix sharing; and no cache quantization. Real engines allocate cache in blocks, may reserve less than the theoretical maximum, and can reuse prefixes or evict blocks.

vLLM documents an FP8 KV-cache option that can reduce the cache footprint and allow more tokens in memory. That is an optimization to evaluate, not permission to divide every number blindly: vLLM recommends calibrated scales for maximum accuracy, and backend support matters. The conservative table above keeps the cache at BF16/FP16.

3. Runtime headroom

Weights plus theoretical cache still do not equal a deployable configuration. The engine needs memory for kernels, CUDA graphs, temporary tensors, communication buffers, and its own allocator. The exact requirement changes with engine version, maximum sequence count, batch policy, tensor parallelism, and quantization backend.

This is why an H200 with 141 GB of VRAM is not an honest BF16 Llama 70B recommendation. The raw-weight estimate already consumes roughly 140 decimal GB. There is effectively no room for the serving process or a useful cache.

Why context and concurrency change the recommendation

4-bit on 48 GB: keep the test narrow

A 48 GB A40, RTX A6000, L40, or L40S is still a sensible place to prove that a 4-bit artifact loads and answers short requests. But it is a poor basis for promising a 32K context or multi-user service.

The raw 4-bit estimate is about 35 GB. Add a theoretical 10 GiB BF16 cache for one filled 32K sequence and the deployment is already close to the card’s advertised capacity before quantization overhead or runtime workspace. The units are not identical—weight arithmetic above uses decimal GB while GPU and cache reporting commonly use binary GiB—but that mismatch does not rescue a configuration with almost no margin.

Treat 48 GB as a compatibility experiment. If the intended product needs long documents, several active chats, or a generous maximum sequence count, start testing at 80 or 96 GB instead.

8-bit on 80 GB: a floor, not a comfortable target

An 80 GB A100 or H100 can attempt 8-bit Llama 70B because the raw weight estimate is about 70 GB. It does not follow that a long-context endpoint will fit.

At 32K, the theoretical BF16 cache is another 10 GiB for one full sequence. That consumes essentially all remaining nominal capacity before the engine’s own allocations. A 96 GB RTX PRO 6000 creates more memory margin, although its actual throughput and software support must be tested rather than inferred from VRAM.

If 8-bit quality is a requirement rather than a habit, compare an 80 GB lab run with a 96 GB production candidate. If the engine still cannot hold the target sequence count, reduce context or concurrency, evaluate KV-cache quantization, or distribute the model.

BF16 on 180 GB: credible, but not unlimited

A 180 GB B200 leaves roughly 40 GB above the simple 140 GB raw-weight estimate. That makes it the first credible current single-GPU capacity tier for BF16 Llama 70B, but it does not guarantee the model plus a full 128K BF16 cache will fit. That cache alone is about 40 GiB, leaving no honest allowance for the runtime.

For a short or moderate context, 180 GB is a defensible first test. For 128K, multiple simultaneous sequences, or generous allocator headroom, a 288 GB B300 or a supported multi-GPU topology is the safer planning tier.

Multi-GPU changes the operational problem. Tensor parallelism adds topology, communication, placement, and failure-recovery constraints. Capacity may fit across two cards while latency or cost becomes unacceptable. Benchmark the exact topology rather than treating total VRAM as one pooled card.

Current cloud price ladder for the capacity tiers

The table below uses selected public one-GPU options from HostFleet’s live GPU pricing table. HostFleet’s complete 21-provider ledger was fully checked on September 10, 2026; Massed Compute’s L40S cell was then source-verified and updated on September 16. Every selected row below was also rechecked against its official pricing page on September 16. These rows illustrate a capacity ladder; they do not prove stock, quota, regional access, or performance.

Capacity tierSelected published optionPublic rate checked Sept. 16720-hour estimateImportant boundary
48 GBMassed Compute L40S VM$0.97/hr$698.40One GPU, 12 vCPU, and 72 GB RAM; the public table lists storage as 625 without a unit
80 GBHyperstack A100 80 GB$1.35/hr$972.00One-GPU VM includes fixed CPU, RAM, root disk, and ephemeral disk; public IP and shared storage are separate
96 GBNebius RTX PRO 6000$1.80/hr$1,296.00One-GPU prescribed configuration includes 24 vCPU and 218 GB RAM
141 GBKoyeb H200$3.00/hr$2,160.00Serverless instance; scale-to-zero is public preview and regional availability is not guaranteed
180 GBKoyeb B200$5.50/hr$3,960.00Serverless instance; the public catalog currently marks the one-GPU product restricted and exposes no regions
288 GBNebius B300$7.85/hr$5,652.00One-GPU prescribed configuration includes 24 vCPU and 346 GB RAM; public price is not evidence of capacity

Estimate assumptions: one GPU remains allocated for 720 hours in a 30-day month; public USD list rates; no discounts, taxes, additional storage, network, or separately billed resources beyond what each exact row includes. The monthly column is arithmetic, not a vendor quote.

The selected products are not interchangeable. Massed Compute, Hyperstack, and Nebius expose VM-like configurations. Koyeb’s rows are per-second serverless GPU instances. The L40S is also not necessarily the cheapest 48 GB way to test a quantized model; it is the selected current example because its rate changed and its billing boundary is documented. Compare all current rows in the live table, then compare lifecycle behavior as well as the accelerator. The serverless GPU pricing matrix separates those product models, while the GPU cost calculator lets you change allocated hours.

What the Massed Compute price change means

Massed Compute’s official page lists the one-GPU L40S VM at $0.97/hour, with 12 vCPU and 72 GB RAM, as checked September 16. HostFleet’s prior tracked rate was $0.88/hour. The arithmetic change is:

rate increase = ($0.97 - $0.88) / $0.88 = 10.2%
old 720-hour estimate = $0.88 × 720 = $633.60
new 720-hour estimate = $0.97 × 720 = $698.40
difference = $64.80

Those are utilization scenarios, not monthly plans or invoice predictions. They assume one VM remains active for every modeled hour and exclude tax or any unlisted add-on. Massed Compute says it has no bandwidth charges, but the public table’s storage column shows 625 without a unit, so this guide does not silently label that number GB.

The operational trap is the release action. Massed Compute’s billing overview, checked September 16, says it sums the hourly cost of active VMs, divides by 60, and debits the account every minute; micro-cent remainders carry into the next charge. Its instance documentation describes Running, Initializing, and a billing-failure Stopped state, while the trash action terminates the VM and deletes all associated data.

The checked public docs do not establish whether Initializing is included in the active-VM sum, the exact cutoff inside asynchronous termination, or partial-minute treatment. No instance was launched for this article. For planning, treat the VM as billable until termination is confirmed, preserve required data first, and do not interpret the documented billing-failure Stopped state as a customer-controlled pause button.

For the guide’s eight-hours-of-work-plus-160-hours-unattended scenario:

useful compute = $0.97 × 8 = $7.76
unattended compute = $0.97 × 160 = $155.20
one-week active total = $0.97 × 168 = $162.96

That is why a cleanup test matters more than the nine-cent hourly increase.

Koyeb’s scale-to-zero documentation, checked August 28, explicitly includes GPU instances but labels the feature public preview. The default idle period is five minutes. A supported new request can wake a sleeping service, but HTTP/2 requests cannot do so, and no public GPU wake-time SLA is documented. At the current H200 rate, a nominal five-minute idle tail is $0.25; for B200 it is about $0.46. Those are rate × 5/60 estimates and exclude active work and wake/model-load time.

If the chosen GPU will stay warm continuously, compare the 720-hour result with the A100 rental price guide or H100 rental price guide, as applicable. If it can genuinely return to zero, compare total allocated seconds—including model loading and idle tails—not just inference time.

The same weight arithmetic for other open-model sizes

The method generalizes, but architecture-specific KV cache does not. The raw-weight columns below use public parameter counts from the linked model cards and the same bits-per-weight formula. The suggested 4-bit tier is only a first capacity test for a short context and one active sequence.

Example modelPublished parameters4-bit raw weights8-bit raw weightsBF16/FP16 raw weightsFirst 4-bit capacity test
Qwen3 8B8.2Babout 4.1 GBabout 8.2 GBabout 16.4 GB16 GB
Mistral Small 3.124Babout 12 GBabout 24 GBabout 48 GB24 GB
Qwen3 32B32.8Babout 16.4 GBabout 32.8 GBabout 65.6 GB24–32 GB
Llama 3.3 70B70Babout 35 GBabout 70 GBabout 140 GB48 GB

Do not reuse Llama’s 0.3125 MiB-per-token cache estimate for those models. KV-cache size depends on layer count, key/value head count, head dimension, cache dtype, and sequence count. Repeat the architecture math or use the inference engine’s cache report.

A deployment test that produces a real answer

The capacity table should narrow the first rental, not replace a test. Use this sequence before committing to a warm month:

  1. Choose the exact artifact. Record model revision, quantization method, file size, and inference-engine version.
  2. Set the real limits. Configure the intended maximum model length, maximum active sequences, batch policy, and cache dtype.
  3. Load at the smallest plausible tier. Capture weight memory, cache capacity reported by the engine, and free memory after initialization.
  4. Fill the context. Test short, typical, and maximum prompt-plus-output lengths rather than one tiny prompt.
  5. Add concurrency gradually. Run one sequence, then the expected active count, while recording peak GPU memory, time to first token, completion latency, and out-of-memory events.
  6. Repeat after restarts. A deployment that fits once but fails during cold load, graph capture, or a version change is not production capacity.
  7. Price allocated time. Include startup, model loading, idle retention, failed runs, and storage—not only successful generation seconds.

For RunPod specifically, read RunPod pricing: Pods vs Serverless before choosing between a Pod and a scale-to-zero worker. The cheaper-looking product can become the more expensive deployment if shutdown behavior does not match the traffic pattern.

Verdict

The useful answer to “what GPU runs Llama 70B?” is a range tied to the serving configuration:

  • 48 GB is the 4-bit, short-context experiment tier.
  • 80–96 GB is the practical zone for longer-context 4-bit work or single-GPU 8-bit testing.
  • 180 GB is the first credible single-GPU BF16 tier for moderate contexts.
  • 288 GB or multiple GPUs is the safer plan for BF16 near 128K or meaningful concurrency.

The key correction is that weights are only the first line. For the Llama 3 70B architecture, one BF16 KV cache consumes about 2.5 GiB at 8K, 10 GiB at 32K, and 40 GiB at 128K per filled sequence. Concurrency multiplies that cache requirement.

Choose the smallest tier that preserves headroom for the intended context and sequence count, then prove it with the exact artifact and runtime. That is less satisfying than a universal GPU name, but it is the difference between a model that loads and a service that stays up.

Sources

Model and runtime sources were accessed August 28, 2026 and confirmed reachable September 16. Selected pricing sources and Massed Compute billing documentation were checked September 16, 2026. Koyeb’s scale-to-zero documentation was checked August 28.

Signing up for a GPU host? Using our affiliate link supports HostFleet’s testing budget at no extra cost to you: RunPod (+$5 credit on your first $10). Links are labeled, and source citations in this article are never affiliate links.