GPUs · 2026-10-07 · Checked 2026-10-10

The architecture ladder: why a card that fits can still be refused

A model can fit in VRAM and still never load. The second check is the one nobody documents: the card’s hardware generation, and whether the engine you picked was compiled for it. Every vendor has a ladder of generations; each step adds data types and matrix instructions, and an engine’s minimum is a hard step on that ladder. This page is the ladder, per vendor, with the capability each step added and what software actually gates on.

The short version: an engine does not run “on a GPU”, it runs kernels that were compiled for a specific architecture. If your card is below the floor the engine was built for, there is no kernel to launch. That is a different failure from the one Will it fit predicts, and the two are usually confused.

NVIDIA: compute capability, the only single number

NVIDIA stamps each die with a compute capability (cc), written sm_XX. It is one ordered line: a higher number is a newer generation with every earlier feature. Your own card reports it directly:

nvidia-smi --query-gpu=compute_cap --format=csv     # 8.6
ccGenerationCardsWhat this step added that engines gate on
7.0VoltaTesla V100, TITAN VFirst tensor cores: FP16 inputs with FP32 accumulate
7.5TuringT4, RTX 20-series, GTX 16-seriesINT8 and 4-bit/1-bit tensor ops (2nd-gen tensor cores)
8.0Ampere (datacenter)A100, A30BF16, TF32, FP64 tensor, 2:4 structured sparsity, async copy
8.6Ampere (consumer)RTX 30-series, A40, A10The same 3rd-gen feature set on consumer silicon
8.9Ada LovelaceRTX 40-series, L4, L40SFP8 (E4M3/E5M2) in tensor cores
9.0HopperH100, H200TMA, thread-block clusters, distributed shared memory, warpgroup MMA
10.0 / 10.3Blackwell (datacenter)B200, GB300FP4/FP6 and the microscaling formats (MXFP8/MXFP4)
12.0 / 12.1Blackwell (consumer, GB10)RTX 50-series, RTX PRO Blackwell, DGX SparkFP4 on consumer and workstation parts

Two steps carry most of the practical weight. BF16 starts at 8.0 — everything below it emulates the type, which is why a Pascal or Turing card has usable-but-slower inference rather than none. FP8 starts at 8.9, and that is where engines that were written around FP8 draw their line.

The toolkit has its own ladder, one release behind: CUDA 12.0 removed Kepler (“Kepler architecture support is removed from CUDA 12.0”, CUDA 12.0 release notes), and CUDA 13.0 removed offline compilation and library support for Maxwell, Pascal and Volta — “CUDA 13.0 supports all NVIDIA architectures from Turing through Grace Blackwell” (CUDA 13.0 release notes). Turing (7.5) is therefore the floor for building anything from source today, whatever the engine claims. NVIDIA’s own cc-per-card tables are at CUDA GPUs and, for retired parts, legacy CUDA GPUs.

AMD: two ladders that never meet

AMD identifies a GPU by a gfx target (gfx1030, gfx1100, gfx1201, gfx942). It reads like a version because it is one: the target is the GFXIP version written as major.minor.stepping, straight from LLVM’s processor definitions — GFX1201 … [12, 0, 1], GFX90A … [9, 0, 0xa], GFX942 … [9, 4, 2] (LLVM GCNProcessors.td). Your card prints it:

rocminfo | grep gfx          # gfx1201

The trap is that AMD has two ladders running in parallel, and they are not comparable:

  • CDNA (gfx9xx) for Instinct — the datacenter line.
  • RDNA (gfx10xx to gfx12xx) for Radeon — the consumer line.

A gfx942 (CDNA 3, MI300X) and a gfx1201 (RDNA 4, RX 9070) are not older and newer versions of each other. They are different lines with different instructions.

Stepgfx targetsCardsMatrix unitsFP8FP4
GCN 3-5 / Vegagfx803, gfx900, gfx906RX 470-580, Vega 56/64, MI25, MI50nonenono
RDNA 1 / 2gfx1010, gfx1030RX 5700 XT, RX 6800/6900 XTnonenono
RDNA 3gfx1100, gfx1101, gfx1102, gfx115xRX 7900 XTX/XT, 7800 XT, 7600, PRO W7900WMMAnono
RDNA 4gfx1200, gfx1201RX 9060/9070, Radeon AI PRO R9700WMMAOCP (E4M3/E5M2)no
CDNA 1 / 2gfx908, gfx90aMI100, MI210/MI250MFMAnono
CDNA 3gfx942MI300A/MI300X/MI325XMFMAFNUZ onlyno
CDNA 4gfx950MI350X, MI355XMFMAOCPyes (E2M1)

Two details worth knowing before you compare an AMD card with an NVIDIA one.

There are two FP8 formats, and generations disagree on which one. CDNA 3 supports AMD’s own FNUZ encoding; CDNA 4 and RDNA 4 support the OCP encoding the rest of the industry uses. CDNA 3 can do FP8 and still be the wrong FP8 for a kernel written against Hopper, which is why ROCm’s own precision table lists them as separate rows with no overlap. RDNA 2 and RDNA 3 have no FP8 at all, and FP4 exists on CDNA 4 only.

“Supported” means AMD ships kernels for that target in that release. ROCm builds its libraries per gfx target; a target that is not in the release’s table has no prebuilt kernels. The current system requirements mark MI25 (gfx900) and MI50 (gfx906) unsupported, and ROCm 7.0 states that “the AMD GPU targets gfx803 and gfx900 are no longer built by default” (ROCm 7.0 release notes). The workaround the projects themselves recommend is to lie about the card’s identity: HSA_OVERRIDE_GFX_VERSION=11.0.0 makes an RDNA 3 part answer as a supported sibling (llama.cpp build docs).

Intel: families, not numbers

Intel has no compute-capability integer. Toolchains key on a codename (dg2, bmg, pvc, lnl, ptl, cri) plus a four-part GMDID version that sycl-ls prints — an Arc A770 comes back as 12.55.8, a Battlemage part as 20.1.4 (llama.cpp SYCL backend).

StepCodenameChipsMatrix hardware
Gen9 / Gen11gen9, gen11Skylake through Ice Lake iGPUsnone
Gen12 / Xe-LPgen12lp, tgl, adlTiger Lake, Alder Lake iGPUs, DG1none (int8 dot-product only)
Xe-HPGdg2, acmArc A-series (A380, A750, A770)XMX: int8, fp16, bf16, restricted shapes
Xe-HPCpvcData Center GPU MaxXMX full: adds tf32
Xe2bmg, lnlArc B-series (B580), Core Ultra 200VXMX: int8, fp16, bf16, tf32
Xe3 / Xe3pptl, criPanther Lake, Crescent IslandFP8 and FP4 DPAS appear here only

Intel’s own compiler still lists Gen9, Gen11, Gen12, Xe and Xe2 as supported platforms (IGC), but the runtime packages split: “regular packages support Gen12 and later devices. Support for Gen8, Gen9 and Gen11 devices will be delivered via packages with legacy1 suffix” (LEGACY_PLATFORMS.md). In practice the oneAPI road starts at 11th-generation Core and up; older Intel GPUs fall back to OpenCL, “although the performance is not optimal, and some GPUs may not support OpenCL nor have any GPGPU capabilities”.

Software gates on capability queries here, not on a comparison: SYCL asks the device which joint_matrix combinations are legal and gets errc::kernel_not_supported otherwise, and the fp8/fp4 DPAS instructions are emitted only for -device cri.

Apple: GPU families, one rung per chip generation

Apple’s ladder is the Metal GPU family, checked with MTLDevice.supportsFamily(_:). The chip-to-family mapping is in Apple’s Metal Feature Set Tables (May 2026 revision), and the rungs that matter for inference are:

FamilyChipsWhat it gates
Apple6A13The bfloat data type exists
Apple7A14, M1SIMD-scoped matrix multiply — the matrix path inference kernels need
Apple8A15, A16, M2—
Apple9A17 Pro, A18, M3, M4—
Apple10A19, M5A neural accelerator in every GPU core, driven by Metal 4 tensor operations

M1 is the entry ticket: simdgroup_matrix appears at Apple7, so an A13 or an Intel Mac GPU has no SIMD-group matmul path. llama.cpp reads exactly these rungs — has_simdgroup_mm = supportsFamily:MTLGPUFamilyApple7, bf16 from Apple6 or Metal 3, and the tensor path from the Metal 4 family, disabled again unless the chip name contains M5, M6 or A19 (ggml-metal-device.m). Note the disagreement to expect: the tables put the bf16 data type at Apple6 while some frameworks require M2 for bf16 maths, so read the engine’s own check, not the marketing.

Apple’s newest rung is a real change rather than a wider feature set: on Apple10 hardware each GPU core carries a neural accelerator, and MLX reaches it through Metal 4 tensor operations (Apple ML research). MLX does not call supportsFamily at all — it reads the GPU architecture generation and enables those kernels at generation 17 and up (MLX device.cpp).

Intel and AMD Macs are not on this ladder: they report the Common and Mac families, so no Apple-family check can pass. They still run the Metal backend, just without the SIMD-group matrix path.

Vulkan: no ladder on purpose

The cross-vendor backend has no generation number at all. It asks the device which optional features it exposes, and the engine enables each one separately: VK_KHR_cooperative_matrix (coopmat1), VK_NV_cooperative_matrix2 (coopmat2), VK_KHR_shader_float16_int8, VK_KHR_shader_bfloat16, VK_KHR_shader_integer_dot_product (ggml-vulkan.cpp). llama.cpp logs the outcome on startup — matrix cores: KHR_coopmat, NV_coopmat2, or none.

There is a second, easily missed gate in the same place: the shaders must have been generated at build time, so the build probes glslc for each extension and silently omits the shaders it cannot compile. A capable GPU paired with an old build reports no matrix cores. The env vars GGML_VK_DISABLE_COOPMAT, GGML_VK_DISABLE_COOPMAT2, GGML_VK_DISABLE_BFLOAT16, GGML_VK_DISABLE_INTEGER_DOT_PRODUCT and GGML_VK_DISABLE_F16 switch each one off by hand — the fastest way to tell whether a crash comes from a path your card claims but cannot run.

Who refuses, and who degrades

The ladder only matters because engines sit at different points on it. Same card, three different outcomes:

EngineWhere the gate isOutcome below the floor
llama.cpp (CUDA)build-time arch listBuilds 50/61/70-virtual when the toolkit is older than 13, plus 75/80-virtual, 86-real, 89-real, 90-virtual. Cards covered by a -virtual entry run through PTX the driver compiles on first launch; a card in neither list fails at run time with the CUDA driver’s “no kernel image is available” message. No refusal, no warning.
vLLMbuild, then runtimeThe build intersects your cards with a supported list per toolkit version and aborts with “No supported CUDA architectures; the build would produce a binary with no usable kernels”. At run time, bf16 raises: “Bfloat16 is only supported on GPUs with compute capability of at least 8.0” (CMakeLists.txt, cuda.py).
TensorFoldbuild, then startupThe build carries one SASS image per target and the default is sm_121: const sms = b.option([]const u8, "sm", "SASS targets, comma separated (121; later 120,89)") orelse "121" (cuda.zig). Its README names GB10 — the DGX Spark — as the qualified CUDA platform — and pins Metal to apple_m1 (README, packaging). Until 1.0.0 the Python engine set MIN_CAPABILITY = (8, 9) and refused a slower card by name; the Zig engine replaced it, and an Ada or Ampere card now has no image to load at all.

That is the whole practical distinction: an engine that compiles per model family refuses, an engine that compiles a broad arch list degrades. “Will it fit” tells you the first failure; the ladder is the second one.

Read your own card’s step

VendorCommandWhat it prints
NVIDIAnvidia-smi --query-gpu=compute_cap --format=csv8.6
AMDrocminfo | grep gfxgfx1201
Intelsycl-lsIntel(R) Arc(TM) A770 Graphics 12.55.8
Applemetal-archthe architecture string behind MTLArchitecture.name

On Linux, clinfo covers OpenCL-only Intel parts, and on Windows the same NVIDIA query works from a nvidia-smi prompt.

What this means in practice

  • Fitting and running are separate checks. Run both before you blame the model or the quant.
  • Read the engine’s own floor, not the vendor’s generation name: the number that matters is the one in its build script or README.
  • Do not compare across lines. gfx942 is not older than gfx1201, Apple8 is not “NVIDIA 8”, and Intel’s 12.55.8 is not a compute capability.
  • The toolkit pins you too. If your card is pre-Turing, a CUDA 13 container cannot build for it at all, so an old card means old images.
  • A card one step below a floor is not slow, it is absent. There is no setting to make a missing kernel image exist.
  • When you buy, the step decides which engines you can run later: Ada (8.9) is currently the entry ticket for FP8 kernels, M1/Apple7 for the Mac matrix path, RDNA 3 for AMD’s WMMA. A cheaper card one rung lower stays on the compatibility list for the degrade-instead-of-refuse engines — GPUs has the numbers, Runtimes says what each engine loads and on what backend.

Related: Out of memory, Which quant should I pick, Will it fit, Will it run.

Questions

My model fits in VRAM, so why will the engine not start?
Because fitting is a memory question and running is a compiled-code question. An engine ships GPU kernels built for particular hardware generations; a card below the engine’s floor has no kernel image to run. The check happens at startup, before any weight is loaded.
Is a higher number always a newer, better card?
Only on NVIDIA, where compute capability is one ordered line. AMD runs two lines that never meet (CDNA for Instinct, RDNA for Radeon), Intel names families instead of numbering them, and Apple’s tiers are per-family feature sets. Do not compare a number across two vendors, or across AMD’s two lines.
Does this matter if I only run llama.cpp?
Less. llama.cpp compiles for a range of architectures, including PTX the driver compiles on first run, so an older card usually runs on a slower path instead of being refused. Engines built per model family (TensorFold, vLLM) refuse instead of degrading.
How do I find my own card's step?
One command per vendor: nvidia-smi --query-gpu=compute_cap, rocminfo | grep gfx, sycl-ls for Intel, metal-arch on a Mac.