GPUs · 2026-10-07 · Checked 2026-10-10
The architecture ladder: why a card that fits can still be refused
A model can fit in VRAM and still never load. The second check is the one nobody documents: the card’s hardware generation, and whether the engine you picked was compiled for it. Every vendor has a ladder of generations; each step adds data types and matrix instructions, and an engine’s minimum is a hard step on that ladder. This page is the ladder, per vendor, with the capability each step added and what software actually gates on.
The short version: an engine does not run “on a GPU”, it runs kernels that were compiled for a specific architecture. If your card is below the floor the engine was built for, there is no kernel to launch. That is a different failure from the one Will it fit predicts, and the two are usually confused.
NVIDIA: compute capability, the only single number
NVIDIA stamps each die with a compute capability (cc), written sm_XX. It is one ordered line: a higher number is a newer generation with every earlier feature. Your own card reports it directly:
nvidia-smi --query-gpu=compute_cap --format=csv # 8.6
| cc | Generation | Cards | What this step added that engines gate on |
|---|---|---|---|
| 7.0 | Volta | Tesla V100, TITAN V | First tensor cores: FP16 inputs with FP32 accumulate |
| 7.5 | Turing | T4, RTX 20-series, GTX 16-series | INT8 and 4-bit/1-bit tensor ops (2nd-gen tensor cores) |
| 8.0 | Ampere (datacenter) | A100, A30 | BF16, TF32, FP64 tensor, 2:4 structured sparsity, async copy |
| 8.6 | Ampere (consumer) | RTX 30-series, A40, A10 | The same 3rd-gen feature set on consumer silicon |
| 8.9 | Ada Lovelace | RTX 40-series, L4, L40S | FP8 (E4M3/E5M2) in tensor cores |
| 9.0 | Hopper | H100, H200 | TMA, thread-block clusters, distributed shared memory, warpgroup MMA |
| 10.0 / 10.3 | Blackwell (datacenter) | B200, GB300 | FP4/FP6 and the microscaling formats (MXFP8/MXFP4) |
| 12.0 / 12.1 | Blackwell (consumer, GB10) | RTX 50-series, RTX PRO Blackwell, DGX Spark | FP4 on consumer and workstation parts |
Two steps carry most of the practical weight. BF16 starts at 8.0 — everything below it emulates the type, which is why a Pascal or Turing card has usable-but-slower inference rather than none. FP8 starts at 8.9, and that is where engines that were written around FP8 draw their line.
The toolkit has its own ladder, one release behind: CUDA 12.0 removed Kepler (“Kepler architecture support is removed from CUDA 12.0”, CUDA 12.0 release notes), and CUDA 13.0 removed offline compilation and library support for Maxwell, Pascal and Volta — “CUDA 13.0 supports all NVIDIA architectures from Turing through Grace Blackwell” (CUDA 13.0 release notes). Turing (7.5) is therefore the floor for building anything from source today, whatever the engine claims. NVIDIA’s own cc-per-card tables are at CUDA GPUs and, for retired parts, legacy CUDA GPUs.
AMD: two ladders that never meet
AMD identifies a GPU by a gfx target (gfx1030, gfx1100, gfx1201, gfx942). It reads like a version because it is one: the target is the GFXIP version written as major.minor.stepping, straight from LLVM’s processor definitions — GFX1201 … [12, 0, 1], GFX90A … [9, 0, 0xa], GFX942 … [9, 4, 2] (LLVM GCNProcessors.td). Your card prints it:
rocminfo | grep gfx # gfx1201
The trap is that AMD has two ladders running in parallel, and they are not comparable:
- CDNA (
gfx9xx) for Instinct — the datacenter line. - RDNA (
gfx10xxtogfx12xx) for Radeon — the consumer line.
A gfx942 (CDNA 3, MI300X) and a gfx1201 (RDNA 4, RX 9070) are not older and newer versions of each other. They are different lines with different instructions.
| Step | gfx targets | Cards | Matrix units | FP8 | FP4 |
|---|---|---|---|---|---|
| GCN 3-5 / Vega | gfx803, gfx900, gfx906 | RX 470-580, Vega 56/64, MI25, MI50 | none | no | no |
| RDNA 1 / 2 | gfx1010, gfx1030 | RX 5700 XT, RX 6800/6900 XT | none | no | no |
| RDNA 3 | gfx1100, gfx1101, gfx1102, gfx115x | RX 7900 XTX/XT, 7800 XT, 7600, PRO W7900 | WMMA | no | no |
| RDNA 4 | gfx1200, gfx1201 | RX 9060/9070, Radeon AI PRO R9700 | WMMA | OCP (E4M3/E5M2) | no |
| CDNA 1 / 2 | gfx908, gfx90a | MI100, MI210/MI250 | MFMA | no | no |
| CDNA 3 | gfx942 | MI300A/MI300X/MI325X | MFMA | FNUZ only | no |
| CDNA 4 | gfx950 | MI350X, MI355X | MFMA | OCP | yes (E2M1) |
Two details worth knowing before you compare an AMD card with an NVIDIA one.
There are two FP8 formats, and generations disagree on which one. CDNA 3 supports AMD’s own FNUZ encoding; CDNA 4 and RDNA 4 support the OCP encoding the rest of the industry uses. CDNA 3 can do FP8 and still be the wrong FP8 for a kernel written against Hopper, which is why ROCm’s own precision table lists them as separate rows with no overlap. RDNA 2 and RDNA 3 have no FP8 at all, and FP4 exists on CDNA 4 only.
“Supported” means AMD ships kernels for that target in that release. ROCm builds its libraries per gfx target; a target that is not in the release’s table has no prebuilt kernels. The current system requirements mark MI25 (gfx900) and MI50 (gfx906) unsupported, and ROCm 7.0 states that “the AMD GPU targets gfx803 and gfx900 are no longer built by default” (ROCm 7.0 release notes). The workaround the projects themselves recommend is to lie about the card’s identity: HSA_OVERRIDE_GFX_VERSION=11.0.0 makes an RDNA 3 part answer as a supported sibling (llama.cpp build docs).
Intel: families, not numbers
Intel has no compute-capability integer. Toolchains key on a codename (dg2, bmg, pvc, lnl, ptl, cri) plus a four-part GMDID version that sycl-ls prints — an Arc A770 comes back as 12.55.8, a Battlemage part as 20.1.4 (llama.cpp SYCL backend).
| Step | Codename | Chips | Matrix hardware |
|---|---|---|---|
| Gen9 / Gen11 | gen9, gen11 | Skylake through Ice Lake iGPUs | none |
| Gen12 / Xe-LP | gen12lp, tgl, adl | Tiger Lake, Alder Lake iGPUs, DG1 | none (int8 dot-product only) |
| Xe-HPG | dg2, acm | Arc A-series (A380, A750, A770) | XMX: int8, fp16, bf16, restricted shapes |
| Xe-HPC | pvc | Data Center GPU Max | XMX full: adds tf32 |
| Xe2 | bmg, lnl | Arc B-series (B580), Core Ultra 200V | XMX: int8, fp16, bf16, tf32 |
| Xe3 / Xe3p | ptl, cri | Panther Lake, Crescent Island | FP8 and FP4 DPAS appear here only |
Intel’s own compiler still lists Gen9, Gen11, Gen12, Xe and Xe2 as supported platforms (IGC), but the runtime packages split: “regular packages support Gen12 and later devices. Support for Gen8, Gen9 and Gen11 devices will be delivered via packages with legacy1 suffix” (LEGACY_PLATFORMS.md). In practice the oneAPI road starts at 11th-generation Core and up; older Intel GPUs fall back to OpenCL, “although the performance is not optimal, and some GPUs may not support OpenCL nor have any GPGPU capabilities”.
Software gates on capability queries here, not on a comparison: SYCL asks the device which joint_matrix combinations are legal and gets errc::kernel_not_supported otherwise, and the fp8/fp4 DPAS instructions are emitted only for -device cri.
Apple: GPU families, one rung per chip generation
Apple’s ladder is the Metal GPU family, checked with MTLDevice.supportsFamily(_:). The chip-to-family mapping is in Apple’s Metal Feature Set Tables (May 2026 revision), and the rungs that matter for inference are:
| Family | Chips | What it gates |
|---|---|---|
| Apple6 | A13 | The bfloat data type exists |
| Apple7 | A14, M1 | SIMD-scoped matrix multiply — the matrix path inference kernels need |
| Apple8 | A15, A16, M2 | — |
| Apple9 | A17 Pro, A18, M3, M4 | — |
| Apple10 | A19, M5 | A neural accelerator in every GPU core, driven by Metal 4 tensor operations |
M1 is the entry ticket: simdgroup_matrix appears at Apple7, so an A13 or an Intel Mac GPU has no SIMD-group matmul path. llama.cpp reads exactly these rungs — has_simdgroup_mm = supportsFamily:MTLGPUFamilyApple7, bf16 from Apple6 or Metal 3, and the tensor path from the Metal 4 family, disabled again unless the chip name contains M5, M6 or A19 (ggml-metal-device.m). Note the disagreement to expect: the tables put the bf16 data type at Apple6 while some frameworks require M2 for bf16 maths, so read the engine’s own check, not the marketing.
Apple’s newest rung is a real change rather than a wider feature set: on Apple10 hardware each GPU core carries a neural accelerator, and MLX reaches it through Metal 4 tensor operations (Apple ML research). MLX does not call supportsFamily at all — it reads the GPU architecture generation and enables those kernels at generation 17 and up (MLX device.cpp).
Intel and AMD Macs are not on this ladder: they report the Common and Mac families, so no Apple-family check can pass. They still run the Metal backend, just without the SIMD-group matrix path.
Vulkan: no ladder on purpose
The cross-vendor backend has no generation number at all. It asks the device which optional features it exposes, and the engine enables each one separately: VK_KHR_cooperative_matrix (coopmat1), VK_NV_cooperative_matrix2 (coopmat2), VK_KHR_shader_float16_int8, VK_KHR_shader_bfloat16, VK_KHR_shader_integer_dot_product (ggml-vulkan.cpp). llama.cpp logs the outcome on startup — matrix cores: KHR_coopmat, NV_coopmat2, or none.
There is a second, easily missed gate in the same place: the shaders must have been generated at build time, so the build probes glslc for each extension and silently omits the shaders it cannot compile. A capable GPU paired with an old build reports no matrix cores. The env vars GGML_VK_DISABLE_COOPMAT, GGML_VK_DISABLE_COOPMAT2, GGML_VK_DISABLE_BFLOAT16, GGML_VK_DISABLE_INTEGER_DOT_PRODUCT and GGML_VK_DISABLE_F16 switch each one off by hand — the fastest way to tell whether a crash comes from a path your card claims but cannot run.
Who refuses, and who degrades
The ladder only matters because engines sit at different points on it. Same card, three different outcomes:
| Engine | Where the gate is | Outcome below the floor |
|---|---|---|
| llama.cpp (CUDA) | build-time arch list | Builds 50/61/70-virtual when the toolkit is older than 13, plus 75/80-virtual, 86-real, 89-real, 90-virtual. Cards covered by a -virtual entry run through PTX the driver compiles on first launch; a card in neither list fails at run time with the CUDA driver’s “no kernel image is available” message. No refusal, no warning. |
| vLLM | build, then runtime | The build intersects your cards with a supported list per toolkit version and aborts with “No supported CUDA architectures; the build would produce a binary with no usable kernels”. At run time, bf16 raises: “Bfloat16 is only supported on GPUs with compute capability of at least 8.0” (CMakeLists.txt, cuda.py). |
| TensorFold | build, then startup | The build carries one SASS image per target and the default is sm_121: const sms = b.option([]const u8, "sm", "SASS targets, comma separated (121; later 120,89)") orelse "121" (cuda.zig). Its README names GB10 — the DGX Spark — as the qualified CUDA platform — and pins Metal to apple_m1 (README, packaging). Until 1.0.0 the Python engine set MIN_CAPABILITY = (8, 9) and refused a slower card by name; the Zig engine replaced it, and an Ada or Ampere card now has no image to load at all. |
That is the whole practical distinction: an engine that compiles per model family refuses, an engine that compiles a broad arch list degrades. “Will it fit” tells you the first failure; the ladder is the second one.
Read your own card’s step
| Vendor | Command | What it prints |
|---|---|---|
| NVIDIA | nvidia-smi --query-gpu=compute_cap --format=csv | 8.6 |
| AMD | rocminfo | grep gfx | gfx1201 |
| Intel | sycl-ls | Intel(R) Arc(TM) A770 Graphics 12.55.8 |
| Apple | metal-arch | the architecture string behind MTLArchitecture.name |
On Linux, clinfo covers OpenCL-only Intel parts, and on Windows the same NVIDIA query works from a nvidia-smi prompt.
What this means in practice
- Fitting and running are separate checks. Run both before you blame the model or the quant.
- Read the engine’s own floor, not the vendor’s generation name: the number that matters is the one in its build script or README.
- Do not compare across lines.
gfx942is not older thangfx1201, Apple8 is not “NVIDIA 8”, and Intel’s12.55.8is not a compute capability. - The toolkit pins you too. If your card is pre-Turing, a CUDA 13 container cannot build for it at all, so an old card means old images.
- A card one step below a floor is not slow, it is absent. There is no setting to make a missing kernel image exist.
- When you buy, the step decides which engines you can run later: Ada (8.9) is currently the entry ticket for FP8 kernels, M1/Apple7 for the Mac matrix path, RDNA 3 for AMD’s WMMA. A cheaper card one rung lower stays on the compatibility list for the degrade-instead-of-refuse engines — GPUs has the numbers, Runtimes says what each engine loads and on what backend.
Related: Out of memory, Which quant should I pick, Will it fit, Will it run.
Questions
- My model fits in VRAM, so why will the engine not start?
- Because fitting is a memory question and running is a compiled-code question. An engine ships GPU kernels built for particular hardware generations; a card below the engine’s floor has no kernel image to run. The check happens at startup, before any weight is loaded.
- Is a higher number always a newer, better card?
- Only on NVIDIA, where compute capability is one ordered line. AMD runs two lines that never meet (CDNA for Instinct, RDNA for Radeon), Intel names families instead of numbering them, and Apple’s tiers are per-family feature sets. Do not compare a number across two vendors, or across AMD’s two lines.
- Does this matter if I only run llama.cpp?
- Less. llama.cpp compiles for a range of architectures, including PTX the driver compiles on first run, so an older card usually runs on a slower path instead of being refused. Engines built per model family (TensorFold, vLLM) refuse instead of degrading.
- How do I find my own card's step?
- One command per vendor:
nvidia-smi --query-gpu=compute_cap,rocminfo | grep gfx,sycl-lsfor Intel,metal-archon a Mac.