89 entries · Checked 2026-10-10
Runtimes
89 engines listed
llama.cpp
The C/C++ core most local AI runs on: GGUF, llama-server, llama-bench. Runs on CPU, CUDA, ROCm, Vulkan, SYCL and Metal.
github.com/ggml-org/llama.cpp runtime6,102Strata
Single-purpose C++ engine that runs Qwen3.8-Flash-Next, a 125 B mixture-of-experts model, on one 12 GB consumer card by keeping the weights it is not using in system RAM. 32 GB of RAM is the floor and 64 GB runs every quant size; two or three cards can share the model. It serves an OpenAI/Anthropic-compatible API on localhost.
MIT licence, first released September 2026. It loads GGUF quantizations of Qwen3.8-Flash-Next and nothing else. The throughput figures in its documentation are the project's own measurements, with no independent benchmark yet.
github.com/Niko1221/Strata runtime3,273ik_llama.cpp
Fork of llama.cpp with extra quants (IQK) and tweaks that often win on CPU and mixed CPU/GPU rigs.
github.com/ikawrakow/ik_llama.cpp runtime182,049Ollama
One command to pull and run a model, with an OpenAI-compatible API. Easiest way in, llama.cpp underneath.
github.com/ollama/ollama runtime5,329LM Studio
Desktop app plus a local server: pick a GGUF, set GPU offload, get an OpenAI endpoint. No terminal needed.
github.com/lmstudio-ai/lms runtime11,924KoboldCpp
Single binary with its own web UI: GGUF chat, image generation and TTS, CUDA/ROCm/Vulkan builds.
github.com/LostRuins/koboldcpp runtime26,154llamafile
Ships a model and the runtime in one executable file that runs on Linux, macOS and Windows.
github.com/mozilla-ai/llamafile runtime77,389GPT4All
Desktop app for running GGUF models locally. Repository has been quiet since May 2025.
github.com/nomic-ai/gpt4all runtime49,369LocalAI
Self-hosted multi-modal engine: text, vision, voice and image models behind OpenAI-compatible endpoints.
github.com/mudler/LocalAI runtime3,066RamaLama
Serves models from OCI containers, so the runtime and its drivers travel with the model.
github.com/containers/ramalama runtime47,721Text generation web UI
Desktop app (formerly oobabooga) for text, vision and tool calling, with an OpenAI-compatible API.
github.com/oobabooga/textgen runtime7,198MLX-LM
Run and quantize LLMs on Apple silicon through MLX. The default path on unified-memory Macs.
github.com/ml-explore/mlx-lm runtime44,759Jan
Offline desktop assistant that can also expose a local OpenAI-compatible server.
github.com/janhq/jan runtime1,658Atomic Chat
Local AI app with its own inference engine, built for agents and multi-model workflows.
github.com/AtomicBot-ai/Atomic-Chat runtime657qvac
On-device AI SDK: GGUF models, RAG, images and music with no cloud and no API keys.
github.com/tetherto/qvac runtime6,084Cactus
Quantization, kernels and runtime aimed at phones, wearables and robots rather than desktops.
github.com/cactus-compute/cactus serving93,059vLLM
The reference serving engine: paged KV cache, continuous batching, tensor parallelism. Strong on both CUDA and ROCm.
github.com/vllm-project/vllm serving36,716SGLang
Serving framework with a prefix cache (RadixAttention) built for multi-turn, agents and structured output.
github.com/sgl-project/sglang serving14,756TensorRT-LLM
NVIDIA's engine: compiles the model into optimized kernels and graphs. Fastest on recent GeForce and datacenter GPUs.
github.com/NVIDIA/TensorRT-LLM serving10,884Text Generation Inference
Hugging Face's serving stack. Still used in production, but development has slowed since early 2026.
github.com/huggingface/text-generation-inference serving1,571ExLlamaV3
Quantization and inference library tuned for one or two consumer GPUs, successor to ExLlamaV2.
github.com/turboderp-org/exllamav3 serving4,633ExLlamaV2
The exl2 engine that made big models fit on 24 GB cards. Superseded by ExLlamaV3, quiet since March 2026.
github.com/turboderp-org/exllamav2 serving23,202MLC-LLM
Compiles models for a target device, from phones to servers, using TVM. Same stack as in-browser inference.
github.com/mlc-ai/mlc-llm serving19,214WebLLM
Runs quantized models inside the browser on WebGPU. No server, no install, model cached locally.
github.com/mlc-ai/web-llm serving12,552OpenLLM
BentoML's packaging layer: turn any open model into an OpenAI-compatible endpoint with a few lines.
github.com/bentoml/OpenLLM serving8,206Dynamo
Distributed serving framework that splits prefill and decode across machines. Datacenter scale, Rust core.
github.com/ai-dynamo/dynamo serving2,189TokenSpeed
Recent high-throughput serving engine, focused on squeezing latency out of the decode path.
github.com/lightseekorg/tokenspeed serving1,869Sonar
Large-scale LLM inference engine, aimed at clusters rather than single rigs.
github.com/dphnAI/sonar serving1,820uzu
Rust inference engine for AI models, portable across CPU and GPU backends.
github.com/trymirai/uzu serving488Crane
Pure Rust engine for LLM, VLM, TTS and OCR, built on Candle. Pitched as a simpler alternative to llama.cpp.
github.com/lucasjinreal/Crane serving529Sarathi-Serve
Research serving engine built around chunked prefill for lower tail latency. Dormant since January 2026.
github.com/microsoft/sarathi-serve serving9,813PowerInfer
Serves local models by keeping hot neurons on the GPU and offloading the rest. Quiet since May 2026.
github.com/Tiiny-AI/PowerInfer serving10,943OpenVINO
Intel's toolkit for CPU, integrated GPU and NPU inference. The practical path on non-NVIDIA hardware.
github.com/openvinotoolkit/openvino serving16,166MNN
Alibaba's lightweight engine for mobile and edge devices, with its own quantized format.
github.com/alibaba/MNN serving19,555KTransformers
Runs large MoE models by keeping attention on the GPU and experts in CPU memory. Built for big sparse models on small VRAM.
github.com/kvcache-ai/ktransformers serving1,077TensorFold
Per-family engine with exact parallel decoding: each model family ships its own kernels and its own draft verification instead of one generic GEMM path. Metal on Apple silicon, CUDA on NVIDIA.
Apache-2.0 from 0.6.0. 1.0.0 is a Zig rewrite: the Python engine the earlier notes measured is gone. Its CUDA builds carry one SASS image per target and default to sm_121, and its README names GB10 -- the DGX Spark -- as the qualified CUDA platform, so an Ada or Ampere card has no image to load until the sm_120 and sm_89 targets its build file lists as later arrive. Metal is pinned to apple_m1 and qualified from M1 to M5. Checkpoints are per family (Nemotron 3.5 Lightning 30B-A3B, Qwen3.8 Flash Next, GLM-5.3-Flash, Qwen3.5-2B), not GGUF, and `tensorfold info` prints each checkpoint's memory floor. The README reports the 27B's served speed below the Python 0.6.6 engine's, so treat earlier speed figures for this project as belonging to the engine it replaced.
github.com/ashhart/TensorFold router60,147LiteLLM
Gateway in front of 100+ providers and your own servers: retries and fallbacks, per-key budgets and rate limits, cost tracking, state in Redis.
Python. The proxy itself is MIT while the enterprise/ directory carries its own licence. The project's own benchmark reports 8 ms P95 at 1,000 requests/s.
github.com/BerriAI/litellm router13,123Portkey AI Gateway
One OpenAI-shaped endpoint over 1,600+ models across providers, with routing rules, retries, fallbacks, weighted load balancing and guardrails set per request.
MIT, TypeScript. The open-source gateway is paired with a hosted control plane; the repository has been quiet since May 2026.
github.com/Portkey-AI/gateway router5,844llama-swap
One endpoint, many local models: it starts and stops llama.cpp, vLLM or ComfyUI processes on demand, so a single card serves a whole list of models and the idle one is unloaded.
MIT. Single Go binary with no dependencies; serves the OpenAI and Anthropic APIs, and a model can be pinned so it is never swapped out.
github.com/mostlygeek/llama-swap router6,036vLLM Semantic Router
Routing layer for mixture-of-models setups: it reads signals from the request and picks or composes the model path, so the quality and cost decision stops being hard-coded in the application.
Apache-2.0, Go. From the vLLM project; its routing and fusion features are documented on vllm-sr.ai.
github.com/vllm-project/semantic-router router2,657vLLM production-stack
Reference Kubernetes stack around vLLM: a request router with round-robin, session-ID and prefix-aware policies, KV-cache offload and its own dashboard.
Apache-2.0. The router is Python; the project's own roadmap lists a faster router and a KV-cache-aware routing algorithm as work in progress.
github.com/vllm-project/production-stack router5,568RouteLLM
Sends each request to a strong or a weak model according to a trained classifier and a cost threshold, instead of paying for the expensive model on every call.
Apache-2.0. Runs as an OpenAI-compatible server, or as a library in your own code. The authors report up to 85% lower cost at 95% of GPT-4 quality on MT-Bench; the repository has been quiet since August 2024.
github.com/lm-sys/RouteLLM router2,980Mixture-of-Agents (MoA)
Fusion rather than routing: several models answer the same request, then an aggregator writes the final answer from their drafts, over one or more rounds.
Apache-2.0. Research code rather than a service, quiet since January 2025. Its own result is 65.1% on AlpacaEval 2.0 against 57.5% for GPT-4o, with open models only.
github.com/togethercomputer/MoAJEV (System One)
TypeSafe AI's decision models: a typed question in, a typed decision out (choice, score, boolean) with a confidence. No text generation.
The category is decision, not chat: score candidate branches off one shared prefix.
typesafe.aijevos
Open-source alternative to Jev for yes/no decisions. C++, GGUF, runs on a laptop CPU behind a FastAPI server.
github.com/feder-cr/jev decision1,009AnyJev
Turns any LLM into a Jev-style decision model - typed decisions with real probabilities, no training.
github.com/nokia-applied-research/AnyJev decision6,702laya-mlx
Native MLX runtime for Laya typed decision models: 7-14 ms short decisions on an M3 Max, no generation.
github.com/mizorewww/laya-mlx decision910TypeLLM
LLMs with type-safe generation: constrain the output to a declared type instead of parsing free text.
github.com/TypeLLM/TypeLLM decision119jev-forge
Training and inference stack for Jev-style models: score dynamic candidate branches from a shared prefix, with calibration.
github.com/zwliJay/jev-forge decision88open-jev-fast
Faster CUDA backend for Open-Jev-27B: fused kernels, prefix tree and CUDA graphs on bf16.
github.com/lyuyiqi/open-jev-fast decision71jevfire
Parallel decisions on top of an existing vLLM server: one context, many decisions, vLLM API.
github.com/kikoncuo/jevfire decision61SelfJev
Qwen3.5-4B plus a LoRA that answers with typed decisions and probabilities on a single GPU.
github.com/Jwuthri/SelfJev decision58sarvam-jev
Decision models for Indic languages: constrained logit readout instead of generated JSON, runs in the browser.
github.com/SAGAR-TAMANG/sarvam-jev diffusion135,830ComfyUI
Node-graph backend for image and video models: FLUX, Qwen Image, Wan and others, scriptable through an API.
github.com/Comfy-Org/ComfyUI diffusion7,498stable-diffusion.cpp
Diffusion inference in plain C/C++ (SD, FLUX, Wan, Qwen Image), the GGUF-minded sibling of llama.cpp.
github.com/leejet/stable-diffusion.cpp diffusion165,182Stable Diffusion web UI
The original A1111 interface. Enormous extension ecosystem, development quiet since March 2026.
github.com/AUTOMATIC1111/stable-diffusion-webui diffusion13,044Stable Diffusion webui Forge
Memory-optimized fork of A1111 for smaller cards. No commits since July 2025.
github.com/lllyasviel/stable-diffusion-webui-forge diffusion28,330InvokeAI
Creative studio around diffusion models: canvas, layers and a professional workflow on top of the same weights.
github.com/invoke-ai/InvokeAI diffusion34,641Diffusers
Hugging Face's Python library for diffusion pipelines. The base most other image tools build on.
github.com/huggingface/diffusers diffusion2,433mflux
Apple MLX implementations of current image and video models, optimized for unified-memory Macs.
github.com/mflux-community/mflux diffusion576Draw Things
Local image generation app for Apple devices and desktop, with its own models and LoRA support.
github.com/drawthingsai/draw-things-community speech54,095whisper.cpp
Whisper speech recognition in C/C++, GGML-quantized. The default local transcriber.
github.com/ggml-org/whisper.cpp speech25,673faster-whisper
CTranslate2 implementation of Whisper, several times faster than the reference at the same accuracy.
github.com/SYSTRAN/faster-whisper speech15,077sherpa-onnx
One ONNX runtime for speech recognition, synthesis, diarization and source separation, embeddable everywhere.
github.com/k2-fsa/sherpa-onnx speech3,695Speaches
Self-hosted speech-to-text and text-to-speech behind OpenAI-compatible endpoints.
github.com/speaches-ai/speaches speech11,298Piper
Fast local neural text-to-speech with small ONNX voices. Quiet upstream since August 2025.
github.com/rhasspy/piper speech9,116Kokoro
Small, high-quality text-to-speech model, usually run through ONNX or MLX wrappers.
github.com/hexgrad/kokoro ui153,791Open WebUI
Self-hosted chat front end for Ollama, vLLM or any OpenAI-compatible endpoint, with users and RAG.
github.com/open-webui/open-webui ui66,668AnythingLLM
Desktop and server app for chatting with your own documents, local models or APIs.
github.com/Mintplex-Labs/anything-llm ui45,203LibreChat
Multi-provider chat interface with agents and MCP support, deployable next to your own models.
github.com/LibreChat-AI/LibreChat orchestrator514OME
Kubernetes operator for LLM serving: GPU scheduling and model lifecycle as cluster resources.
github.com/ome-projects/ome orchestrator6,058KServe
Standard inference platform for Kubernetes, covering both predictive and generative models.
github.com/kserve/kserve orchestrator43,963Ray Serve
Composes multi-model pipelines and scales them across machines; used as the layer under several engines.
github.com/ray-project/ray runtime6Slipstream
C++/Metal engine for Apple Silicon that keeps a large MoE's hot experts in memory and streams the rest from SSD, so a 95.5 GiB checkpoint runs on a 64 GB Mac.
Developer-reported numbers, no independent benchmark yet: 41-52 tok/s against 23.1 tok/s for a llama.cpp fork on the same 95.5 GiB Qwen3.8-Flash-Next checkpoint, over 3,086 requests. Apple Silicon and macOS 15+ only, and the weights need about 100 GB of disk.
github.com/npanj/slipstream agent34,098omp (oh-my-pi)
Terminal coding agent with IDE-grade tooling that reads custom OpenAI-compatible providers (base URL) from a config file, so it can be pointed at a local server.
MIT licence.
github.com/can1357/oh-my-pi agent250,715Hermes Agent
Nous Research's terminal-native autonomous agent: it runs against Nous Portal, OpenRouter or any OpenAI-compatible endpoint, including a local one.
MIT licence.
github.com/NousResearch/hermes-agent agent148,941Claude Code
Anthropic's terminal coding tool. It takes a self-hosted gateway through ANTHROPIC_BASE_URL, so a local model needs a proxy that speaks the Anthropic Messages format, not the OpenAI one.
Source not published; the repository is the public issue tracker.
github.com/anthropics/claude-code agent127,620OpenAI Codex CLI
OpenAI's terminal coding agent. Its config defines model providers with a base URL and an OpenAI-compatible wire API, with local Ollama and LM Studio given as examples.
Apache-2.0 licence.
github.com/openai/codex agent211,451opencode
Terminal and desktop coding agent that takes custom providers through the AI SDK's OpenAI-compatible package.
MIT licence.
github.com/anomalyco/opencode agent49,332Aider
Terminal pair-programming tool that reaches a local model by pointing OPENAI_API_BASE at an OpenAI-compatible endpoint.
Apache-2.0 licence.
github.com/Aider-AI/aider agent69,721Cline
Coding agent for IDE, terminal and desktop with a dedicated OpenAI-compatible provider that takes a custom base URL.
Apache-2.0 licence.
github.com/cline/cline agent24,286Roo Code
VS Code extension that runs agentic coding tasks and takes a local server through its OpenAI-compatible provider.
Apache-2.0 licence; the repository is archived, so treat it as frozen.
github.com/RooCodeInc/Roo-Code agent54,867Goose
Rust desktop and CLI agent that lets you add a local OpenAI-compatible server as a custom provider with its own API URL.
Apache-2.0 licence.
github.com/aaif-goose/goose agent28,452Crush
Terminal coding agent that takes custom providers and a base URL, so it can talk to a local llama.cpp or Ollama server.
Source-available under FSL-1.1-MIT, which is not an open-source licence.
github.com/charmbracelet/crush agent89,797OpenHands
Agent platform whose own documentation covers connecting local servers (LM Studio, Ollama, vLLM, SGLang) as OpenAI-compatible endpoints.
MIT licence.
github.com/OpenHands/OpenHands agent28,274Qwen Code
Terminal coding agent that takes a local inference server through an OpenAI-compatible base URL alongside the hosted Qwen, OpenAI and Gemini APIs.
Apache-2.0 licence.
github.com/QwenLM/qwen-code agent107,206Gemini CLI
Google's terminal agent. It authenticates against the Gemini API or Vertex AI only, and documents no base-URL override, so it is a hosted-only tool.
Apache-2.0 licence; no documented path to a local model.
github.com/google-gemini/gemini-cli agent36,084Continue
Coding agent for CLI, VS Code and JetBrains whose docs show setting apiBase to reach an OpenAI-compatible server; the repository is now read-only.
Apache-2.0 licence; repository read-only, no longer actively maintained.
github.com/continuedev/continue agent27,790Cua
Infrastructure for computer-use agents instead of another coding harness: it hands an agent a real desktop. Cua Spaces runs full desktops on your own Mac and machines, Cua Driver operates apps on macOS, Windows and Linux and installs itself as an MCP server for Claude Code, Codex and Cursor, Lume runs local macOS and Linux VMs on Apple Silicon, and Cua Bench builds tasks and exports trajectories.
MIT licence. Bring your own agent and model: the agent library ships loops for the Anthropic, OpenAI and Gemini APIs and for local transformers VLMs, so nothing here speaks an OpenAI-compatible endpoint by itself. The CUA-S1 decision models are LoRA adapters on a frozen Qwen3.5-4B, weights published separately on Hugging Face as an early research release.
github.com/trycua/cuaSix layers, not one thing
“The engine” covers six distinct layers. Separating them clarifies benchmarks and bug reports.
- Kernels. The math: cuBLAS, ROCm, Metal, Vulkan, SYCL, AVX-512. Higher layers inherit their speed and limits.
- Engine. Loads weights, allocates the KV cache, and splits work: llama.cpp, vLLM, SGLang, ExLlamaV3, MLX.
- Server. Exposes the model: an OpenAI-compatible API port, a task queue, or a gRPC endpoint.
- Gateway. Routes requests to one or more servers. It swaps models on the same card, sends requests to cheaper models, or spreads load across backends. It handles keys, budgets, retries, and fallbacks. It loads no weights. Single-machine setups often skip this layer, which causes port conflicts when two tools run on one rig.
- App. The user interface: Open WebUI, ComfyUI, LM Studio, KoboldCpp.
- Agent. Uses the model on your behalf. A coding agent reads files, runs commands, and manages its own context. It connects via the same OpenAI-compatible port as any client, so a local server suffices for offline use. Agents drive context pressure: they resend a growing transcript every turn. This creates a long-context problem, not a model-size problem.
A “slow model” usually reflects a bottleneck at one of these six layers, not a slow GPU.
Which one, by what you have
- One card up to 24 GB, or CPU only. GGUF and
llama.cpp(or a UI built on it: Ollama, LM Studio, KoboldCpp). Use quantized weights, KV cache inq8_0, and partial GPU offload. Useik_llama.cppfor extra quant formats. - Two to four consumer cards. Use llama.cpp for GGUF, or
vLLM/ExLlamaV3if you hold full-precision weights. Tensor parallel and layer split matter in this range. - 96 GB and up on one machine. Use
vLLMorSGLangwith batching for multiple users, long contexts, and an OpenAI endpoint. UseTensorRT-LLMif the stack is NVIDIA-only and you need the last 10 % performance. - Apple silicon, 128 GB and up. Use
MLX/MLX-LM, or GGUF through llama.cpp with Metal. - No NVIDIA at all. Use ROCm with llama.cpp or vLLM on recent Radeons,
OpenVINOon Intel, or Vulkan as the fallback. - Image and video. Use
ComfyUI, orstable-diffusion.cppfor a single binary with GGUF quantization. - Speech. Use
whisper.cpp(orfaster-whisper) for input,Piper/Kokorofor output, andsherpa-onnxfor embedded needs. - A whole cluster. Use
OMEorKServeon Kubernetes, withDynamoorRay Serveunderneath. - Several models on one card, or one endpoint for several servers. Use
llama-swapto start and stop local models on demand behind a single port. UseLiteLLMto add keys, budgets, and cloud providers to that port. UsevLLM Semantic Routerwhen the choice depends on the request content. Useproduction-stackfor a cluster of vLLM pods.
The decision engines: JEV and friends
JEV (TypeSafe AI’s System One models) is not a chat model and not a smaller vLLM. It takes a typed question and returns a typed decision — a choice, a score, or a boolean — with confidence. It reads logits from one forward pass over a shared prefix. It generates no tokens. VRAM and latency drop by an order of magnitude: a few milliseconds and a few gigabytes on a laptop.
JEV fits classification, routing, rubric scoring, verification, and agent guardrails. It replaces the pattern of paying for a chat model and parsing its answer. Open implementations exist (jevos, AnyJev, laya-mlx). The same idea appears in TypeLLM for type-safe generation.
How to read this page
Cards group by type and filter. Two rows control filtering and combine: the first selects software kind, the second selects Loads — the file format an engine runs. Selecting GGUF shows only engines that run GGUF. backends shows current hardware support. multi-GPU indicates whether one model splits across cards. KV cache indicates whether the context cache quantizes — the cheapest way to buy context on a small card.
Star counts snapshot popularity on 2026-10-02 (2026-10-05 for routing cards, 2026-10-07 for TensorFold). They signal rough popularity; a quiet repository is not always dead. Notes flag silent projects.
Some engines enforce a hardware floor. The server refuses to start if the card falls below a minimum generation. These floors appear in the card notes. The underlying numbers sit on The architecture ladder. Will it run checks one card against all floors and reports whether the engine stops or loses fast paths.
Questions
- Which engine should I start with?
- One card up to 24 GB, or CPU only: GGUF with llama.cpp, or a UI built on it such as Ollama, LM Studio or KoboldCpp, with a quantised KV cache and partial offload.
- Do I need vLLM?
- Only when you serve several users with long contexts, or hold full-precision weights across two to four cards and up. That is where batching and tensor parallelism pay for themselves; for one user llama.cpp is simpler and often faster.
- What makes a coding agent heavy on memory?
- It re-sends a growing transcript every turn, so its cost is a long-context problem rather than a model-size problem. The KV cache, not the weights, is what you run out of.
- Will an older card run these engines?
- Not every one of them. A few engines are compiled for a minimum generation of card and refuse to start below it, so the card that fits the model can still be rejected by the software: TensorFold 1.0.0 builds one CUDA image per target and defaults to sm_121, the DGX Spark’s GB10, and names no other qualified CUDA platform, so an RTX 30-series or 40-series card has no image to load. Such floors are written in the note on the card, and /gpus/architecture-ladder/ explains what each generation number means per vendor. /will-it-run/ compares your card with every floor this site has been able to source, and quotes the line it came from.