89 entries · Checked 2026-10-10

Runtimes

Loads

89 engines listed

runtime130,123

llama.cpp

The C/C++ core most local AI runs on: GGUF, llama-server, llama-bench. Runs on CPU, CUDA, ROCm, Vulkan, SYCL and Metal.

Loads
gguf
Runs on
CPU · CUDA · ROCm · Vulkan · SYCL · Metal
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/ggml-org/llama.cpp
runtime6,102

Strata

Single-purpose C++ engine that runs Qwen3.8-Flash-Next, a 125 B mixture-of-experts model, on one 12 GB consumer card by keeping the weights it is not using in system RAM. 32 GB of RAM is the floor and 64 GB runs every quant size; two or three cards can share the model. It serves an OpenAI/Anthropic-compatible API on localhost.

Loads
gguf
Runs on
CUDA (RTX 20-50) · ROCm (RX 6000-9000, R9700)
Multi-GPU
yes
KV cache
full precision
API
OpenAI-compatible

MIT licence, first released September 2026. It loads GGUF quantizations of Qwen3.8-Flash-Next and nothing else. The throughput figures in its documentation are the project's own measurements, with no independent benchmark yet.

github.com/Niko1221/Strata
runtime3,273

ik_llama.cpp

Fork of llama.cpp with extra quants (IQK) and tweaks that often win on CPU and mixed CPU/GPU rigs.

Loads
gguf
Runs on
CPU · CUDA · ROCm
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/ikawrakow/ik_llama.cpp
runtime182,049

Ollama

One command to pull and run a model, with an OpenAI-compatible API. Easiest way in, llama.cpp underneath.

Loads
gguf
Runs on
CPU · CUDA · ROCm · Metal
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/ollama/ollama
runtime5,329

LM Studio

Desktop app plus a local server: pick a GGUF, set GPU offload, get an OpenAI endpoint. No terminal needed.

Loads
gguf
Runs on
CPU · CUDA · ROCm · Vulkan · Metal
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/lmstudio-ai/lms
runtime11,924

KoboldCpp

Single binary with its own web UI: GGUF chat, image generation and TTS, CUDA/ROCm/Vulkan builds.

Loads
gguf
Runs on
CPU · CUDA · ROCm · Vulkan
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/LostRuins/koboldcpp
runtime26,154

llamafile

Ships a model and the runtime in one executable file that runs on Linux, macOS and Windows.

Loads
gguf
Runs on
CPU · CUDA · Metal
Multi-GPU
no
KV cache
quantizable
API
OpenAI-compatible
github.com/mozilla-ai/llamafile
runtime77,389

GPT4All

Desktop app for running GGUF models locally. Repository has been quiet since May 2025.

Loads
gguf
Runs on
CPU · CUDA · Vulkan · Metal
Multi-GPU
no
KV cache
quantizable
API
OpenAI-compatible
github.com/nomic-ai/gpt4all
runtime49,369

LocalAI

Self-hosted multi-modal engine: text, vision, voice and image models behind OpenAI-compatible endpoints.

Loads
gguf · safetensors
Runs on
CPU · CUDA · ROCm · Metal
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/mudler/LocalAI
runtime3,066

RamaLama

Serves models from OCI containers, so the runtime and its drivers travel with the model.

Loads
gguf · safetensors
Runs on
CPU · CUDA · ROCm
Multi-GPU
yes
KV cache
full precision
API
OpenAI-compatible
github.com/containers/ramalama
runtime47,721

Text generation web UI

Desktop app (formerly oobabooga) for text, vision and tool calling, with an OpenAI-compatible API.

Loads
gguf · exl2 · safetensors
Runs on
CPU · CUDA · ROCm · Metal
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/oobabooga/textgen
runtime7,198

MLX-LM

Run and quantize LLMs on Apple silicon through MLX. The default path on unified-memory Macs.

Loads
mlx · safetensors
Runs on
Metal
Multi-GPU
no
KV cache
quantizable
API
other
github.com/ml-explore/mlx-lm
runtime44,759

Jan

Offline desktop assistant that can also expose a local OpenAI-compatible server.

Loads
gguf
Runs on
CPU · CUDA · Vulkan · Metal
Multi-GPU
no
KV cache
quantizable
API
OpenAI-compatible
github.com/janhq/jan
runtime1,658

Atomic Chat

Local AI app with its own inference engine, built for agents and multi-model workflows.

Loads
gguf
Runs on
CPU · CUDA · Metal
Multi-GPU
no
KV cache
quantizable
API
OpenAI-compatible
github.com/AtomicBot-ai/Atomic-Chat
runtime657

qvac

On-device AI SDK: GGUF models, RAG, images and music with no cloud and no API keys.

Loads
gguf
Runs on
CPU · CUDA · Metal
Multi-GPU
no
KV cache
quantizable
API
other
github.com/tetherto/qvac
runtime6,084

Cactus

Quantization, kernels and runtime aimed at phones, wearables and robots rather than desktops.

Loads
gguf
Runs on
CPU · Metal · NPU
Multi-GPU
no
KV cache
quantizable
API
other
github.com/cactus-compute/cactus
serving93,059

vLLM

The reference serving engine: paged KV cache, continuous batching, tensor parallelism. Strong on both CUDA and ROCm.

Loads
safetensors
Runs on
CUDA · ROCm · CPU
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/vllm-project/vllm
serving36,716

SGLang

Serving framework with a prefix cache (RadixAttention) built for multi-turn, agents and structured output.

Loads
safetensors
Runs on
CUDA · ROCm · SYCL (Arc)
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/sgl-project/sglang
serving14,756

TensorRT-LLM

NVIDIA's engine: compiles the model into optimized kernels and graphs. Fastest on recent GeForce and datacenter GPUs.

Loads
safetensors
Runs on
CUDA
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/NVIDIA/TensorRT-LLM
serving10,884

Text Generation Inference

Hugging Face's serving stack. Still used in production, but development has slowed since early 2026.

Loads
safetensors
Runs on
CUDA · ROCm · CPU
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/huggingface/text-generation-inference
serving1,571

ExLlamaV3

Quantization and inference library tuned for one or two consumer GPUs, successor to ExLlamaV2.

Loads
exl3 · safetensors
Runs on
CUDA · ROCm
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/turboderp-org/exllamav3
serving4,633

ExLlamaV2

The exl2 engine that made big models fit on 24 GB cards. Superseded by ExLlamaV3, quiet since March 2026.

Loads
exl2
Runs on
CUDA · ROCm
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/turboderp-org/exllamav2
serving23,202

MLC-LLM

Compiles models for a target device, from phones to servers, using TVM. Same stack as in-browser inference.

Loads
mlc
Runs on
CUDA · ROCm · Vulkan · Metal
Multi-GPU
no
KV cache
full precision
API
OpenAI-compatible
github.com/mlc-ai/mlc-llm
serving19,214

WebLLM

Runs quantized models inside the browser on WebGPU. No server, no install, model cached locally.

Loads
mlc
Runs on
Metal
Multi-GPU
no
KV cache
full precision
API
other
github.com/mlc-ai/web-llm
serving12,552

OpenLLM

BentoML's packaging layer: turn any open model into an OpenAI-compatible endpoint with a few lines.

Loads
safetensors · gguf
Runs on
CUDA · ROCm · CPU
Multi-GPU
yes
KV cache
full precision
API
OpenAI-compatible
github.com/bentoml/OpenLLM
serving8,206

Dynamo

Distributed serving framework that splits prefill and decode across machines. Datacenter scale, Rust core.

Loads
safetensors
Runs on
CUDA
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/ai-dynamo/dynamo
serving2,189

TokenSpeed

Recent high-throughput serving engine, focused on squeezing latency out of the decode path.

Loads
safetensors
Runs on
CUDA
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/lightseekorg/tokenspeed
serving1,869

Sonar

Large-scale LLM inference engine, aimed at clusters rather than single rigs.

Loads
safetensors
Runs on
CUDA
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/dphnAI/sonar
serving1,820

uzu

Rust inference engine for AI models, portable across CPU and GPU backends.

Loads
safetensors
Runs on
CPU · CUDA · Metal
Multi-GPU
yes
KV cache
quantizable
API
other
github.com/trymirai/uzu
serving488

Crane

Pure Rust engine for LLM, VLM, TTS and OCR, built on Candle. Pitched as a simpler alternative to llama.cpp.

Loads
safetensors · gguf
Runs on
CPU · CUDA · Metal
Multi-GPU
yes
KV cache
quantizable
API
other
github.com/lucasjinreal/Crane
serving529

Sarathi-Serve

Research serving engine built around chunked prefill for lower tail latency. Dormant since January 2026.

Loads
safetensors
Runs on
CUDA
Multi-GPU
yes
KV cache
full precision
API
OpenAI-compatible
github.com/microsoft/sarathi-serve
serving9,813

PowerInfer

Serves local models by keeping hot neurons on the GPU and offloading the rest. Quiet since May 2026.

Loads
gguf
Runs on
CUDA · CPU
Multi-GPU
yes
KV cache
full precision
API
other
github.com/Tiiny-AI/PowerInfer
serving10,943

OpenVINO

Intel's toolkit for CPU, integrated GPU and NPU inference. The practical path on non-NVIDIA hardware.

Loads
onnx · ir
Runs on
CPU · SYCL · NPU
Multi-GPU
no
KV cache
full precision
API
other
github.com/openvinotoolkit/openvino
serving16,166

MNN

Alibaba's lightweight engine for mobile and edge devices, with its own quantized format.

Loads
mnn
Runs on
CPU · Vulkan · Metal
Multi-GPU
no
KV cache
full precision
API
other
github.com/alibaba/MNN
serving19,555

KTransformers

Runs large MoE models by keeping attention on the GPU and experts in CPU memory. Built for big sparse models on small VRAM.

Loads
safetensors · gguf
Runs on
CPU · CUDA
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible
github.com/kvcache-ai/ktransformers
serving1,077

TensorFold

Per-family engine with exact parallel decoding: each model family ships its own kernels and its own draft verification instead of one generic GEMM path. Metal on Apple silicon, CUDA on NVIDIA.

Loads
safetensors · mlx · exl3
Runs on
CUDA (12.1) · Metal
Multi-GPU
yes
KV cache
quantizable
API
OpenAI-compatible

Apache-2.0 from 0.6.0. 1.0.0 is a Zig rewrite: the Python engine the earlier notes measured is gone. Its CUDA builds carry one SASS image per target and default to sm_121, and its README names GB10 -- the DGX Spark -- as the qualified CUDA platform, so an Ada or Ampere card has no image to load until the sm_120 and sm_89 targets its build file lists as later arrive. Metal is pinned to apple_m1 and qualified from M1 to M5. Checkpoints are per family (Nemotron 3.5 Lightning 30B-A3B, Qwen3.8 Flash Next, GLM-5.3-Flash, Qwen3.5-2B), not GGUF, and `tensorfold info` prints each checkpoint's memory floor. The README reports the 27B's served speed below the Python 0.6.6 engine's, so treat earlier speed figures for this project as belonging to the engine it replaced.

github.com/ashhart/TensorFold
router60,147

LiteLLM

Gateway in front of 100+ providers and your own servers: retries and fallbacks, per-key budgets and rate limits, cost tracking, state in Redis.

Fronts
OpenAI API · Anthropic API · any OpenAI-compatible
Runs on
Linux · macOS · Docker · Kubernetes
Endpoint
OpenAI-compatible

Python. The proxy itself is MIT while the enterprise/ directory carries its own licence. The project's own benchmark reports 8 ms P95 at 1,000 requests/s.

github.com/BerriAI/litellm
router13,123

Portkey AI Gateway

One OpenAI-shaped endpoint over 1,600+ models across providers, with routing rules, retries, fallbacks, weighted load balancing and guardrails set per request.

Fronts
OpenAI API · Anthropic API · any OpenAI-compatible
Runs on
Linux · macOS · Docker · Kubernetes
Endpoint
OpenAI-compatible

MIT, TypeScript. The open-source gateway is paired with a hosted control plane; the repository has been quiet since May 2026.

github.com/Portkey-AI/gateway
router5,844

llama-swap

One endpoint, many local models: it starts and stops llama.cpp, vLLM or ComfyUI processes on demand, so a single card serves a whole list of models and the idle one is unloaded.

Fronts
OpenAI API · Anthropic API · llama.cpp · vLLM
Runs on
Linux · macOS · Windows
Endpoint
OpenAI-compatible

MIT. Single Go binary with no dependencies; serves the OpenAI and Anthropic APIs, and a model can be pinned so it is never swapped out.

github.com/mostlygeek/llama-swap
router6,036

vLLM Semantic Router

Routing layer for mixture-of-models setups: it reads signals from the request and picks or composes the model path, so the quality and cost decision stops being hard-coded in the application.

Fronts
OpenAI API · any OpenAI-compatible
Runs on
Linux · Docker · Kubernetes
Endpoint
OpenAI-compatible

Apache-2.0, Go. From the vLLM project; its routing and fusion features are documented on vllm-sr.ai.

github.com/vllm-project/semantic-router
router2,657

vLLM production-stack

Reference Kubernetes stack around vLLM: a request router with round-robin, session-ID and prefix-aware policies, KV-cache offload and its own dashboard.

Fronts
OpenAI API · vLLM
Runs on
Kubernetes
Endpoint
OpenAI-compatible

Apache-2.0. The router is Python; the project's own roadmap lists a faster router and a KV-cache-aware routing algorithm as work in progress.

github.com/vllm-project/production-stack
router5,568

RouteLLM

Sends each request to a strong or a weak model according to a trained classifier and a cost threshold, instead of paying for the expensive model on every call.

Fronts
OpenAI API · any OpenAI-compatible
Runs on
Linux · macOS · Docker
Endpoint
OpenAI-compatible

Apache-2.0. Runs as an OpenAI-compatible server, or as a library in your own code. The authors report up to 85% lower cost at 95% of GPT-4 quality on MT-Bench; the repository has been quiet since August 2024.

github.com/lm-sys/RouteLLM
router2,980

Mixture-of-Agents (MoA)

Fusion rather than routing: several models answer the same request, then an aggregator writes the final answer from their drafts, over one or more rounds.

Fronts
OpenAI API · any OpenAI-compatible
Runs on
Linux · macOS · Docker
Endpoint
OpenAI-compatible

Apache-2.0. Research code rather than a service, quiet since January 2025. Its own result is 65.1% on AlpacaEval 2.0 against 57.5% for GPT-4o, with open models only.

github.com/togethercomputer/MoA
decision

JEV (System One)

TypeSafe AI's decision models: a typed question in, a typed decision out (choice, score, boolean) with a confidence. No text generation.

Loads
api
Runs on
Cloud · CUDA · CPU
Multi-GPU
no
KV cache
full precision
API
other

The category is decision, not chat: score candidate branches off one shared prefix.

typesafe.ai
decision1,175

jevos

Open-source alternative to Jev for yes/no decisions. C++, GGUF, runs on a laptop CPU behind a FastAPI server.

Loads
gguf
Runs on
CPU · CUDA
Multi-GPU
no
KV cache
full precision
API
other
github.com/feder-cr/jev
decision1,009

AnyJev

Turns any LLM into a Jev-style decision model - typed decisions with real probabilities, no training.

Loads
safetensors · gguf
Runs on
CPU · CUDA
Multi-GPU
yes
KV cache
full precision
API
other
github.com/nokia-applied-research/AnyJev
decision6,702

laya-mlx

Native MLX runtime for Laya typed decision models: 7-14 ms short decisions on an M3 Max, no generation.

Loads
mlx
Runs on
Metal
Multi-GPU
no
KV cache
full precision
API
other
github.com/mizorewww/laya-mlx
decision910

TypeLLM

LLMs with type-safe generation: constrain the output to a declared type instead of parsing free text.

Loads
safetensors
Runs on
CUDA · CPU
Multi-GPU
no
KV cache
full precision
API
other
github.com/TypeLLM/TypeLLM
decision119

jev-forge

Training and inference stack for Jev-style models: score dynamic candidate branches from a shared prefix, with calibration.

Loads
safetensors
Runs on
CUDA
Multi-GPU
no
KV cache
full precision
API
other
github.com/zwliJay/jev-forge
decision88

open-jev-fast

Faster CUDA backend for Open-Jev-27B: fused kernels, prefix tree and CUDA graphs on bf16.

Loads
safetensors
Runs on
CUDA
Multi-GPU
no
KV cache
full precision
API
other
github.com/lyuyiqi/open-jev-fast
decision71

jevfire

Parallel decisions on top of an existing vLLM server: one context, many decisions, vLLM API.

Loads
api
Runs on
CUDA
Multi-GPU
no
KV cache
full precision
API
other
github.com/kikoncuo/jevfire
decision61

SelfJev

Qwen3.5-4B plus a LoRA that answers with typed decisions and probabilities on a single GPU.

Loads
safetensors
Runs on
CUDA
Multi-GPU
no
KV cache
full precision
API
other
github.com/Jwuthri/SelfJev
decision58

sarvam-jev

Decision models for Indic languages: constrained logit readout instead of generated JSON, runs in the browser.

Loads
api
Runs on
Cloud · CPU
Multi-GPU
no
KV cache
full precision
API
other
github.com/SAGAR-TAMANG/sarvam-jev
diffusion135,830

ComfyUI

Node-graph backend for image and video models: FLUX, Qwen Image, Wan and others, scriptable through an API.

Loads
safetensors · gguf
Runs on
CUDA · ROCm · CPU · Metal
Multi-GPU
yes
KV cache
full precision
API
other
github.com/Comfy-Org/ComfyUI
diffusion7,498

stable-diffusion.cpp

Diffusion inference in plain C/C++ (SD, FLUX, Wan, Qwen Image), the GGUF-minded sibling of llama.cpp.

Loads
gguf · safetensors
Runs on
CPU · CUDA · Vulkan · Metal
Multi-GPU
no
KV cache
full precision
API
other
github.com/leejet/stable-diffusion.cpp
diffusion165,182

Stable Diffusion web UI

The original A1111 interface. Enormous extension ecosystem, development quiet since March 2026.

Loads
safetensors
Runs on
CUDA · CPU · Metal
Multi-GPU
no
KV cache
full precision
API
OpenAI-compatible
github.com/AUTOMATIC1111/stable-diffusion-webui
diffusion13,044

Stable Diffusion webui Forge

Memory-optimized fork of A1111 for smaller cards. No commits since July 2025.

Loads
safetensors
Runs on
CUDA
Multi-GPU
no
KV cache
full precision
API
OpenAI-compatible
github.com/lllyasviel/stable-diffusion-webui-forge
diffusion28,330

InvokeAI

Creative studio around diffusion models: canvas, layers and a professional workflow on top of the same weights.

Loads
safetensors
Runs on
CUDA · CPU · Metal
Multi-GPU
no
KV cache
full precision
API
OpenAI-compatible
github.com/invoke-ai/InvokeAI
diffusion34,641

Diffusers

Hugging Face's Python library for diffusion pipelines. The base most other image tools build on.

Loads
safetensors · diffusers
Runs on
CUDA · ROCm · CPU · Metal
Multi-GPU
yes
KV cache
full precision
API
other
github.com/huggingface/diffusers
diffusion2,433

mflux

Apple MLX implementations of current image and video models, optimized for unified-memory Macs.

Loads
mlx
Runs on
Metal
Multi-GPU
no
KV cache
full precision
API
other
github.com/mflux-community/mflux
diffusion576

Draw Things

Local image generation app for Apple devices and desktop, with its own models and LoRA support.

Loads
safetensors · mlx
Runs on
Metal
Multi-GPU
no
KV cache
full precision
API
other
github.com/drawthingsai/draw-things-community
speech54,095

whisper.cpp

Whisper speech recognition in C/C++, GGML-quantized. The default local transcriber.

Loads
ggml
Runs on
CPU · CUDA · Vulkan · Metal
Multi-GPU
no
KV cache
full precision
API
other
github.com/ggml-org/whisper.cpp
speech25,673

faster-whisper

CTranslate2 implementation of Whisper, several times faster than the reference at the same accuracy.

Loads
ctranslate2
Runs on
CPU · CUDA
Multi-GPU
yes
KV cache
full precision
API
OpenAI-compatible
github.com/SYSTRAN/faster-whisper
speech15,077

sherpa-onnx

One ONNX runtime for speech recognition, synthesis, diarization and source separation, embeddable everywhere.

Loads
onnx
Runs on
CPU · CUDA · Vulkan · Metal
Multi-GPU
no
KV cache
full precision
API
other
github.com/k2-fsa/sherpa-onnx
speech3,695

Speaches

Self-hosted speech-to-text and text-to-speech behind OpenAI-compatible endpoints.

Loads
onnx
Runs on
CPU · CUDA
Multi-GPU
no
KV cache
full precision
API
OpenAI-compatible
github.com/speaches-ai/speaches
speech11,298

Piper

Fast local neural text-to-speech with small ONNX voices. Quiet upstream since August 2025.

Loads
onnx
Runs on
CPU
Multi-GPU
no
KV cache
full precision
API
other
github.com/rhasspy/piper
speech9,116

Kokoro

Small, high-quality text-to-speech model, usually run through ONNX or MLX wrappers.

Loads
onnx · mlx
Runs on
CPU · Metal
Multi-GPU
no
KV cache
full precision
API
other
github.com/hexgrad/kokoro
ui153,791

Open WebUI

Self-hosted chat front end for Ollama, vLLM or any OpenAI-compatible endpoint, with users and RAG.

Loads
n/a
Runs on
n/a
Multi-GPU
no
KV cache
full precision
API
other
github.com/open-webui/open-webui
ui66,668

AnythingLLM

Desktop and server app for chatting with your own documents, local models or APIs.

Loads
n/a
Runs on
n/a
Multi-GPU
no
KV cache
full precision
API
other
github.com/Mintplex-Labs/anything-llm
ui45,203

LibreChat

Multi-provider chat interface with agents and MCP support, deployable next to your own models.

Loads
n/a
Runs on
n/a
Multi-GPU
no
KV cache
full precision
API
other
github.com/LibreChat-AI/LibreChat
orchestrator514

OME

Kubernetes operator for LLM serving: GPU scheduling and model lifecycle as cluster resources.

Loads
n/a
Runs on
CUDA
Multi-GPU
yes
KV cache
full precision
API
other
github.com/ome-projects/ome
orchestrator6,058

KServe

Standard inference platform for Kubernetes, covering both predictive and generative models.

Loads
n/a
Runs on
CUDA · CPU
Multi-GPU
yes
KV cache
full precision
API
other
github.com/kserve/kserve
orchestrator43,963

Ray Serve

Composes multi-model pipelines and scales them across machines; used as the layer under several engines.

Loads
n/a
Runs on
CUDA · CPU
Multi-GPU
yes
KV cache
full precision
API
other
github.com/ray-project/ray
runtime6

Slipstream

C++/Metal engine for Apple Silicon that keeps a large MoE's hot experts in memory and streams the rest from SSD, so a 95.5 GiB checkpoint runs on a 64 GB Mac.

Loads
gguf
Runs on
Metal
Multi-GPU
no
KV cache
full precision
API
OpenAI-compatible

Developer-reported numbers, no independent benchmark yet: 41-52 tok/s against 23.1 tok/s for a llama.cpp fork on the same 95.5 GiB Qwen3.8-Flash-Next checkpoint, over 3,086 requests. Apple Silicon and macOS 15+ only, and the weights need about 100 GB of disk.

github.com/npanj/slipstream
agent34,098

omp (oh-my-pi)

Terminal coding agent with IDE-grade tooling that reads custom OpenAI-compatible providers (base URL) from a config file, so it can be pointed at a local server.

Talks to
OpenAI API · Anthropic API · Ollama · LM Studio · vLLM
Runs on
macOS · Linux · Windows
Local model
any OpenAI-compatible server

MIT licence.

github.com/can1357/oh-my-pi
agent250,715

Hermes Agent

Nous Research's terminal-native autonomous agent: it runs against Nous Portal, OpenRouter or any OpenAI-compatible endpoint, including a local one.

Talks to
Nous Portal · OpenRouter · OpenAI API · Anthropic API · any OpenAI-compatible
Runs on
Linux · macOS · WSL2 · Termux
Local model
any OpenAI-compatible server

MIT licence.

github.com/NousResearch/hermes-agent
agent148,941

Claude Code

Anthropic's terminal coding tool. It takes a self-hosted gateway through ANTHROPIC_BASE_URL, so a local model needs a proxy that speaks the Anthropic Messages format, not the OpenAI one.

Talks to
Anthropic API · Bedrock · Vertex AI
Runs on
macOS · Linux · Windows
Local model
via a proxy

Source not published; the repository is the public issue tracker.

github.com/anthropics/claude-code
agent127,620

OpenAI Codex CLI

OpenAI's terminal coding agent. Its config defines model providers with a base URL and an OpenAI-compatible wire API, with local Ollama and LM Studio given as examples.

Talks to
OpenAI API · Ollama · LM Studio · OpenAI-compatible
Runs on
macOS · Linux · Windows
Local model
any OpenAI-compatible server

Apache-2.0 licence.

github.com/openai/codex
agent211,451

opencode

Terminal and desktop coding agent that takes custom providers through the AI SDK's OpenAI-compatible package.

Talks to
OpenAI API · Anthropic API · OpenRouter · Ollama · LM Studio
Runs on
macOS · Linux · Windows
Local model
any OpenAI-compatible server

MIT licence.

github.com/anomalyco/opencode
agent49,332

Aider

Terminal pair-programming tool that reaches a local model by pointing OPENAI_API_BASE at an OpenAI-compatible endpoint.

Talks to
OpenAI API · Anthropic API · DeepSeek · OpenRouter · any OpenAI-compatible
Runs on
macOS · Linux · Windows
Local model
any OpenAI-compatible server

Apache-2.0 licence.

github.com/Aider-AI/aider
agent69,721

Cline

Coding agent for IDE, terminal and desktop with a dedicated OpenAI-compatible provider that takes a custom base URL.

Talks to
Anthropic API · OpenAI API · OpenRouter · Ollama · LM Studio
Runs on
macOS · Linux · Windows
Local model
any OpenAI-compatible server

Apache-2.0 licence.

github.com/cline/cline
agent24,286

Roo Code

VS Code extension that runs agentic coding tasks and takes a local server through its OpenAI-compatible provider.

Talks to
OpenAI API · Anthropic API · Ollama · LM Studio · OpenAI-compatible
Runs on
macOS · Linux · Windows
Local model
any OpenAI-compatible server

Apache-2.0 licence; the repository is archived, so treat it as frozen.

github.com/RooCodeInc/Roo-Code
agent54,867

Goose

Rust desktop and CLI agent that lets you add a local OpenAI-compatible server as a custom provider with its own API URL.

Talks to
Anthropic API · OpenAI API · Ollama · LM Studio · OpenRouter
Runs on
macOS · Linux · Windows
Local model
any OpenAI-compatible server

Apache-2.0 licence.

github.com/aaif-goose/goose
agent28,452

Crush

Terminal coding agent that takes custom providers and a base URL, so it can talk to a local llama.cpp or Ollama server.

Talks to
OpenAI-compatible · Anthropic-compatible · Ollama · OpenRouter · Bedrock
Runs on
macOS · Linux · Windows · BSD
Local model
any OpenAI-compatible server

Source-available under FSL-1.1-MIT, which is not an open-source licence.

github.com/charmbracelet/crush
agent89,797

OpenHands

Agent platform whose own documentation covers connecting local servers (LM Studio, Ollama, vLLM, SGLang) as OpenAI-compatible endpoints.

Talks to
OpenAI-compatible · Ollama · LM Studio · vLLM · SGLang
Runs on
macOS · Linux · Windows
Local model
any OpenAI-compatible server

MIT licence.

github.com/OpenHands/OpenHands
agent28,274

Qwen Code

Terminal coding agent that takes a local inference server through an OpenAI-compatible base URL alongside the hosted Qwen, OpenAI and Gemini APIs.

Talks to
Qwen API · OpenAI API · Anthropic API · Ollama · vLLM
Runs on
macOS · Linux · Windows
Local model
any OpenAI-compatible server

Apache-2.0 licence.

github.com/QwenLM/qwen-code
agent107,206

Gemini CLI

Google's terminal agent. It authenticates against the Gemini API or Vertex AI only, and documents no base-URL override, so it is a hosted-only tool.

Talks to
Gemini API · Vertex AI
Runs on
macOS · Linux · Windows
Local model
—

Apache-2.0 licence; no documented path to a local model.

github.com/google-gemini/gemini-cli
agent36,084

Continue

Coding agent for CLI, VS Code and JetBrains whose docs show setting apiBase to reach an OpenAI-compatible server; the repository is now read-only.

Talks to
OpenAI API · Anthropic API · Ollama · OpenAI-compatible
Runs on
macOS · Linux · Windows
Local model
any OpenAI-compatible server

Apache-2.0 licence; repository read-only, no longer actively maintained.

github.com/continuedev/continue
agent27,790

Cua

Infrastructure for computer-use agents instead of another coding harness: it hands an agent a real desktop. Cua Spaces runs full desktops on your own Mac and machines, Cua Driver operates apps on macOS, Windows and Linux and installs itself as an MCP server for Claude Code, Codex and Cursor, Lume runs local macOS and Linux VMs on Apple Silicon, and Cua Bench builds tasks and exports trajectories.

Talks to
Anthropic API · OpenAI API · Gemini API · local VLMs (Qwen3-VL, InternVL, Holo, Moondream, UI-TARS) · MCP
Runs on
macOS · Linux · Windows
Local model
—

MIT licence. Bring your own agent and model: the agent library ships loops for the Anthropic, OpenAI and Gemini APIs and for local transformers VLMs, so nothing here speaks an OpenAI-compatible endpoint by itself. The CUA-S1 decision models are LoRA adapters on a frozen Qwen3.5-4B, weights published separately on Hugging Face as an early research release.

github.com/trycua/cua

Six layers, not one thing

“The engine” covers six distinct layers. Separating them clarifies benchmarks and bug reports.

  • Kernels. The math: cuBLAS, ROCm, Metal, Vulkan, SYCL, AVX-512. Higher layers inherit their speed and limits.
  • Engine. Loads weights, allocates the KV cache, and splits work: llama.cpp, vLLM, SGLang, ExLlamaV3, MLX.
  • Server. Exposes the model: an OpenAI-compatible API port, a task queue, or a gRPC endpoint.
  • Gateway. Routes requests to one or more servers. It swaps models on the same card, sends requests to cheaper models, or spreads load across backends. It handles keys, budgets, retries, and fallbacks. It loads no weights. Single-machine setups often skip this layer, which causes port conflicts when two tools run on one rig.
  • App. The user interface: Open WebUI, ComfyUI, LM Studio, KoboldCpp.
  • Agent. Uses the model on your behalf. A coding agent reads files, runs commands, and manages its own context. It connects via the same OpenAI-compatible port as any client, so a local server suffices for offline use. Agents drive context pressure: they resend a growing transcript every turn. This creates a long-context problem, not a model-size problem.

A “slow model” usually reflects a bottleneck at one of these six layers, not a slow GPU.

Which one, by what you have

  • One card up to 24 GB, or CPU only. GGUF and llama.cpp (or a UI built on it: Ollama, LM Studio, KoboldCpp). Use quantized weights, KV cache in q8_0, and partial GPU offload. Use ik_llama.cpp for extra quant formats.
  • Two to four consumer cards. Use llama.cpp for GGUF, or vLLM/ExLlamaV3 if you hold full-precision weights. Tensor parallel and layer split matter in this range.
  • 96 GB and up on one machine. Use vLLM or SGLang with batching for multiple users, long contexts, and an OpenAI endpoint. Use TensorRT-LLM if the stack is NVIDIA-only and you need the last 10 % performance.
  • Apple silicon, 128 GB and up. Use MLX / MLX-LM, or GGUF through llama.cpp with Metal.
  • No NVIDIA at all. Use ROCm with llama.cpp or vLLM on recent Radeons, OpenVINO on Intel, or Vulkan as the fallback.
  • Image and video. Use ComfyUI, or stable-diffusion.cpp for a single binary with GGUF quantization.
  • Speech. Use whisper.cpp (or faster-whisper) for input, Piper/Kokoro for output, and sherpa-onnx for embedded needs.
  • A whole cluster. Use OME or KServe on Kubernetes, with Dynamo or Ray Serve underneath.
  • Several models on one card, or one endpoint for several servers. Use llama-swap to start and stop local models on demand behind a single port. Use LiteLLM to add keys, budgets, and cloud providers to that port. Use vLLM Semantic Router when the choice depends on the request content. Use production-stack for a cluster of vLLM pods.

The decision engines: JEV and friends

JEV (TypeSafe AI’s System One models) is not a chat model and not a smaller vLLM. It takes a typed question and returns a typed decision — a choice, a score, or a boolean — with confidence. It reads logits from one forward pass over a shared prefix. It generates no tokens. VRAM and latency drop by an order of magnitude: a few milliseconds and a few gigabytes on a laptop.

JEV fits classification, routing, rubric scoring, verification, and agent guardrails. It replaces the pattern of paying for a chat model and parsing its answer. Open implementations exist (jevos, AnyJev, laya-mlx). The same idea appears in TypeLLM for type-safe generation.

How to read this page

Cards group by type and filter. Two rows control filtering and combine: the first selects software kind, the second selects Loads — the file format an engine runs. Selecting GGUF shows only engines that run GGUF. backends shows current hardware support. multi-GPU indicates whether one model splits across cards. KV cache indicates whether the context cache quantizes — the cheapest way to buy context on a small card.

Star counts snapshot popularity on 2026-10-02 (2026-10-05 for routing cards, 2026-10-07 for TensorFold). They signal rough popularity; a quiet repository is not always dead. Notes flag silent projects.

Some engines enforce a hardware floor. The server refuses to start if the card falls below a minimum generation. These floors appear in the card notes. The underlying numbers sit on The architecture ladder. Will it run checks one card against all floors and reports whether the engine stops or loses fast paths.

Questions

Which engine should I start with?
One card up to 24 GB, or CPU only: GGUF with llama.cpp, or a UI built on it such as Ollama, LM Studio or KoboldCpp, with a quantised KV cache and partial offload.
Do I need vLLM?
Only when you serve several users with long contexts, or hold full-precision weights across two to four cards and up. That is where batching and tensor parallelism pay for themselves; for one user llama.cpp is simpler and often faster.
What makes a coding agent heavy on memory?
It re-sends a growing transcript every turn, so its cost is a long-context problem rather than a model-size problem. The KV cache, not the weights, is what you run out of.
Will an older card run these engines?
Not every one of them. A few engines are compiled for a minimum generation of card and refuse to start below it, so the card that fits the model can still be rejected by the software: TensorFold 1.0.0 builds one CUDA image per target and defaults to sm_121, the DGX Spark’s GB10, and names no other qualified CUDA platform, so an RTX 30-series or 40-series card has no image to load. Such floors are written in the note on the card, and /gpus/architecture-ladder/ explains what each generation number means per vendor. /will-it-run/ compares your card with every floor this site has been able to source, and quotes the line it came from.