What you’ll need

  • A computer — 8 GB of RAM at the very least (16 GB or more is better), 22 GB of free disk for the smallest model, and an internet connection. Linux, macOS, or Windows. A graphics card is optional.
  • A fast disk if you want the big models. For the streaming models the disk sets the speed — an NVMe drive helps most.
  • Bandwidth for the download. 23 GB for the smallest chat model; 429 GB for GLM-5.2; 1.56 TB for Kimi K3. Downloads resume where they stopped.
  • git and python3 (plus build-essential on Linux). The setup prints the exact install command if anything is missing.

Step 1 — run the one-step setup

# Debian / Ubuntu first:
sudo apt install git python3 build-essential

git clone https://github.com/JustVugg/colibri && cd colibri
./start-here.sh

On macOS: xcode-select --install and brew install libomp git python, then the same two lines. On Windows, download the ZIP and double-click START-HERE.bat (it offers to install Python if it's missing). The setup does five things: it surveys your machine (RAM, free disk, CPU, GPUs), recommends the most capable model that fits — the always-resident part plus a minimum cache of experts must fit in your RAM, the download on your disk —, builds the engine for your machine (or downloads a prebuilt one: CPU always, Vulkan or CUDA for a GPU), downloads the model with progress and resume, then starts colibri and opens the dashboard. Run it again later and it just starts — no rebuild, no re-download. If you use an AI coding assistant, you can also have it do all of this: the repo ships docs/AI_SETUP.md with every step as a command, plus an MCP server (coli mcp) that can detect hardware, recommend, install, start, and stop.

Step 2 — survey the model catalog

./start-here.sh --list

This shows every supported model against your machine. RAM is two numbers: below the first the model does not start, from the second it runs as measured. Some highlights from the project's own benchmark table:

  • Qwen3.6-35B-A3B (--model qwen36-35b): 23 GB download, 10/20 GB RAM, ~6 tok/s on the CPU — the sane first model.
  • DeepSeek V4 Flash, 284B (--model deepseek-v4-flash): 167 GB, 16/32 GB, 0.93 tok/s on a Ryzen 7 5800X.
  • GLM-5.2, 744B (--model glm-5.2): 429 GB, 16/24 GB, 0.05–0.1 tok/s cold on a 25 GB laptop — a demo at that speed, not a chatbot.
  • Kimi K3, 2.8T (--model kimi-k3): 1.56 TB, 32/64 GB, about 9.4 s per token.

Install one with ./start-here.sh --model qwen36-35b, or pass --yes to take the recommendation with no questions. Other useful options: --dir DIR to put the models on another disk, --backend cpu to skip the GPU entirely, --reconfigure to switch models later.

Step 3 — chat in the terminal

./coli chat

The banner tells you the whole story, e.g. colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU / √ ready in 32s · resident 9.9 GB. That last number is the point: the entire 744B model lives on disk, under 10 GB sits in RAM. c/coli status shows what's installed and whether it runs; c/coli stop stops it. Handy extras on the same launcher: ./coli plan --model /nvme/qwen36 previews where the model will live (VRAM, RAM, disk), ./coli doctor is a read-only readiness check, and ./coli tune measures and saves this machine's fastest safe settings.

A hummingbird feeding at a flower, wings a blur — only the experts the router asks for leave the disk
Only the routed experts leave the disk — the cache learns which ones your conversations use. Photo: Pexels (free stock photo).

Step 4 — serve it to your other apps

./coli serve            # API + dashboard, no browser

The setup already starts this server for you; it prints the addresses other apps can use. It is one server with several APIs: OpenAI-compatible (/v1/chat/completions, /v1/completions, /v1/models — with streaming, JSON replies, stop sequences, and logprobs), Anthropic-compatible (/v1/messages, so Claude Code and the Anthropic SDKs work against it), tool calling on every chat engine except Inkling and OLMoE, and image input on the vision-capable models. Try it:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "<the id `coli status` prints>",
       "messages": [{"role": "user", "content": "In one sentence: why can a mixture-of-experts model stream its experts from disk?"}]}'

Coding CLIs and editors connect as to any OpenAI-compatible provider: base URL http://127.0.0.1:8000/v1, the model id coli status prints, any non-empty key.

Step 5 — watch the router work in the dashboard

./coli web              # API + dashboard, opens a browser

The dashboard also lives at http://127.0.0.1:8000/. Beyond chat, it has the System One page (give the model a document and the only answers it may pick — you get a probability per allowed answer), the Profiling page (where each turn spends its time, phase by phase), and the Brain: the measured expert atlas of GLM-5.2, 13,260 characterized experts in ten regions, with live routing that flashes every expert your turn actually fired. It's the clearest visual explanation of why the model is small per token.

A hummingbird hovering in mid-air — on the Brain page, one cell per expert flashes as the router fires
The Brain page: one cell per expert, colored by storage tier, flashing as the router fires. Photo: Pexels (free stock photo).

What you built

A local inference stack for frontier mixture-of-experts models: one-command install, terminal chat, an OpenAI- and Anthropic-compatible API on 127.0.0.1:8000, and a dashboard that shows the MoE router working in real time — all on hardware you already own.

Honest limitations

  • The 744B headline is a demo at laptop scale. 0.05–0.1 tokens per second on a 25 GB laptop means 10–20 seconds per token. The everyday-usable models are the smaller ones: Qwen3.6-35B at ~6 tok/s on CPU (~9.9 with Vulkan on integrated graphics).
  • The disk is the speed limit. For streaming models, how fast experts arrive from the disk sets the pace — an NVMe helps most, an ordinary disk is slower, and RAM still matters for the resident dense part plus the expert cache.
  • The downloads are enormous. 429 GB for GLM-5.2, 1.56 TB for Kimi K3 — resumable, but plan the bandwidth and the free space.
  • Only MoE architectures benefit. Dense models have no experts to stream; the trick works because each token activates ~5.4% of the model. The engine covers thirteen families — ten chat, one image model, and decision models.
  • GPU builds can fall back. If a build fails (e.g. the installed CUDA toolkit no longer supports the card), the setup picks the next backend itself — Vulkan, then CPU — and ./start-here.sh --backend cpu forces CPU from the start.