Skip to content
Saafir HaleSecurity & InfrastructureAtlanta, GA
← Software & Apps

AI Orchestra.

A local multi-model inference cluster across CPU-only laptops on a LAN, behind one OpenAI-compatible endpoint.

Personal build
About

AI Orchestra turns a handful of ordinary CPU-only laptops into one inference service. Every client talks to a single OpenAI-compatible gateway that routes each request to whichever node and model can actually serve it.

The design is driven by measurements on the real hardware rather than assumptions. On an i7-8650U with 16 GB and no discrete GPU, qwen2.5:3b generates about 10.4 tok/s and llama3.2:1b about 14.7 tok/s, which made the 1B model the router and tie-breaker and the 3B the workhorse. qwen3:4b was benchmarked and rejected.

The same measurements exposed that prefill, not generation, dominates latency — roughly 64 seconds of a 90-second response went to reading context. That finding shaped routing, context budgets, and the rule for when a request escalates to a cloud model instead of being served locally.

Built with
PythonOllamaOpenAI-compatible APIOpen WebUIPowerShell
NextStudyDesk →