Home / infra/local-ai.md

Local AI

Source: /workspace/tailscale/LOCAL-AI.md.

oMLX on the Mac mini. OpenAI-compatible /v1. Shared with tailscale serve (tailnet only, no Funnel).
Models: Ornith-1.5-9B (main chat, about 17 tok/s), gemma-4-e4b-it-4bit (fast), bge-m3-mlx-8bit (embeddings), Qwen3-TTS.
One 9B model at a time. Max 2 concurrent requests. First call loads for about 6 s.
Box calls go through the SOCKS5 proxy. The API key is a file on the box: ~/.config/omlx/key. Never print it.
Ollama runs on the mini, local only. Reach it over SSH. Models (2026-10-16): qwen3:8b, qwen3.5:9b, qwen3:4b, gemma3:4b, gemma4:e4b, Qwen3.6-35B-A3B IQ2_M (too big for the GPU budget).
GPU budget on the 16 GB M4 is about 11.3 GiB. Never run local models on the box (no GPU, low RAM).
Model routing research and experiments: /workspace/model-orchestration/ (research.md, tracker.csv with EXP-001 to EXP-009, all proposed). Ledger is proposed as owner of the numbers.
Caveat: oMLX is not a login item. After a mini reboot run omlx start.
From the box, reach it only through the SOCKS5 proxy. A direct call returns nothing (Hermes Tuner audit 2026-10-16). Whether oMLX answers through the proxy is still unverified.