topic

inference.

Everything we've published on inference across guides, agents, hardware reviews and glossary entries: 6 entries in total.

Guides (1)

  • GPU vs CPU for self-hosted AI inference: when each wins in 2026AI Agents · 2026-10-17

    When does a GPU pay for itself in self-hosted AI inference, and when is a modern CPU the better answer? A reasoned comparison of four kinds of host (Raspberry Pi 5, Intel NUC 13, Mac Mini M4, a single-GPU box): what each can run, what it costs to keep on, and where the CPU wins. Orders of magnitude, not a measured benchmark.

Hardware (3)

  • Raspberry Pi 5

    Raspberry Pi 5 for self-hosted AI: four Cortex-A76 cores, 1 to 16 GB of LPDDR4X. Hosts Hermes Agent (no browser tool) or Nanobot; local models up to about 3B.

  • Mac Mini M4 / M4 Pro

    Mac mini M4 and M4 Pro for local LLMs: 16 to 64 GB of unified memory at 120 or 273 GB/s, the two specs that decide which models load and how fast.

  • Orange Pi 5 Plus

    Orange Pi 5 Plus: Rockchip RK3588, 4, 8 or 16 GB of LPDDR4/4X and a 6 TOPS NPU. The Raspberry Pi 5's closest competitor for self-hosted AI on ARM.

Glossary (2)

  • Quantisation :Reducing the numerical precision of model weights to shrink memory footprint and speed up inference, with quality tradeoff.
  • llama.cpp :C++ inference engine for LLMs. The runtime under most local-LLM setups, including Ollama.