Local LLM benchmarks on a Pi 5, mini PC, Mac or GPU

In short

• Raspberry Pi 5: 1B models at about 11 tokens per second, 3B models at about 5, in the Slips project's tests with Ollama. No published 7B figure with its setup.
• Mini PCs on integrated graphics: 2.93 tokens per second on an Intel N150 and 19.91 on an AMD Ryzen 7000-series chip, generating with a 7B model.
• Apple Silicon: 24.11 on an M4, about 50 on an M4 Pro, 92.14 on an M3 Ultra with 80 GPU cores, same 7B model.
• Nvidia cards: 93.51 on an RTX 5060 Ti, 185.10 on an RTX 5080, same 7B model.
• What fits: a 7B or 8B model at 4-bit is a 4 to 5 GB download; Llama 3.3 70B is 43 GB. Memory rules out more pairings than speed does.

Where the numbers come from

PocketClaw has run none of these tests. An earlier version of this page carried a table of tokens per second, pass rates and wattages for eight machines. Nothing was measured to produce it, so it is gone. What replaces it is published elsewhere, with the configuration of each measurement:

• llama.cpp, Apple Silicon scoreboard: discussion #4167, opened by llama.cpp's maintainer in November 2023, instructions last revised on 25 August 2026, read on 17 October 2026. Model: Llama 2 7B (6.74 billion parameters) in GGUF at Q4_0, a 3.56 GiB file. Tool: llama-bench with every layer on the GPU (-ngl 99, Metal backend), a 512-token prompt (pp512) and 128 generated tokens (tg128). The M1 to M4 rows were all run on one build, 8e672ef of 21 November 2023, so that they compare with each other; the M5 and M6 rows shown below were run on build c1d0e7a.
• llama.cpp, Vulkan scoreboard: discussion #10879, opened by a llama.cpp collaborator in December 2024, read on 17 October 2026. Same model, same quantisation, same two tests, every layer on the GPU (-ngl 100), flash attention off. Each row was run on its own build, named in the table below. The scoreboard names the chip family, not always the exact chip.
• Slips project, Raspberry Pi 5: Research and analysis of performance of LLM on the Raspberry Pi 5, Slips 1.1.24 documentation, Stratosphere Laboratory, read on 17 October 2026. Ollama on the Pi's CPU, Q4_K_M models, tokens per second on "a simple prompt". The page does not say which Pi 5 memory size was used.
• Model sizes: the tag pages of the Ollama library, read on 17 October 2026.
• Memory ceilings and bandwidth: the makers' own pages, cited in the hardware buyer's guide.

The two llama.cpp tests measure two different things. pp512 is how fast the model reads a prompt; tg128 is how fast it writes, one token after the other. An agent does a lot of both: it sends long prompts (tool definitions, history, retrieved documents) and waits for the answer.

Generation speed on one 7B model

The full figures, prompt reading included:

Apple has since replaced these Macs. The scoreboard has the current chips too, measured on the August 2026 build c1d0e7a. Newer builds are faster, so do not compare these rows with the ones above: on the same M2 Ultra, generation went from 94.27 tokens per second on the 2023 build to 125.21 on c1d0e7a with flash attention, per the same discussion.

Mini PCs are listed by chip family, as the scoreboard names them. The Ryzen 7040 chips in mini PCs such as the Beelink SER7 and the Minisforum UM790 Pro belong to the Ryzen 7000 family, but the scoreboard entry may come from another member of it. The Vulkan scoreboard has no entry for the 13th-generation Intel Core chips of the Intel NUC 13 and the Geekom IT13.

Raspberry Pi 5: small models on the CPU

In the Slips tests, Ollama runs the models on the Pi 5's CPU, its four Cortex-A76 cores. The speeds they measured:

Two more findings from the same page matter for an agent. Long prompts are slow: one of their tests sends a prompt of about 5,000 tokens, and at about 5 tokens per second a 3B model needs roughly 15 minutes for it. And llama.cpp ran 10 to 20% faster than Ollama in their comparison, which is why they deploy llama.cpp and keep Ollama for development.

We found no published measurement of a 7B model on a Pi 5 with its setup stated, so this page gives none.

Which model fits in which memory

Speed only matters once the model loads. A quantised model has to sit in memory (in RAM, in Apple's unified memory, or in a graphics card's own memory), next to its context, the operating system and the agent. The download size is the floor. Sizes of the 4-bit (Q4_K_M) builds in the Ollama library:

Against the memory the makers sell:

• Raspberry Pi 5: 1, 2, 4, 8 or 16 GB. In the Slips table above, llama3.2:3b used 4.7 GB of memory: a 3B model wants the 8 GB board.
• Mac mini M4: 16, 24 or 32 GB. 7B and 8B models fit at any size; the 13 GB Mistral Small file leaves little room at 16 GB, and the 20 GB Qwen 2.5 32B file does not load there.
• Mac mini M4 Pro: 24, 48 or 64 GB. Llama 3.3 70B is 43 GB: it does not load at 24 GB, and at 48 GB it leaves about 5 GB for macOS, the context and the agent. 64 GB is the size that leaves room.
• Mac Studio M3 Ultra: 96 or 256 GB, enough for any model in the table.

The memory ceilings come from Raspberry Pi and Apple's tech specs for the Mac mini (2024) and the Mac Studio (2025). For mini PCs, the ceiling is whatever the maker allows for its memory slots: check the model before buying.

What the measurements show

Generation follows memory bandwidth. To write each token, the machine reads the whole model from memory, so the speed of that memory sets the pace; the Apple discussion plots generation speed against bandwidth for that reason. Apple gives 120 GB/s for the M4, 273 GB/s for the M4 Pro and 819 GB/s for the M3 Ultra; the measured generation speeds are 24.11, about 50 and about 90 tokens per second.

Prompt reading follows compute. Going from 16 to 20 GPU cores on the M4 Pro lifts prompt reading from 364.06 to 439.78 tokens per second and barely moves generation (49.64 to 50.74), since both versions have the same memory bandwidth. For an agent that sends long prompts, the prompt column matters as much as the generation one.

Integrated graphics range widely. On the same 7B model, the Intel N150 writes 2.93 tokens per second, the Intel Core Ultra 100 series 8.21, and the AMD Ryzen 7000 series 19.91, close to an Apple M4 (24.11).

A discrete card is fastest when the model fits in its own memory. The RTX 5080 writes 185.10 tokens per second on this 7B model, more than seven times an M4. A model larger than the card's memory has to spill into system RAM, which these tests do not measure.

The Pi 5 is for 1B to 3B models. About 11 tokens per second at 1B and about 5 at 3B is enough for short prompts and short answers, such as classifying a message; it is slow for long prompts.

What these numbers leave out

• How good the answers are. These are speed tests: a faster machine writes the same model's answers sooner, not better. Which model suits an agent is a separate question, and this page does not score it.
• 70B models. Neither scoreboard has a 70B row, so this page gives no speed for them.
• Power draw during inference. None of these sources measures it. Apple publishes idle and maximum draw for whole machines: 4 W and 65 W for the Mac mini M4, 5 W and 140 W for the M4 Pro, 9 W and 270 W for the Mac Studio M3 Ultra.
• Other quantisations. The llama.cpp rows use Q4_0. Ollama's default 4-bit builds are Q4_K_M, slightly larger files, so expect slightly different speeds.

Measure your own machine

The scoreboards publish their command so that anyone can compare. With llama.cpp built on your machine:

./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -p 512 -n 128 -ngl 99

With Ollama, --verbose prints the prompt and generation rates after each answer:

ollama run llama3.1:8b --verbose

Related guides

• Hardware buyer's guide 2026: what to buy, by budget
• Pocket AI guide: which self-hosted agents run on small hardware
• GPU vs CPU for self-hosted AI
• Raspberry Pi 5 and Mac Mini M4 pages