What a Local LLM Can Do in 2026: Gemma 4 on llama.cpp 1.0

A 14.62 GB file on my MacBook wrote ten JavaScript functions in 51 seconds. I ran each one against assertions I had written before the model saw the task, and all ten passed.

Then I let the same model think before answering and gave it the same ten tasks. It passed five, and the run took 14 minutes.

llama.cpp is the C++ library that runs LLMs on your own hardware. It tagged v1.0.0 on 2026-11-13, about three and a half years in.

The changelog has a link. I wanted to know what a model that fits on a laptop can do now, where it breaks, and what each answer costs in seconds and gigabytes.

So I wrote 138 checks with the answers fixed before any run, and pointed them at three sizes of Google’s Gemma 4, with thinking off and on. All three models ran 87 of them, and the smallest ran all 138.

Every number below is a count from those runs, and the failures are in here too.

I’ve had a few MLX models on this machine, but I’d never set up llama.cpp. It’s one binary and one command:

1
llama-server -hf ggml-org/gemma-4-12B-it-GGUF:Q4_0 -c 32768 -np 1

-hf pulls the model from Hugging Face on first use, then loads it from the cache. -c 32768 sets the context window in tokens, and -np 1 gives the whole window to one request slot.

That last flag matters more than it looks. With -c 16384 -np 4 the server log prints n_slots = 4, n_ctx_slot = 4096: four requests at once, each with a quarter of the window.

Once it prints listening on http://127.0.0.1:9931 you have an OpenAI-compatible API. Any client that speaks /v1/chat/completions works against it, with no key.

This is the machine and the build, because every speed in this post depends on them:

The laptop wasn’t idle. Other jobs kept the one-minute load average between 35 and 90 on those 14 cores for every run, so these speeds come from a busy machine and a quiet one will beat them.

These are the three models. All of them are Apache 2.0, all take images as well as text, and I used the 4-bit Q4_0 file of each:

The 26B A4B is a mixture-of-experts model. It keeps 25.2 billion parameters on disk and runs 3.8 billion of them for each token, which is why the biggest file here turns out to be the fastest.

First download and load took 245 seconds for E4B and 709 seconds for the 26B on my connection. After that, a restart loads any of them from disk in 3 to 10 seconds.

138 checks, with the answers fixed first

Every check has one right answer that I wrote down before the run. A script sends the prompt, takes the reply, and scores it with code: it runs the JavaScript, compiles the C, executes the SQL, or compares fields.

All three models ran the first six batteries, 87 checks, in both modes:

The last five batteries, 51 checks, need a second pass per model, and only E4B finished it. By then other jobs had left this laptop 6 to 11 GB of free memory, and loading a 10 GB model into that was getting their compilers killed.

So for the agent loop, SQL, regex, logic and bug finding, the 12B and 26B numbers are missing below. Missing means not run, not zero.

Every request used temperature 0, so a rerun repeats. I ran E4B’s ten JavaScript tasks twice with thinking on and got the same 10,487 tokens both times.

Tool calls first, because they decide whether a model can do anything beyond chat. I described three functions to the model: npm_package(name), github_repo(owner, repo) and convert_currency(amount, from, to).

Then I asked 14 questions. Nine need a call, one of those needs two, and five need none. This is the whole reply to “What is the latest version of zod on npm?”:

1
{
2
  "role": "assistant",
3
  "content": "",
4
  "tool_calls": [{
5
    "type": "function",
6
    "function": { "name": "npm_package", "arguments": "{\"name\":\"zod\"}" },
7
    "id": "l4ftTaXdBZYOWIRGwFad84FUPcHOjL6E"
8
  }]
9
}

The model can’t know the version, and it didn’t guess one. It stopped with finish_reason: "tool_calls" and waited.

I pulled the real numbers from the npm registry and sent them back as the tool result. The answer: “The latest version of zod on npm is 4.6.7, and it received 460,607,664 downloads last week.” Both figures match the registry.

E4B and the 26B scored 14 of 14, in both modes. The 12B scored 13 of 14 in both modes, and it missed the same question each time.

I asked for “the latest release of the Astro repo, withastro/astro”. The 12B sent this:

1
{ "name": "github_repo", "arguments": "{\"owner\":\"astro\",\"repo\":\"astro\"}" }

The owner was in the question, and it wrote a different one. A real API would have answered 404, or worse, returned somebody else’s repository.

The five no-tool questions went the right way everywhere. “What is 17 times 23?” got an answer with no call, and for “What’s the weather in Tallinn today?” no function fits, so every model declined:

I’m sorry, but I don’t have access to real-time weather data.

None of them tried to bend github_repo into a weather service. And for “Compare the star counts of facebook/react and vuejs/core”, all three sent two calls in one reply.

Next, structured extraction. I wrote ten sentences that each describe a release, and asked for four fields with a JSON schema in response_format:

All six runs turned “a dozen” into 12, read 13.11.2026 as the 13th of November, and worked out that two days before Christmas is 2025-12-23.

All 60 replies parsed as JSON. That part isn’t the model’s doing: llama-server turns the schema into a grammar and only samples tokens that fit it, so a reply can’t come out malformed.

The values are still the model’s. E4B and the 26B scored 40 of 40 with thinking off, and the other four runs scored 39, all on this sentence:

1
Released 2026/09/30: typescript@6.1.0-beta (3 breaking changes, see notes).

1
{ "package": "typescript@6.1.0-beta", "version": "6.1.0-beta", "date": "2026-09-30", "breaking_changes": 3 }

The version landed in the right field and stayed glued to the package name too. A grammar guarantees the shape of the JSON, and nothing guarantees what goes in it.

Then code. Ten tasks, each with assertions the model never sees: a slugifier, a semver comparison, interval merging, Roman numerals, a CSV line parser, an LRU cache, a topological sort, a binary search with a hang to fix, a duration formatter and a word wrapper.

With thinking off:

The 12B’s formatDuration declares a constant and then reassigns it:

1
const seconds = Math.floor(ms / 1000);
2
const d = Math.floor(seconds / (24 * 3600));
3
seconds %= (24 * 3600);

1
TypeError: Assignment to constant variable.

I changed that one const to let and all seven assertions passed. A linter would have caught it in a second, but the model had no linter and didn’t run its own code.

E4B failed on a name. I asked for mergeIntervals, and it wrote mergeInterval, so my test hit ReferenceError: mergeIntervals is not defined. With the name fixed, that code passes too.

One pass deserves an asterisk. The 26B’s compareSemver never strips build metadata, so "0+build5" parses to NaN, and every comparison against NaN is false:

1
compareSemver("1.0.0+build5", "1.0.0"); // 0, which my assertion wanted
2
compareSemver("1.0.1+build5", "1.0.0"); // 0, which is wrong

It passed my nine assertions by accident. I count it as a pass because the answer key was fixed before the run, and I’m showing it because ten green ticks would have hidden it.

Now the same ten tasks with thinking on. Gemma 4 can write a private scratchpad before the reply, and llama-server turns that on by default (--reasoning auto):

Thinking made every model worse at this. The 26B dropped from ten to five, and all five failures look the same: finish_reason: "length", an empty answer, and between 8,611 and 11,944 characters of reasoning.

The reply limit was 4,096 tokens, and thinking tokens count against it. On half the tasks the model spent the whole budget planning and never wrote the function.

llama-server has a flag for this. --reasoning-budget N caps the thinking at N tokens, and --reasoning off removes it:

1
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF:Q4_0 --reasoning-budget 1024

On E4B a 1,024-token budget finished the ten tasks in 207.7 seconds instead of 254.7, with the same 8 passes. Raising the reply limit to 16,384 changed nothing for E4B either, because E4B never hit the limit: its two failures are the wrong name and a semver comparison that returns 1 where it should return -1.

The budget run for the 26B would show whether its five lost tasks come back. It’s one of the runs I couldn’t fit in memory.

The C port is the job I cared about most. I diffed llama.h between the v0.9.0 and v1.0.0 tags: 1.0.0 deletes 36 deprecated functions.

I wrote a 15-line C file that calls six of them. Against the 0.9.0 header it compiles with six deprecation warnings, and against the 1.0.0 header it fails with 8 errors.

Each model got the file and the six warnings, with one instruction: make it compile against 1.0.0. These are three of the six lines the 12B changed:

1
llama_token bos = llama_token_bos(vocab);
2
int32_t n_vocab = llama_n_vocab(vocab);
3
size_t state_size = llama_get_state_size(ctx);
4
llama_token bos = llama_vocab_bos(vocab);
5
int32_t n_vocab = llama_vocab_n_tokens(vocab);
6
size_t state_size = llama_state_get_size(ctx);

I didn’t read the answers first. The script handed each one to the compiler:

1
cc -std=c11 -fsyntax-only -Wall -I inc-v1.0.0 src/model.c

Exit code 0 and no warnings, in all six runs. It’s a mechanical job, since each warning names its replacement, and it’s the kind of chore I’d rather hand off.

The next four batteries ran on E4B only, and they’re where thinking pays.

For SQL I gave the model a three-table SQLite schema (packages, releases, downloads) and eight questions. The script runs each query on a real database and compares the rows with a reference query.

Without thinking, E4B got 5 of 8. Asked for the percentage of releases with zero breaking changes, rounded to one decimal, it returned this:

1
SELECT CAST(SUM(CASE WHEN breaking_changes = 0 THEN 1 ELSE 0 END) AS REAL) * 100 / COUNT(*) FROM releases;

1
45.4545454545455

The number is right and the rounding I asked for is missing. A second query wrote breaking_changes = 1 where the question said “at least one”, and a third returned every release of each package where I’d asked for the latest. With thinking on, all eight passed.

Regex went worst. Six patterns, each run against strings it must match and strings it must reject, and E4B passed 2 without thinking and 3 with it.

Three of the six no-thinking answers weren’t regular expressions at all: Node threw SyntaxError: Invalid regular expression on them. Its CSS hex color pattern parsed, and then missed #fff:

1
/^#{3}([0-9a-fA-F]{3}|0-9a-fA-F]{6})$|^#[0-9a-fA-F]{6}$/

#{3} means three hash characters. I wouldn’t ship a regex from a 4.59 GB model without running it first.

The twelve arithmetic and logic questions each have one answer. E4B got 7 without thinking and 9 with it, and one miss survived both modes:

A build takes 75 seconds. How many full builds fit in 2 hours?

It answered 16, and the answer is 96. With thinking on it also counted 8 letter r’s in “strawberry raspberry”, which has 6.

Bug finding shows a numbered snippet and a symptom, and asks for the line that has to change. E4B named 5 of 6 without thinking and 6 of 6 with it.

Two of my answer keys were too strict, and I’m saying so because it changes the score. For a strcpy into an 8-byte buffer I’d keyed the strcpy line, and E4B named the buffer declaration, which fixes it too. I accept both, and under my original key the scores would be 4 and 5.

Screenshots next. Every Gemma 4 file comes with a projector that turns an image into tokens, so the same endpoint takes a PNG.

The first two screenshots hold 17 facts: a bar chart with a title and four bars, and a table with three rows of four cells.

The 12B read every title, label and number in both modes. E4B without thinking dropped the tallest bar from the chart, the one labelled “Yes” with 207, and cut the decimals off the table: 53.8 became 53 and 71.1 became 71.

The 26B without thinking read every cell and then swapped two of them, giving 46 as the slowest speed and 43.6 as the fastest. Thinking fixed both models’ tables.

Three harder screenshots went to E4B only: a code block, a diff and the top of a page, 13 facts. It read 5 without thinking and 8 with it.

The misses are small-text misses. 256GB came back as 256B, the date 2026-11-15 as 2021-11-15, and in the diff llama_n_vocab shrank to llama_vocab.

The server log explains part of it. By default an image gets at most 512 tokens, however big it is:

1
load_model: cap image_max_tokens (original=1120) to 512 (n_ubatch = 512)
2
load_model: increase n_ubatch (-ub) to increase vision token budget

I restarted E4B with -ub 2048 and ran all five screenshots again. The scores didn’t move: 12 of 17 and 5 of 13. On this model the image token budget wasn’t the limit.

The last battery pastes a whole file into the prompt. llama.h from the 1.0.0 tag is 89.6 KB, which the tokenizer turns into 25,280 tokens.

I asked five questions that each have one function name as the answer, such as “Which function copies rows from host memory into the h_nextn device buffer?” The 12B found all five, and E4B and the 26B found four.

Both of those missed the same one. “Which function returns the number of NextN layers of a model?” has the answer llama_model_n_layer_nextn, and both replied llama_model_target_layer_ids_n.

The header explains why. The wrong function carries a comment and the right one doesn’t:

1
LLAMA_API int32_t llama_model_n_layer_nextn(const struct llama_model * model);
2

3
// returns the number of extracted layers from target model
4
LLAMA_API uint32_t llama_model_target_layer_ids_n(const struct llama_model * model);

They matched my question against the comment text. The 12B read the name.

Pasting the file is one way to answer questions about it. The other is to let the model look things up, so the agent loop gives it three tools over a folder with both headers and two C files: list_files, grep, and read_file, which returns 120 lines per call.

Six questions, up to ten turns each, and no hints about which tool to use. E4B answered 3 of 6 without thinking and 5 of 6 with it.

The failures come from how it used the tools. Asked how many lines of the 0.9.0 header contain DEPRECATED, it sent this:

1
{ "name": "grep", "arguments": "{\"count_only\":true,\"pattern\":\"DEPRECATED\"}" }

No path, so the count covered all four files and came back 43. The header has 40.

Asked whether llama_n_vocab is still declared in the 1.0.0 header, it read the first 120 lines of a 1,737-line file and answered “Yes”. The function was deleted, and one grep would have shown that.

With thinking on it sent the grep with a path both times and got both right. Its one remaining miss needed two hops: find the free call in old.c, then find that function’s deprecation note. After seven turns it answered llama_free where the note says llama_model_free.

What it costs, and where it stops

Thinking is the first cost. This is how many tokens each run generated for the same ten JavaScript tasks:

The 26B wrote 15 times more tokens with thinking on, and tokens are time on a laptop:

The 12B took 21 minutes and 48 seconds to pass six tasks it had passed nine of in 2 minutes and 16 seconds.

So thinking isn’t a setting to switch off and forget. On E4B it cost time on code and bought correct answers nearly everywhere else:

The price: the eight SQL questions took 18.9 seconds without thinking and 119.4 with it. I’d leave thinking on for queries and lookups, and turn it off or cap it for code.

Speed is the second cost. Across every reply in a run, weighted by tokens, E4B generated 40 tokens per second, the 12B 18 to 21, and the 26B A4B 38 to 45.

The 26B A4B writes two and a half times faster than the 12B from a file twice the size. That’s the 3.8 billion active parameters at work: per token it does less arithmetic than the 12B does.

Single replies swing with the load on the machine. The 26B’s slowest reply ran at 26 tokens per second and its fastest at 82.

Every model reads faster than it writes, and how much faster depends on the model. This is the time each one took to process the 25,280-token header before its first answer:

The 12B needs two minutes and 18 seconds before it says a word about that file. The follow-up questions then came back in under 1.5 seconds each.

That’s the prompt cache. llama-server compares each prompt with the previous one and only evaluates the part it hasn’t seen, so my four follow-ups cost 25 to 33 new tokens each instead of 25,280.

The reading cost grows in a straight line with the prompt. I sent E4B five slices of the header, each as a fresh prompt:

That’s about 1,000 tokens per second from 778 tokens to 22,614, and generation held between 37 and 49 tokens per second behind every one of them. A long prompt costs you the wait up front and nothing after.

One server can also answer several requests at once. I restarted the 12B with four slots and sent the same question 1, 2 and 4 times in parallel:

Four requests together got twice the throughput of one. Each single request slowed from 21 tokens per second to between 12 and 15, so batch work gains and a chat window loses.

Memory is the third cost, and it decides which of these you can run at all.

footprint counts the memory the process owns. The weights are also mapped from disk, 6.8 GB for the 12B and 13.7 GB for the 26B, and macOS can drop those pages and read them back.

The footprint alone didn’t stop me. This laptop runs compilers and other agents all day, and out of 36 GB it had 6 to 11 GB to spare when I wanted the second pass.

E4B fit every time. The 12B and the 26B raised macOS memory pressure to the warning level within 90 seconds of loading, and another job’s watchdog started killing compiles, so I stopped them.

If the laptop is also your work machine, budget for that. A 16 GB Mac runs E4B next to an editor and a browser, and I wouldn’t plan on more.

The fourth wall is what the weights know. With no tools attached, I asked each model for the latest llama.cpp release and told it to say so if it didn’t know. The 12B, in an earlier run:

I do not know because llama.cpp has not officially released a version 1.0.0.

It said that while running on llama.cpp 1.0.0. In the scored runs all three declined to name a version, which is the right answer, and the fix is the tool call from the first battery.

I’d also planned to run gpt-oss-20b and Qwen3.8-27B through the same checks. Those are 12.1 GB and 19.4 GB files, and for the same memory reason neither one ran.

The last wall is size, and for that I’d read the 1.0.0 pull requests. llama.cpp makes every PR disclose LLM use, and Georgi Gerganov, who started the project, fills his in with the model and the hardware.

He has 17 PRs in this release, and 14 disclose an LLM. All 14 name an open model in llama.cpp’s own GGUF format:

One of them is #30565, “llama : prune”. That’s the PR that deleted the 36 functions I had Gemma port away from. This is its disclosure:

1
- AI usage disclosure: YES
2
  - Harness: `dsh:llama.cpp`
3
  - Hardware: `Mac Studio, M5 Ultra, 256GB`
4
  - Model: ggml-org/DeepSeek-V4.1-Flash-1017-GGUF
5
  - Stats: `◷ 1.8h · ⚙ 8.4m · ↑ 965k · ↓ 233k · ⇈ 399.3 tok/s · ⇊ 59.2 tok/s · ≡ 764 · ⌨ 51`

I read that stats line as a 1.8 hour session: 965k tokens in, 233k out, at 59.2 tokens per second. The harness is DeepSeek Harness, which I wrote about last week.

So the public API of llama.cpp 1.0 was trimmed by a local model running on llama.cpp. My laptop did the 15-line version of that job, and his desktop did it across the library.

That’s the map as I measured it. A 4.59 GB model handles tool calls, extraction and mechanical code changes, and needs thinking turned on before I’d trust its SQL or its lookups.

A 14.62 GB model writes ten working functions in under a minute. Hours of agent work on a C++ codebase run locally too, if you own 128 GB of memory.

I use Claude Code every day, and none of these three replaces it. But if I had a folder of screenshots or release notes to turn into JSON, I’d run it here before I paid per token.