This is a submission for the Gemma 4 Challenge: Write About Gemma 4

When choosing Gemma 4 models for your local system, knowing how much video RAM your GPU has dedicated might not be the only factor. GPU offloading, and reducing it while still having ample system RAM, might make larger models more accessible than previously considered.

1. Introduction

Why would anyone bother running a local AI model?

That is a fair question.

Claude, Gemini, ChatGPT, and other frontier systems are already extremely capable. They are easy to access, constantly improving, and in many cases they are better than anything most of us can run on a laptop. If the only question were raw intelligence, then local models would often lose.

But raw intelligence is not the only question.

There are several practical reasons someone might want a local model available.

First, some data is private, sensitive, experimental, or simply not something you want to send to a hosted service. That does not always mean the data is secret in a dramatic sense. Sometimes it is client code. Sometimes it is business planning. Sometimes it is personal financial data. Sometimes it is just a half-formed idea you are not ready to put anywhere else.

Second, local models can keep working when connectivity is unreliable. If you are commuting, working from a plane, traveling, or sitting in a cabin with power but no reliable signal, a cloud model is only as useful as the connection between you and it. A local model gives you another option. It may not replace a frontier model, but it lets you keep experimenting, drafting, coding, and learning when the network is unavailable or unstable.

Third, local models give you more control over tooling. If you want an assistant to read local files, call scripts, use a local development environment, or interact with private tooling, you may want tighter boundaries around what the model can access and what it is allowed to do. Running locally does not magically solve every security problem, but it changes the control surface. You can decide what gets exposed, where the model runs, and how it connects to your workflow.

Fourth, local models are getting better. Gemma 4 is part of a broader trend: models that are increasingly capable while becoming more practical to run on consumer hardware. Gaming laptops and desktop GPUs are no longer just for games, rendering, or CUDA experiments. They can now run useful language models locally. The catch is that not every model package is the same, and not every model that technically loads is pleasant to use.

That is where the confusion starts.

When you begin downloading local models, you quickly run into names and labels like GGUF, Q4, Q5, Q8, E2B, E4B, A4B, MoE, context length, GPU offload, KV cache, and tokens per second. Those terms matter. They are the difference between a model that runs smoothly, one that barely fits, and one that overwhelms your machine.

This article is about making those terms practical.

Instead of treating local AI as magic, I wanted to answer a concrete question: how do I prove whether a Gemma 4 model can run well on my own hardware?

To explore that, I tested several Gemma 4 model sizes using LM Studio on a consumer laptop with an NVIDIA RTX 5080 Laptop GPU and 16 GB of dedicated VRAM. Along the way, the real lesson was not just which model was fastest. The real lesson was learning how model size, quantization, VRAM, system RAM, context length, and GPU offload all interact.

The rest of this article walks through that process: the hardware, the terminology, the LM Studio setup, the benchmarking approach, and the practical lessons from trying to make these models useful in a real developer workflow.

2. The Test System

Before getting into the models themselves, it helps to define the machine I used for testing. Local AI performance is highly hardware-dependent, and two systems with the same amount of system RAM can behave very differently depending on the GPU, available VRAM, drivers, and offload behavior.

For this testing, I used a 2025 Lenovo Legion Pro 7 16IAX10H with the following configuration:

This is a powerful consumer laptop, but it is still not a datacenter machine. That distinction matters. A system like this can run useful local models, but it still has real constraints. The most important one for this article is not the 64 GB of system RAM. It is the roughly 16 GB of dedicated VRAM attached to the NVIDIA GPU.

That VRAM number determines a lot of what feels practical. If a model and its runtime memory fit comfortably inside GPU memory, responses can feel fast and interactive. If the model barely fits, leaves no room for the context cache, or spills too much work into system RAM, the experience can become sluggish even if the model technically loads.

For my actual workflow, that matters because I may not only be chatting with a model. I may also have a browser, an IDE, GIS tools, a spreadsheet, or other GPU-aware applications open. If the model consumes nearly all available VRAM, the whole machine can start to feel constrained.

How I Checked the System Specs

I used two approaches from PowerShell.

The first was the built-in PowerShell command:

Get-ComputerInfo

That command provides a very detailed system report, including Windows version, CPU, installed memory, BIOS information, uptime, and other operating system details. It is useful, but it is also extremely verbose.

The second tool was Fastfetch:

fastfetch

Fastfetch gave a much more readable summary. It showed the Windows version, machine model, CPU, GPUs, visible memory, swap, disk usage, displays, and shell version in a compact format. For this kind of article, that output is probably easier for readers to understand at a glance.

Fastfetch is available here:

The useful part of the Fastfetch output for this article was this kind of summary:

OS: Windows 11 Home (25H2) x86_64
Host: 83F5 (Legion Pro 7 16IAX10H)
Shell: PowerShell 7.6.1
CPU: Intel(R) Core(TM) Ultra 9 275HX (24) @ 5.40 GHz
GPU 1: NVIDIA GeForce RTX 5080 Laptop GPU @ 3.09 GHz (15.63 GiB) [Discrete]
GPU 2: Intel(R) Graphics (128.00 MiB) [Integrated]
Memory: 38.15 GiB / 63.43 GiB (60%)
Swap: 1.58 GiB / 27.00 GiB (6%)
Disk (C:): 824.06 GiB / 951.65 GiB (87%) - NTFS

This tells us several important things before we ever load a model:

• The machine has both a discrete NVIDIA GPU and integrated Intel graphics.
• The NVIDIA GPU has about 16 GB of dedicated VRAM.
• The machine has 64 GB of system RAM, but that is separate from dedicated VRAM.
• The system already has other memory and GPU usage before LM Studio enters the picture.
• Storage is large enough to download several model files, but large models can still consume disk space quickly.

If you do not want to install Fastfetch, Windows can still show most of what you need.

Open Settings > System > About to find the Windows version, processor, installed RAM, and basic device information.

Then open Task Manager > Performance and click each GPU. This is where Windows shows whether a GPU has dedicated memory, shared memory, or both. For local AI, that distinction is important. A discrete NVIDIA GPU with dedicated VRAM is very different from an integrated GPU that borrows shared memory from system RAM.

Task Manager is also useful for understanding the current state of the machine before benchmarking. It can show whether the system is relatively clean, whether a lot of background software is already running, or whether memory and GPU resources appear to be tied up by applications that were recently closed but have not fully released their resources yet.

For casual experimentation, that may not matter much. For real benchmarking, it matters a lot. I found it better to start from a fresh restart and open only the tools needed for the test: LM Studio, the model being tested, Task Manager or another monitoring tool, and whatever notes or prompts are required. That gives a cleaner baseline and makes it easier to see what the model itself is doing to the system.

One other practical caveat: different local AI tools may behave differently on the same hardware.

For example, Ollama was easy for me to install and start using, but I found it harder to get it to take full advantage of the NVIDIA GPU and dedicated VRAM on this particular laptop. LM Studio, by contrast, made that much easier in my testing. It recognized and used the NVIDIA GPU in a way that felt practical almost out of the box.

That does not mean LM Studio will always be better for every machine, or that Ollama cannot be configured correctly. It means local AI performance depends on the full stack: hardware, drivers, runtime, GPU detection, model format, and the application you use to load the model. Your results may vary, which is another reason it helps to inspect your system first and then benchmark rather than assume.

This system information step may seem basic, but it is one of the first practical checkpoints in local AI benchmarking. Before asking “which model should I download?” it helps to ask “what resources do I actually have available?”

In this case, the answer was: a strong CPU, plenty of system RAM, a fast SSD, and a discrete NVIDIA laptop GPU with about 16 GB of VRAM. That is enough to make local AI interesting, but not enough to ignore memory limits.

That leads to the next major issue: why 64 GB of RAM does not mean the same thing as 16 GB of VRAM.

3. A Word About the Acronyms

Once you know what hardware you have, the next challenge is figuring out what you are actually downloading.

Local model names can look like alphabet soup: GGUF, Q4, Q5, Q8, E2B, E4B, A4B, MoE, Instruct, context length, GPU offload, KV cache, and tokens per second. Those labels are not decoration. They are clues about how the model was built, how much memory it may need, what kind of hardware it expects, and what kind of performance you can reasonably hope for.

A local model file name is often a compressed technical summary. It may tell you the model family, the model generation, the parameter count, whether it was tuned for chat, what format the file uses, and how aggressively it was compressed.

For example, a model name may communicate several ideas at once:

Gemma-4-Example-Instruct-Q4-GGUF

A simplified way to read that kind of name would be:

Not every model name follows the exact same pattern, but the principle is the same. The name is trying to tell you what kind of model it is and how it was packaged.

The most important terms for this article are:

• Parameters: The numeric weights inside the model. When people say 2B, 4B, 9B, or 27B, the B usually means billions of parameters.
• Dense model: A model where most or all of the model participates in each prediction.
• MoE: Mixture of Experts. These models contain multiple expert subnetworks, but only some are active for a given token.
• Active parameters: The portion of the model actively used during a prediction, especially relevant for MoE models.
• Instruct: A version tuned to respond better to instructions, chat prompts, or assistant-style use.
• GGUF: A file format commonly used for local inference.
• Quantization: A way of storing the model’s numbers with less precision to reduce memory usage.
• Context window: How much text the model can consider at once.
• KV cache: Runtime memory used to remember the context during generation.
• GPU offload: Moving some or all of the model’s work from the CPU to the GPU.
• Tokens per second: A rough speed measurement for how quickly the model generates output.

The reason these terms matter is that model names are not just marketing labels. They influence whether a model fits in VRAM, how much system RAM might be needed, how quickly the model responds, and whether the experience feels smooth enough to use.

This is also why two downloads from the same model family can behave very differently. One package might be small and fast because it is heavily quantized. Another might be larger and higher quality but require much more VRAM. Another might technically load but become unpleasant once the context window grows.

For the rest of this article, the two terms that matter most are quantization and memory. Quantization changes how much memory the model needs. Memory determines whether the model can run smoothly on the hardware you actually have.

4. Quantization as a Precision Tradeoff

Quantization is one of the main reasons local AI is practical on consumer hardware.

At a simple level, quantization means storing the model’s internal numbers with less precision. Since a language model is made up of billions of numeric weights, the way those numbers are stored has an enormous effect on memory usage.

A full-precision or higher-precision model stores those numbers with more detail. That can preserve quality, but it also uses much more memory. A quantized model stores those numbers with fewer bits. That usually makes the model smaller, easier to load into VRAM, and often faster to run.

The tradeoff is precision.

This is a mathematical compromise. If the original model uses high-precision numbers, quantization converts those numbers into lower-precision approximations. In many cases, the model still works surprisingly well because the approximation is good enough for useful language generation. But if quantization is too aggressive, quality can degrade. The model may become less nuanced, less reliable, or more likely to make mistakes.

A useful analogy is image compression.

A high-resolution image preserves more detail but uses more storage. A compressed image is smaller and easier to share, and it may still look good. But if you compress it too much, artifacts appear and detail is lost.

Quantization has a similar feel for models:

Higher precision model  -> larger file, more memory, potentially better quality
Quantized model         -> smaller file, less memory, often faster
Over-quantized model    -> smallest file, but quality may suffer

That is why local model files often include labels like Q8, Q6, Q5, Q4, or Q3.

A rough practical interpretation is:

For a consumer GPU, Q4 and Q5 are often where experimentation becomes practical. They can make a model small enough to fit into available VRAM while preserving enough quality to be useful. Q8 may be better when memory allows. Lower quantization may help a model load on constrained hardware, but they may also reduce answer quality.

This matters directly for a machine with 16 GB of VRAM. A model that would be impossible or impractical at high precision might become usable as a Q4 or Q5 model. But quantization does not remove every limitation. The model still needs room for runtime memory, the context cache, GPU overhead, and any other applications using the GPU.

So quantization is not simply “smaller is better.” It is a precision, memory, and quality tradeoff.

For benchmarking, that means the question is not only “which model did I run?” It is also “which version of the model did I run?” A Gemma 4 model at Q4 and the same family at Q8 may behave very differently on the same machine.

For this specific benchmark, every Gemma 4 model I tested was a 4-bit quantized package. I did not compare Q3, Q5, Q6, Q8, or higher-precision variants. That matters because the results are really about these specific Q4 packages, not every possible version of Gemma 4.

5. RAM vs VRAM

One of the first mistakes people can make when thinking about local AI is focusing only on system RAM.

At first glance, that seems reasonable. If a laptop has 64 GB of RAM, it feels like it should be able to run a very large model. And system RAM does matter. But for GPU-accelerated local AI, the more important number is often VRAM: the dedicated memory attached directly to the graphics card.

System RAM and VRAM are both memory, but they are not the same kind of memory and they are not used in the same way.

System RAM is the large working memory available to the CPU and the operating system. Windows, browsers, IDEs, background services, datasets, and normal applications all use system RAM. Having 64 GB gives the machine a lot of breathing room, especially compared to a smaller laptop with 16 GB or 32 GB.

VRAM is the high-bandwidth working memory attached directly to the GPU. For local AI, this is where you ideally want the model weights, active computation, runtime buffers, and context-related memory to live. The GPU can perform the tensor math much faster when the data it needs is already in VRAM.

That is why a 16 GB NVIDIA GPU can matter more than 64 GB of system RAM for model performance.

If the model fits comfortably in VRAM, generation can feel responsive. If the model does not fit, or if it barely fits, the software may have to split work between the GPU and CPU, move data through system RAM, or rely more heavily on slower memory paths. The model may still run, but the experience can become much slower.

This is where the phrase “can load” can be misleading.

A model that completes loading is not necessarily a model that is pleasant to use. It might respond slowly, pause frequently, consume all available GPU memory, or become unstable as the context grows. For benchmarking, the better question is not only “did it start?” but “does it run well enough to be useful?”

Total VRAM Is Not Available VRAM

Another practical issue is that total VRAM is not the same as available VRAM.

Takeaway:

A better local model is often one that leaves enough room for the operating system, LM Studio, the context cache, and any other applications you need to keep open.

On paper, my RTX 5080 Laptop GPU has about 16 GB of dedicated VRAM. In practice, some of that memory may already be used before a model is loaded. Windows desktop rendering, external displays, browsers, video tools, IDEs, games, CAD software, geospatial tools, and other GPU-accelerated applications can all consume VRAM.

That means a model that appears to fit inside 16 GB may still be a poor choice if it leaves no headroom. A better local model is often one that leaves enough room for the operating system, LM Studio, the context cache, and any other applications you need to keep open.

For my use case, that matters because I may not only be chatting with a model. I may also have development tools, browser tabs, GIS software, or other GPU-aware applications running. If the model consumes nearly all available VRAM, the whole machine can start to feel constrained.

The Hidden Cost of Context

The model file is not the only thing that uses memory.

As a model processes a prompt and generates output, it maintains internal state about the context. This is commonly discussed as the KV cache, or key/value cache. The exact details are more technical than this article needs to cover, but the practical point is simple: longer context windows require more runtime memory.

That means a model can appear to fit when first loaded, but become less comfortable as the prompt grows, the conversation gets longer, or the configured context length increases. A short test prompt may work fine. A long coding session with pasted files, logs, or large explanations may apply very different memory pressure.

This is one reason benchmarking should include more than a single short prompt. If the goal is to use a local model for real work, it helps to test the kinds of prompts and context sizes you actually expect to use.

What Happens When Memory Spills Over

When the system is under high memory pressure, Windows has its own memory management behavior. It maintains page files, caches data, and may write virtual memory to disk when physical RAM is under pressure.

That can keep the system from crashing, but it does not make disk behave like RAM, and it definitely does not make disk behave like VRAM. If memory spills to disk, storage speed starts to matter. A fast NVMe SSD will behave very differently from an older SATA drive or a slow external disk. Thunderbolt storage may be faster than many older external options, but it still changes the performance profile compared with memory that is directly available to the CPU or GPU.

For local AI, this can affect how quickly the system retrieves and uses parts of the data involved in a session, especially if the machine is already under heavy load. The model may technically continue operating, but responsiveness can drop sharply.

This is why a clean benchmark matters. A fresh restart, minimal background applications, and watching Task Manager can help distinguish the model’s real behavior from the behavior of an already overloaded system.

Practical Mental Model
A useful way to think about it is this:

• VRAM -> the GPU’s fast workbench
• System RAM -> the computer’s larger general workspace
• Page file -> overflow storage when RAM is under pressure
• Disk -> much slower storage, useful but not a replacement for memory

For local AI, the ideal situation is to keep as much of the active model work as possible on the GPU, with enough VRAM left over for runtime overhead and context. System RAM is helpful, especially for partial offload or preventing crashes, but it is not a substitute for dedicated GPU memory.

This is why quantization and VRAM are connected. Quantization can shrink the model enough to fit better in VRAM. Fitting better in VRAM can make the model feel dramatically more responsive. But if the model is still too large, or if the context cache grows too much, the system may fall back to slower paths.

So the practical rule is simple:

Do not ask only whether your machine has enough total memory. Ask whether the model, quantization, context size, and workload fit comfortably inside the memory that matters most for the way you are running it.

6. Gauging Model Size

After understanding RAM, VRAM, and quantization, the next practical question is: how do you gauge the size of a model before downloading it?

The model name is usually the first clue.

For this article, I looked at four Gemma 4 variants:

The obvious number to look at is the B value. In model names, B usually means billions of parameters. A 2B model has roughly two billion parameters. A 4B model has roughly four billion. A 31B model is much larger.

That alone gives you a rough sense of scale, but it is not the whole story.

Total Parameters vs Active Parameters

One of the most important distinctions is total parameters versus active parameters.

A dense model generally uses most or all of its parameters during each prediction. If the model is 31B, you should think of it as a large model that places heavy pressure on memory and compute.

A Mixture-of-Experts model can be different. In an MoE model, the total model may contain many parameters, but only a subset of those parameters are active for a given token. That is where a name like 26B A4B becomes important.

In that label, I read 26B as the approximate total parameter count and A4B as roughly four billion active parameters during inference. In practical terms, that means the model may have a larger overall capacity than a small dense model, while behaving more efficiently per token because only part of the model is active at a time.

That does not make it free. The total model still has to be stored and managed, and runtime behavior still depends on quantization, context length, offload settings, and available VRAM. But it helps explain why a model with a large total parameter count may not behave exactly like a dense model of the same size.

What Does E Mean?

The E in names like E2B and E4B appears to describe an effective size class. For this article, I treated Gemma 4 E2B as an effective 2B-class model and Gemma 4 E4B as an effective 4B-class model.

The practical interpretation is simple: E2B should be the lighter model, and E4B should require more memory and compute but may provide better results. These are the kinds of models that are most likely to feel fast and responsive on consumer hardware.

Because model naming conventions can vary across publishers and packages, I would not treat the letter alone as the full specification. The safer habit is to read the whole model card or package description when available. But as a practical benchmarker, the name still gives you useful hints.

What Does A Mean?

The A in A4B is best understood as active parameters.

That is especially relevant when discussing MoE models. A model labeled 26B A4B suggests a larger total model with about 4B parameters active during inference. That matters because memory and compute behavior are not always the same as a dense 26B model.

For readers new to this distinction, the important point is:

Total parameters tell you how large the model is overall.
Active parameters tell you how much of the model is used for a given prediction.

Both matter.

Total parameters affect model storage, download size, memory pressure, and how much has to be managed. Active parameters affect how much computation may be involved per token. Quantization then changes how much memory those parameters require.

So model size is not a single number. It is a combination of:

• Total parameters
• Active parameters
• Dense vs MoE architecture
• Quantization level
• Context length
• Runtime overhead
• GPU offload behavior

A Practical Way to Think About the Four Models

For testing purposes, I treated the models like a ladder.

Gemma 4 E2B was the lightweight baseline. If this model did not run well, something was probably wrong with the setup or hardware configuration.

Gemma 4 E4B was the practical small-model step up. It should still be light enough to run comfortably while giving a better sense of whether the model family is useful for coding, explanation, and structured responses.

Gemma 4 26B A4B was the interesting middle case. It has a much larger total parameter count, but the active-parameter label suggests MoE-style efficiency. This makes it a useful test of how far a consumer GPU can go before memory and runtime behavior become limiting.

Gemma 4 31B was the stress test. A model in that size class is where the difference between “it loads” and “it is practical” becomes very important. Quantization, context length, and available VRAM matter much more at this scale.

A useful rule of thumb is to start small and work upward.

Do not begin with the largest model just because it looks most capable. Start with a smaller model to confirm that the runtime is using the correct GPU, that tokens are generating at a reasonable speed, and that memory usage looks sane. Then move up one size at a time.

That approach gives you a much better picture of where your system’s practical limit actually is.

One More Clarification: Size Can Mean Several Things

It is also worth separating a few different meanings of “size.”

When people talk about a 2B, 4B, 26B, or 31B model, they are usually talking about parameter count. That gives you a rough sense of the model’s scale and potential compute requirements.

But that is not the same as the size of the downloaded file on disk. The persisted model package may be smaller or larger depending on quantization, file format, and how the model was packaged. This is the size you see when LM Studio downloads the model to storage.

There is also the loaded or runtime size. Once the model is loaded, it may require additional memory beyond the model file itself. Runtime memory can include buffers, GPU overhead, the KV cache, and anything needed by the inference engine. This is the memory footprint that matters when you are watching Task Manager and asking whether the model actually fits comfortably on your system.

So there are at least three sizes to keep separate:

Parameter count       -> how large the model is conceptually
Downloaded file size  -> how much storage the model package uses on disk
Runtime memory size   -> how much RAM/VRAM the model needs while running

This distinction matters because a model can look manageable as a download but still be uncomfortable in memory. Or it may technically fit in memory, but leave too little room for the context cache and other applications.

Model Size Is Only the First Filter

The final caution is that model size is only the first filter.

A 31B model at one quantization level may be completely impractical on a 16 GB GPU, while another heavily quantized package may load but run slowly. A smaller model at higher precision may use more memory than expected. A model that works with a short context may become painful with a longer one.

So the practical question is not:

How big is the model?

The better question is:

How big is this specific package of the model, at this quantization, with this context size, on this machine, using this runtime?

That is why benchmarking matters. Model names give you clues, but testing tells you what your system can actually do.

7. Setting Up LM Studio for Local Model Testing

With the hardware, model naming, quantization, and memory concepts in place, the next step is actually loading models and observing how they behave.

For this article, I used LM Studio as the practical testing environment. I chose it because it gave me a friendly way to search for models, inspect packages, download quantized versions, load them into chat, and observe runtime behavior without first writing my own inference script.

On my machine, LM Studio also made it easier to use the NVIDIA GPU. Ollama was easy to install and start using, but I found it harder to get it to take full advantage of the dedicated NVIDIA VRAM on this laptop. LM Studio recognized and used the RTX 5080 Laptop GPU in a way that felt much more practical for this experiment.

That is not a universal claim that LM Studio is always better. It is a practical observation from this system. Different machines, drivers, operating systems, and model runtimes can behave differently. That is why this article focuses on testing and observing rather than assuming.

Start from the Chat Screen

After opening LM Studio, the first screen I used was the chat workspace. In a new chat, the top bar shows an option to select a model to load. Until a model is selected, the chat window is just a workspace. It does not tell us anything yet about model speed, memory use, or usability.

This is the first useful checkpoint: before testing anything, confirm that no model is already loaded. Starting from an unloaded chat makes it easier to observe what changes when a model is loaded.

Search for Gemma 4

From there, I moved to the model search and discovery area and searched for “gemma 4.” This is where LM Studio starts to expose the information needed to make a reasonable choice.

The search results may include several kinds of entries:

• Official or staff-picked model family entries
• Community-published GGUF packages
• Different Gemma 4 sizes
• Older Gemma 3 models
• Modified variants
• Different quantizations and package sizes

That last point is important. Search results are not all interchangeable. Two results may both say Gemma 4, but they may differ in publisher, quantization, format, capabilities, download size, or intended use.

In the Gemma 4 search results, the models I focused on were:

• Gemma 4 E2B
• Gemma 4 E4B
• Gemma 4 26B A4B
• Gemma 4 31B

The search screen is useful because selecting a model shows a detail panel with more information. For example, the Gemma 4 31B entry showed the model family, parameter count, architecture, format, capabilities, download option, and README-style description.

Read the Model Detail Panel

Before downloading or loading a model, I used the detail panel to answer a few questions:

• What model family is this?
• How many parameters does it claim?
• Is it dense or MoE-style?
• Is it an instruct/chat model?
• What format is available?
• What quantization or package option is available?
• How large is the download?
• What capabilities are listed?
• Does the README mention context length, reasoning, vision, tool use, or coding?

This step matters because the model name alone is not the whole story. The detail panel helps connect the name to the actual package being downloaded.

For example, a 31B model entry may show a package size of about 19.89 GB. That tells me the approximate model package size on disk. It does not tell me the full runtime memory footprint after loading, but it does tell me whether I have enough storage and gives me a clue about how large the model package is compared with smaller variants.

Move to the Local Model List

After downloading models, LM Studio’s local model list becomes especially useful.

In the local model view, LM Studio shows the downloaded models with columns such as architecture, parameter count, publisher, model name, quantization, disk size, and modified date. Selecting a local model also shows additional details in the side panel, including the file name, format, quantization, architecture, capabilities, and size on disk.

This view is valuable because it brings together many of the concepts from the previous sections:

This is also where the earlier clarification about “size” becomes practical. The Size column shows the model package on disk. It does not directly show how much VRAM or RAM the model will use while running. To learn that, the model has to be loaded and monitored.

Load One Model at a Time

For benchmarking, I would not load models randomly. I would start small and move upward. I would also start from a fresh boot or clean login, allow the system to settle, and keep the same core tools open for each run. That way, I can better see how the model affects the GPU, RAM, and CPU.

The practical order is:

Gemma 4 E2B
Gemma 4 E4B
Gemma 4 26B A4B
Gemma 4 31B

Starting with the smallest model helps confirm that the basic setup works. If the E2B model does not load or does not use the expected GPU, then the problem is probably not model size. It may be a configuration, driver, runtime, or GPU-selection issue.

Once the smallest model works, moving upward gives a clearer picture of where the system starts to feel constrained.

When loading each model, I want to record:

• Did the model load successfully?
• How long did loading take?
• Did LM Studio remain responsive?
• Did dedicated GPU memory increase?
• Did system RAM increase?
• Did the model use the NVIDIA GPU rather than integrated graphics?
• Was there still VRAM headroom after loading?

Note:

the first few times I benchmarked the smaller models, they loaded so quickly that I realized I needed to use a stopwatch to capture useful load times.

Confirm GPU Usage

This is one of the most important checks.

A model may run locally but not use the GPU you expect. On a hybrid laptop with both integrated Intel graphics and a discrete NVIDIA GPU, it is worth verifying where the work is going.

I used Task Manager > Performance to watch GPU activity and dedicated GPU memory. LM Studio may also show useful runtime information, but Task Manager provides an independent Windows-level view of GPU and memory behavior.

The important thing is not just that the model runs. The important thing is whether it runs using the hardware path you intended to test.

Note: I have a powershell script that auto loads LM Studio and chooses a Gemma 4 2B or 4B model right after login is successful, for this test, I had to unload that model, before doing any real capture, to make the benchmarks as similar as possible.

After unloading the model (by ejecting it from the Model listing or from a chat window) you should see dedicated GPU memory return to a stable normal as in this picture. If after ejecting, it still seems to be taking up valuable video memory, you may need a fresh reboot to ensure a model is not running.

Once you’ve reached a state where no model is loaded you should see a fairly low count on your dedicated GPU memory. Some machines may have an AI Boost component like this Intel(R) AI Boost NPU, be sure to check that you are looking at your dedicated graphics GPU when benchmarking .

Adjust Settings Carefully

LM Studio can expose runtime settings such as context length, offload behavior, and other advanced options depending on the model and version. These settings matter, but they should be changed carefully.

For benchmarking, I prefer to change one variable at a time. If I change model size, quantization, context length, and offload behavior all at once, then I cannot tell which change caused the difference.

A better approach is:

• Pick one model package.
• Load it with reasonable defaults.
• Record baseline behavior.
• Change one setting if needed.
• Record what changed.

That keeps the benchmark useful instead of turning it into guesswork.

A few of the settings that can affect results include:

• Context length: Larger context windows require more memory, especially because of the KV cache.
• GPU offload: Controls how much of the model work is moved to the GPU instead of staying on the CPU.
• CPU thread pool size: Affects how much CPU parallelism the runtime may use.
• Evaluation batch size: Can affect throughput and memory behavior during generation.

In this experiment, the main setting I modified was GPU offload. For a smaller set of tests, I may also have increased the context length, possibly doubling the default context window. I did not rely on evaluation batch size as a primary tuning variable for these results.

One important detail: in LM Studio, GPU Offload is not a percentage. A value such as 19 means that 19 model layers are being offloaded to the GPU. The remaining layers continue to run through the CPU/system RAM path.

That makes GPU offload one of the most important tuning controls for larger models. More layers on the GPU can improve speed, but only if there is enough VRAM left for the model, runtime overhead, and KV cache. If too many layers are offloaded, the model may fail to load, consume nearly all VRAM, or slow down because there is not enough memory headroom left for context and cache.

The practical rule is to increase GPU offload until performance stops improving, VRAM headroom becomes too tight, or the model becomes unstable. For final benchmark numbers, I would want to record the exact GPU offload value, context length, whether KV cache was offloaded to GPU memory, and the observed VRAM usage for each run.

The key lesson is that these settings can change the result. If one model is tested with a different context length, offload level, or batch size than another, the comparison may not be fair. For a clean benchmark, the settings should either remain consistent or be recorded clearly when they differ.

8. Benchmark Methodology

For this article, I treated “usable” as more important than “technically loads.”

A model is useful if it:

• Loads reliably
• Uses the intended GPU
• Leaves some memory headroom
• Responds at a usable speed
• Handles the prompt types I actually care about
• Does not make the rest of the system unusable

A model is less useful if it:

• Barely loads
• Consumes nearly all VRAM
• Runs painfully slowly
• Becomes unstable as context grows
• Requires closing everything else on the machine just to function

That distinction matters because the goal is not simply to prove that a large model can start. The goal is to find models that are practical for real work.

Benchmark Test Categories

The original test plan included several categories:

• Load test: Does the model load without crashing?
• Memory test: How much VRAM does it consume idle and during generation?
• Speed test: How quickly does it respond? How many tokens/sec does LM Studio report when available?
• Coding test: Can it refactor or explain TypeScript accurately?
• Reasoning test: Can it explain tradeoffs and compare approaches?
• Structured output test: Can it return tables, lists, and code blocks cleanly?
• Long-context test: Does performance degrade as context grows?

These categories mattered because no single prompt tells the whole story. A tiny model may feel excellent for quick coding help but weaker for deep reasoning. A larger model may produce richer answers but become too slow to use interactively. Unfortunately, the number of types of prompts for a given model are large, and I did not test image/vision parsing or generation in these benchmarks. So be sure to curate your prompt tests in accordance with the general type of work you will be applying for this model.

Benchmark Prompt Selection

A benchmark prompt should not be too trivial. If the prompt only asks for a short answer, the model may finish before there is enough time to observe GPU utilization, VRAM behavior, CPU usage, or throughput. On the other hand, if the prompt is too open-ended or inconsistent, the results may be hard to compare between runs.

For the main sustained-generation test, I used a structured CAP theorem prompt:

Explain how distributed systems handle consistency, availability, and partition tolerance (CAP theorem).

Then:
1. Compare strong consistency vs eventual consistency with real-world examples
2. Describe how databases like Cassandra and PostgreSQL handle tradeoffs differently
3. Provide a step-by-step scenario of a network partition and how each system responds
4. Summarize tradeoffs in a comparison table

Be detailed and structured. Aim for ~600-800 words.

This prompt works well because it forces sustained generation. It asks for explanation, comparison, scenario reasoning, and structured output. It also gives the model a target length, which makes repeated runs easier to compare.

While the model runs, the useful things to watch are:

The most important benchmarking rule is to use the same exact prompt every time. Otherwise, I may be benchmarking prompt differences instead of model/runtime differences. Even with identical prompts, models have some variation in their outputs and are not deterministic, so for us accuracy or correctness is not a comparison 1 for 1 character by character for the output returned, but in whether the model successfully replies to the prompt with sufficient detail and thought to be useful.

For smaller models, I also tested rapid iteration and practical coding prompts, such as a TypeScript refactor, JavaScript closures, a React follow-up, and a REST API explanation. This mattered because smaller models are not always meant to win deep reasoning tests. Sometimes their best use case is fast iteration.

Other Benchmark Prompts Used

The CAP theorem prompt was the main sustained-generation benchmark, but it was not the only prompt used. I also used shorter prompts to test coding quality, conversational speed, context retention, and general explanation throughput.

The TypeScript refactor prompt was:

You are helping me refactor code.

Here is a TypeScript function:

function processData(data: any[]) {
  let result = [];
  for (let i = 0; i < data.length; i++) {
    if (data[i].active === true) {
      result.push(data[i].value * 2);
    }
  }
  return result;
}

1. Refactor this to be more functional and readable
2. Add type safety
3. Explain your changes briefly

The JavaScript closures prompt was:

Explain closures in JavaScript in simple terms, then give 2 examples.

The React follow-up prompt was:

Now show me a real-world use case in React.

The REST API speed prompt was:

Write a detailed explanation of how a REST API works, including request lifecycle, headers, and status codes.

The optional database index prompt was:

Write a detailed explanation of how a database index works, including B-trees, hashing, and query optimization. Provide examples.

Fresh Restart Baseline

Before doing the real tuning pass, I restarted Windows and opened only the tools needed for the test. That gave me a cleaner baseline before loading any model.

For the formal notes, I did not try to create an artificial bare-metal lab environment. Each test began after a restart, once normal startup apps had settled. LM Studio, Task Manager, Snipping Tool, Notepad++ for observations, and Excel for logging were open. Normal background tools such as Bitdefender, JetBrains Toolbox, WebView2, password manager, Steam client, Edge background processes, and other usual services were still present. That is intentional. This was a practical workstation benchmark, not a synthetic lab benchmark.

In that state, the system was generally idle: low CPU usage, no meaningful GPU activity, and about 0.1 to 1.1 GB of GPU memory used before a model was loaded.

That baseline matters. Even before loading a model, the machine is not starting from zero VRAM usage. Windows, displays, LM Studio, CUDA initialization, and other GPU reservations can already consume memory. Total VRAM is not the same thing as available VRAM.

Benchmark Caveats

This was a practical benchmark, not a lab-grade controlled benchmark. Some runs used text notes rather than full screenshots. Some measurements were approximate. Not every run captured tokens/sec. Some prompts were repeated after previous model runs, so runtime caching, warmed filesystem cache, LM Studio state, or other reuse effects may have influenced later runs. I did not verify whether any thinking-mode behavior attempted external searching or benefited from cached information.

That means these results should not be read as universal scores for the models. They describe how these specific model packages behaved on this laptop, using this version of LM Studio, with these runtime settings, under a practical workstation baseline.

9. Results and Observations

After working through the setup and tuning process, the benchmark became more interesting than I expected.

At first, the story looked simple: E2B and E4B were practical, while the larger 26B and 31B models looked too large for interactive work on a 16 GB VRAM laptop.

But after retuning GPU offload, the story changed. The larger models were not simply “too big.” They were painful when over-offloaded to the GPU. Reducing GPU offload left more VRAM headroom, shifted more work to CPU and system RAM, and dramatically improved responsiveness.

Summary Table

Gemma 4 E2B: The Lightweight Baseline

E2B was the fastest and lightest model in this test set. It loaded in only a few seconds and settled around 3.9 GB of dedicated VRAM after load. That left plenty of headroom on a 16 GB GPU.

For quick prompts, it felt genuinely responsive. The TypeScript refactor, JavaScript closure explanation, React follow-up, REST API explanation, and CAP theorem benchmark all completed quickly enough to feel interactive.

Representative times included:

The main caveat was quality. E2B was fast and useful, but not perfect. In one closure example, it produced a questionable JavaScript snippet using let name = name;, which is the kind of mistake I would want to catch before trusting the output. That makes E2B useful for quick local assistance, but not necessarily the model I would trust most for careful code review.

My practical read: E2B is a great sanity check and a very fast fallback model. It proves the local setup is working and is useful when speed matters more than depth.

Gemma 4 E4B: The Best Practical Balance

E4B was slower than E2B, but it produced stronger, cleaner answers. It used more VRAM, settling around 5.4 GB after load, but that is still well within the available GPU budget on this machine.

The E4B model also ran with a larger context length: 8192 instead of E2B’s 4096. That is important because the comparison is not perfectly apples-to-apples. Even so, E4B still felt practical.

Representative times included:

The TypeScript refactor answer was more polished than E2B’s. The closure explanation avoided the obvious bug I saw in the E2B example. The REST API and CAP theorem outputs were also more structured.

My practical read: E4B is the best daily-driver candidate so far. It is not as instant as E2B, but it still feels responsive, leaves plenty of VRAM headroom, and produces better output.

Gemma 4 26B A4B: Initial Over-Offload Result

The 26B A4B model was the first major reality check.

On paper, it is tempting to assume the larger model will be better. In practice, the first configuration was not better for interactive work.

The model loaded, but loading took around 42 seconds. After load, it consumed roughly 15.5 GB of dedicated VRAM. That left almost no practical headroom on a 16 GB GPU.

The first TypeScript refactor run accidentally had thinking mode enabled. That alone took 7 minutes and 44 seconds before the full output completed at about 15 minutes and 7 seconds. Turning thinking off helped, but not enough: the same type of refactor still took about 7 minutes and 32 seconds at roughly 1.03 tokens per second.

The CAP theorem benchmark was even worse for interactive use, taking about 32 minutes and 20 seconds.

This was the clearest example of the difference between “it loads” and “it is useful.” The 26B model did run, but it consumed nearly the entire VRAM budget and was not competitive with E4B in that configuration.

Gemma 4 26B A4B: Retuned Result

One of the most surprising results from the benchmark came after reducing GPU offload from 24 layers to 16 layers on the 26B A4B model.

More GPU offload was not automatically better

At first, I expected lower GPU offload to make the model slower. Instead, the opposite happened.

With GPU offload reduced to 16 layers, dedicated VRAM usage dropped from roughly 15.5 GB to about 12.2 GB. Total GPU memory stabilized around 12.4 GB while system RAM rose substantially into the 35-38 GB range. CPU utilization increased into the 40-44% range.

Most importantly, the model became dramatically more responsive.

Representative timings after retuning included:

This was shocking compared with the earlier GPU-heavy configuration where some prompts took many minutes.

The practical interpretation is that the earlier configuration was likely over-offloaded to the GPU. The model technically fit, but it left too little VRAM headroom for efficient runtime behavior, KV cache growth, and the rest of the inference pipeline.

By reducing GPU offload, the system allowed more work to flow through CPU and system RAM instead of trying to force nearly everything into constrained GPU memory. Even though CPU utilization increased significantly, the overall runtime improved.

In other words, more GPU offload was not automatically better.

This became one of the most important lessons of the benchmark. A balanced workload between GPU VRAM and system RAM can outperform an over-constrained all-GPU configuration, especially on a 16 GB laptop GPU where memory headroom matters.

Another interesting observation was memory recovery behavior after generation completed. System RAM usage appeared to fall gradually after prompts finished, suggesting that portions of the runtime allocation, cache, or working memory were being reclaimed over time rather than instantly released.

Gemma 4 31B: Initial Stress Test

I also tried loading the Gemma 4 31B model at what appeared to be its default runtime settings: context length 4096, GPU offload 35, CPU thread pool size 9, evaluation batch size 512, and max concurrent predictions 4.

At those settings, the model loaded in about 56.83 seconds. After load, system RAM was around 38.9 GB, dedicated GPU memory was around 15.4 GB, and total GPU memory was around 15.7 GB.

That memory profile looked very similar to the earlier over-offloaded 26B case. The model loaded, but it left very little VRAM headroom.

I started a JavaScript closures prompt with thinking off. It got through the initial thinking phase at around 40 seconds, but the actual response generation was extremely slow. Since the goal is practical usability rather than proving that a huge model can technically grind through a prompt, I stopped the run around the two-minute mark and treated the default 31B setting as an over-offload stress test.

Gemma 4 31B: Retuned Result

After the first 31B attempt at 35 GPU-offloaded layers proved too slow, I reduced GPU offload to 24 layers while keeping context length at 4096, CPU thread pool size at 9, evaluation batch size at 512, and max concurrent predictions at 4.

That changed the behavior dramatically.

At 24 GPU-offloaded layers, the 31B model loaded in about 12.98 seconds. Dedicated GPU memory dropped to about 11.6 GB, with total GPU memory around 11.8 GB. At rest, CPU usage was around 4 percent and system RAM was around 39.4 GB.

During the TypeScript refactor test, system RAM rose to about 40.5 GB, CPU usage reached about 44 percent, dedicated VRAM stayed around 11.8 GB, and total GPU memory was around 12.0 GB. The TypeScript refactor completed in about 1 minute 17.69 seconds, using about 13 percent of the context window.

The CAP theorem benchmark took about 6 minutes 1.13 seconds. LM Studio reported about 4.61 tokens per second, 1699 tokens used, and about 1.99 seconds before the EOS token was found. During that run, CPU usage rose to about 42 percent, system RAM was around 35.2 GB, dedicated VRAM was around 11.7 GB, and total GPU memory was around 11.9 GB.

This was still slower than the retuned 26B model, but it was no longer unusable. Reducing GPU offload again changed the model from a memory-constrained crawl into a model that could complete the benchmark.

One possible additional factor was thermals. During the 31B CAP run, GPU temperature appeared to move between roughly 71 and 78 degrees Celsius. At the higher end of that temperature range, the CPU clock appeared to fall from around 4.77 GHz toward roughly 4.15-4.2 GHz as temperatures came back down. That raises the possibility that heat or power management may have influenced sustained performance. I was not tracking thermals rigorously enough to prove throttling, so this remains an observation rather than a conclusion.

Another useful observation: GPU utilization percentage did not appear to rise above about 25 percent, even while GPU memory use stayed nearly constant. That suggests the bottleneck was not simply raw GPU compute utilization. Memory pressure, CPU participation, thermal behavior, or synchronization overhead may have mattered more than the GPU utilization percentage alone.

The 31B test reinforces the larger lesson: model tuning is not only about pushing more layers onto the GPU. On this laptop, reducing GPU offload preserved VRAM headroom and made a larger model significantly more usable.

10. What Surprised Me

Several things surprised me during this process.

First, the smaller models were more capable than expected. E2B was not perfect, but it was genuinely useful and extremely fast. E4B was even better and felt like a practical local daily driver.

Second, the 26B model was not simply “too large.” It was terrible when I over-offloaded it to the GPU, but dramatically better after reducing GPU offload and letting CPU/system RAM participate more.

Third, the 31B model also became more usable after reducing GPU offload. It was still slower than the retuned 26B model, but it crossed from “not worth waiting for” into “this can complete the benchmark.”

Fourth, GPU utilization percentage by itself was not enough to explain what was happening. A low GPU utilization percentage did not mean the run was cheap or efficient. GPU memory, CPU utilization, system RAM, thermals, and model settings all mattered.

Fifth, tokens/sec and total completion time are not the same thing. A larger model may generate fewer tokens, or answer more compactly, while still having lower tokens/sec. That means total prompt time depends both on generation speed and on how much the model decides to say.

Finally, the biggest surprise was that the best GPU offload setting was not necessarily the model default, the highest setting available, or the highest setting that technically fit in VRAM. Lowering GPU offload below the default improved performance for the larger models by preserving headroom.

11. Lessons Learned

The biggest lesson is simple: start small and work upward.

The smaller models are not just training wheels. They are the best way to confirm that the runtime is configured correctly, the intended GPU is being used, and the machine can generate responses at a useful speed.

The second lesson is that “loads” does not mean “usable.” A model can load and still be unpleasant if it consumes nearly all VRAM, leaves no room for runtime behavior, or generates too slowly.

The third lesson is that VRAM matters more than total system RAM for this kind of GPU-accelerated local inference. Having 64 GB of system RAM is helpful, but it does not make a 16 GB GPU behave like a 24 GB or 48 GB GPU.

The fourth lesson is that available VRAM is less than advertised VRAM. Windows, displays, runtime overhead, LM Studio, KV cache, and other applications all consume memory before and during generation.

The fifth lesson is that smaller models may be better daily drivers. E4B was not the largest model I tried, but it had the best balance of speed, output quality, and headroom.

The sixth lesson is that settings matter. Context length, GPU offload, KV cache behavior, thinking/reasoning mode, thread pool size, and batch settings can change the result dramatically.

The seventh lesson is that maximum GPU offload is not automatically best. On constrained VRAM systems, a lower offload setting can leave enough memory headroom to make the whole pipeline faster.

Finally, record the settings. If you do not record context length, GPU offload layers, quantization, tokens/sec, and memory use, it becomes very hard to explain later why one run felt better than another.

12. Practical Recommendations

On this class of laptop, I would treat E4B as the practical default model from this test set.

E2B is useful as a very fast fallback and sanity-check model. It is lightweight, responsive, and easy to keep around. But its output needs a little more review.

E4B is the better daily assistant candidate. It is still fast enough to feel interactive, but its explanations and code responses were stronger.

The 26B A4B model is no longer something I would dismiss as unusable. After retuning GPU offload downward, it became surprisingly practical for larger reasoning prompts. I still would not keep it as my default daily model on a 16 GB VRAM system, but it is worth keeping as a tuned larger-model option.

The 31B model is also not impossible, but it requires more patience and careful tuning. At its higher/default offload setting it behaved like a stress test. After reducing offload, it could complete real prompts, but it remained slower than the 26B model.

For someone with similar hardware, my recommendation would be:

• Start with E2B to confirm the setup.
• Try E4B as the likely daily driver.
• Try 26B A4B only after understanding GPU offload and VRAM headroom.
• Treat 31B as a larger-model experiment, not a default.
• Leave VRAM headroom for the operating system, context cache, and other applications.
• Do not assume the default GPU offload setting is best.
• Try lower GPU offload values if a model loads but performs badly.

Hardware Starting Points

Use Case Starting Points

For someone with less VRAM, start smaller and be more aggressive about quantization.

For someone with more VRAM, the larger models become more interesting, especially if you want to keep a fast model and a reasoning model available without constantly fighting memory pressure.

13. Conclusion

Local AI is not only about saving money.

It is about privacy, offline access, control, experimentation, and learning how models actually behave on the hardware you own.

The encouraging part is that useful local AI is already possible on consumer hardware. The caution is that model names alone do not tell you whether the experience will be good. Neither does parameter count. Neither does the fact that a model loads.

The most surprising lesson from this test was that the largest useful configuration was not the one that pushed the most layers onto the GPU. In fact, some of the worst results came from trying to keep too much of the model in VRAM.

The best model is not simply the biggest model that loads. The best setting is also not necessarily the highest GPU offload value, the model’s default offload value, or even the highest value that fits inside available VRAM. On a constrained VRAM system, lowering GPU offload below the default may actually improve local model performance.

That changed how I think about local model tuning.

The goal is not maximum GPU offload. The goal is a balanced configuration that leaves enough VRAM for the model, runtime overhead, KV cache, and the rest of the system.

For everyday use on this laptop, E4B still looks like the best default model. It is fast, useful, and leaves plenty of headroom. E2B is a great lightweight fallback. The larger 26B and 31B models are better treated as reasoning or stress-test models that require careful tuning before they become practical.

A local model is not useful merely because it loads. It is useful when it fits your hardware, your workflow, and your patience.

The real benchmark is not “how big a model can I start?”

The better benchmark is: can I use this model, with these settings, on this machine, without breaking my flow?

If you’ve tried tuning Gemma 4 or other local models on your workstation or laptop, what GPU offload settings have worked best for your VRAM? I was surprised that lowering offload actually improved my speeds.