Are Open LLMs Still Underrated, or Just Not Ready for Enterprise?

are-open-llms-still-underrated,-or-just-not-ready-for-enterprise?

Geeta Kakrani

Open models already run in production at 42% of enterprises, yet they take only about 5% of what those enterprises spend on AI. That gap, not model quality, is the real story.

In Futurum’s September 2026 survey, production users said open weights now carry about 34% of their AI tokens, up from 23% a year earlier. Google’s Gemma runs in 55% of production accounts, up from 35%. But the money did not follow: the tokens move to open weights, and the budget reappears as servers, compute and engineering time. The survey is small (200 enterprises, 83 production users, mostly North America), so read it as a signal, not a verdict.

So are open LLMs underrated, or just not ready? The short answer is neither. They are ready for the routine layer of an AI system, and they pay off only when they sit behind a router with a budget on top. This article covers what open models really cost (hosting, quality, latency, skills), and how to recover from each cost.

The promise: stop paying frontier prices for routine work

Most agent traffic is routine: tagging, extraction, formatting, simple lookups. Sending all of it to the biggest model is the main reason API bills climb. Open models attack exactly this part, and the research on routing supports the idea.

  • FrugalGPT reports matching the best single LLM (for example GPT-4) with up to 98% lower cost, using prompt adaptation, model approximation and a cascade of models.
  • RouteLLM reports cutting cost by more than 2x in some cases without compromising response quality.
  • In the Futurum survey, low-complexity workloads grew from 39% to 51% of production users’ open-weight usage, so enterprises are already using open models where it is safest.

These are best-case, benchmark-style results. Your savings depend on how much of your traffic is truly routine, and on what the open model costs to run, which is the next section.

The hidden bill: four costs the API price hides

An open model has no per-token fee, but it is not free. The cost moves from your API invoice to hosting, quality work, latency tuning and people. Each deserves a number before you commit.

1. Hosting charges. Utilization decides everything. One third-party self-hosting guide estimates that at about 10% GPU load the effective cost per 1K tokens rises roughly tenfold, and that hidden costs add a 3–5x multiplier over raw GPU rental. It puts the crossover for a mid-priced API near 250M tokens a month, and only at 60–70% utilization. Another analysis finds that an 8x H200 node (about $10.5K a month) cannot beat an API bill below roughly that amount. Both are opinionated third-party estimates with May 2026 pricing, so rerun them with your own traffic.

2. Quality. Quantization is cheaper than most teams expect. A large study of quantized models calls FP8 essentially lossless, INT8 about 1–3% per-task loss, and INT4 competitive with INT8. Reasoning and math are more sensitive, so test them explicitly. Very small models also struggle: in a study of fine-tuned models under one billion parameters, they matched large models on sentiment and summarization but not on chain-of-thought reasoning or code generation.

3. Latency. Serving software matters as much as the model. One benchmark in the same guide shows 79 ms time-to-first-token on SGLang versus 103 ms on vLLM for an 8B model on an H100. The quantization study reports INT4 can cut latency by about 1.5–2x for 8B and 70B models in one-query-at-a-time serving. None of these numbers compare against a managed API, so benchmark your own baseline.

4. Lack of knowledge. This is the quiet cost. The guide cites a DevOps salary near $145K a year and about $12K for each model update cycle. In the Techstrong/ETR survey, 62% of pilot organizations said incomplete security and compliance reviews block production, and 50% named self-hosting cost.

How smart routing fits

The strongest case for open models is not replacing a frontier model. It is sitting inside a layered system where every layer is cheaper than the next one.

A request meets the cheapest layer first. A cache hit or a simple lookup costs no model tokens, the router sends routine work to a small open model, and only hard reasoning reaches the frontier model. A token budget caps spend across all of it, so one stuck loop cannot drain the rest.

Pros and cons at a glance

In a routed setup, open models win on cost control and data control, and lose on operations and hard reasoning.

Sources for the cells above are in the previous section and in the Futurum and emissions studies.

How to recover from each challenge

Every cost above has a known fix. The common thread is to shrink the traffic first, then host only what is left.

Hosting charges: shrink, then size.

  • Cache and route first, so the GPU only sees work that needs a model.
  • Quantize. The quantization study reports about 2–3x lower cost per query for 8B and 70B models with INT4 in one-query-at-a-time serving.
  • Own the base, rent the peak. A vendor model from Nadir (synthetic traffic, stated assumptions) shows GPUs sized for steady load plus API overflow costing less than GPUs sized for peak. Treat it as an illustration, not a benchmark.
  • Below roughly 250M tokens a month, stay on an API until volume grows.

Quality: prove it on your own tasks.

  • Run the open model in shadow mode beside the current one and compare answers before moving traffic.
  • Test reasoning and math explicitly if you quantize to 4 bits.
  • Fine-tune small models for narrow jobs. In the emissions study, fine-tuned small models matched large ones on four of six tasks.
  • Keep the frontier model as the path for hard cases.

Latency: measure, then tune.

  • Benchmark against your own API baseline, because no source here offers a universal comparison.
  • Test more than one serving engine, and use prefix caching when many requests share a long prompt.
  • Quantize, which the study links to roughly 1.5–2x lower latency on smaller models.

Lack of knowledge: start narrow.

  • Pick one routine task, one model and one owner, and run it for a few weeks before expanding.
  • Use a gateway that already tracks spend per key, such as LiteLLM virtual keys, rather than building your own.
  • Start security and compliance review early, since it is the most common blocker in the ETR survey.
  • Write down what worked, so the next team does not start from zero.

Can this really be sustainable?

Yes, but only as a routed, right-sized system. Swapping everything to open models is neither cheaper nor greener.

Sustainable has three meanings here, and a design has to pass all three:

  • Cost. The total cost of ownership must beat the API once hosting, people and idle time are counted. That happens when utilization is high and volume is steady, which is why the base-and-overflow pattern works.
  • Energy. Small models can use far less energy on suitable tasks. The emissions study estimates per-query emissions up to about 13,000x lower for sentiment analysis, but these are estimates for fine-tuned models under 250M parameters, and the same models could not match large ones on reasoning or code generation.
  • People. A setup nobody on the team can maintain is not sustainable. This is why starting with one task, with budgets and cost tracking in place, matters more than the model choice.

The honest limit: the evidence for energy savings is thin and mostly modeled, and the cost evidence comes from third-party analyses. Measure your own cost per task and your own utilization before you claim a saving.

A checklist before you move traffic to an open model

If you can tick most of these, an open model is likely worth a pilot. If you cannot, stay on an API for now.

  • ☐ The task is routine and repetitive: tagging, extraction, formatting, classification
  • ☐ Monthly volume is large and steady, ideally above a billion tokens, so GPUs stay busy
  • ☐ Repeated questions and simple lookups are already handled by a cache or by rules
  • ☐ You have tested the open model against your current one on your own data, in shadow mode
  • ☐ Each agent has its own key, and budgets exist per key, per model and per task
  • ☐ Cost per task is tracked, not only the monthly bill
  • ☐ A frontier model remains the path for hard reasoning
  • ☐ Someone owns hosting, updates and security review
  • ☐ Privacy, compliance or data-residency needs justify the extra work

If volume is under roughly 250M tokens a month, or most of your tasks need hard reasoning, an API is usually the simpler and cheaper answer. These thresholds are rules of thumb from third-party guides, so rerun them with current prices.

So, underrated or not ready?

Both are half right. Open LLMs are underrated for routine, high-volume work, and not ready as a blanket replacement for frontier models.

The winning design is a layered one: cache what repeats, answer simple lookups with rules, send routine tasks to a small open model, keep a frontier model for hard reasoning, and put a token budget across all of it. Done that way, open models cut API cost, and the hosting, quality, latency and skills costs stay manageable.

Where in your own stack could a small open model take the routine work today, and what is stopping you?

Sources

All pages were opened on 31 October 2026. Prices and survey figures change quickly, so recheck them before you decide.


Are Open LLMs Still Underrated, or Just Not Ready for Enterprise? was originally published in Google Developer Experts on Medium, where people are continuing the conversation by highlighting and responding to this story.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
leveling-the-playing-field:-building-a-100%-offline-kubernetes-docs-rag-exam-assistant-with-webai

Leveling the Playing Field: Building a 100% Offline Kubernetes Docs RAG Exam Assistant with WebAI

Next Post

Three Booleans Is Not a State Machine: Modelling UI Logic That Can’t Be in Two States at Once

Related Posts