Compare Cloud and On-Device AI Costs Without Inventing Energy Numbers

“On-device AI saves battery” and “cloud AI is more efficient” can both sound plausible. Neither is a measurement.

The placement decision crosses at least four different budgets:

user wait + network transfer + provider spend + device energy

Do not collapse them into one vague “cost” number. Measure each with its own unit and evidence boundary.

Start by identifying the actual execution path

I reviewed MonkeyCode mobile code at commit c58bcd4. The task stream opens a server-supported WebSocket. The speech-to-text hook also participates in a server-supported streaming path.

That reviewed path is not evidence of on-device model inference. So a fair current study would measure a mobile client using remote task and voice services. An on-device alternative would be a separate prototype with its model, runtime, and packaging declared.

Record a measurement envelope

The included CSV template begins with these fields:

sample_id,sample_kind,placement,device,os,framework,model,network,input_tokens,output_tokens,latency_ms,bytes_up,bytes_down,energy_joules,cost_usd

Why so many?

• device, os, and framework make thermal and runtime results interpretable;
• model and token counts keep workload size visible;
• network separates offline, Wi-Fi, and cellular behavior;
• latency is milliseconds, transfer is bytes, energy is joules, and provider spend is currency;
• sample_kind prevents synthetic examples from masquerading as device measurements.

Battery percentage is too coarse for short runs. It is affected by display, radio, background work, battery health, temperature, and OS estimation. If you cannot collect energy with an appropriate platform profiler or external power measurement, leave energy_joules empty.

Use matched user flows

Compare the same tasks, not unrelated model demos:

Warm-up should be reported separately. Randomize case order, control display state and temperature, repeat enough times to show a distribution, and record failures rather than deleting them.

Make the analyzer refuse fake energy conclusions

The companion synthetic-samples.csv is explicitly labeled synthetic. Its numbers exist only to test parsing and summary behavior.

Run:

node analyze-costs.mjs synthetic-samples.csv
node test-analysis.mjs

The report includes:

{
  "dataset": "synthetic",
  "samples": 3,
  "mean_latency_ms": 987,
  "total_network_bytes": 66100,
  "total_cost_usd": 0.0084,
  "energy_conclusion": "REFUSED: energy requires measured joules for every row"
}

Those latency, transfer, and cost values are synthetic test output, not a MonkeyCode benchmark, a phone benchmark, or evidence that one placement wins.

The test then supplies measured-labeled rows with joule values and confirms that an energy summary becomes possible. In a real pipeline, provenance should be stronger than a label: ingest profiler exports, preserve raw files, and attach the collection command and timestamp.

Decide with separate budgets

A release decision can now be explicit:

• P95 interaction latency must meet the user-flow target;
• network bytes must fit the cellular/offline policy;
• provider spend must fit the workload forecast;
• measured energy and thermal behavior must meet a device-specific budget;
• privacy must account for which content leaves the device;
• quality must be evaluated on the same task set.

An on-device model can remove provider calls while adding download size, RAM pressure, thermal load, and model-update work. A cloud model can reduce device compute while adding radio use, service dependency, and data transfer. Units keep those tradeoffs honest.

Disclosure: I contribute to the MonkeyCode project. The product observations are source-based and limited to the linked paths at commit c58bcd4. All included sample data is synthetic; no device energy, latency, or cost benchmark is claimed.