FunASR’s MCP server just became one of the most-starred speech tools in the ecosystem. modelscope/FunASR sits at roughly 21k stars with a ready-to-run MCP endpoint that hands any agent a transcribe_audio tool backed by SenseVoiceSmall (or larger Fun-ASR models). The Glama ranking and the v1.3.3 release notes make the pitch clear: drop a local, private, OpenAI-compatible or MCP transcription backend into Claude Code, Cursor, or any agent loop without shipping audio to the cloud.
That is useful. It is also not the hard part if you are building a real-time overlay for calls or interviews on Windows.
The agent-facing half is solved
The MCP server is intentionally simple. A minimal config for Claude Code looks like this:
{
"mcpServers": {
"funasr": {
"command": "python",
"args": ["https://dev.to/path/to/examples/mcp_server/funasr_mcp.py"],
"env": {"FUNASR_DEVICE": "cuda", "FUNASR_MODEL": "iic/SenseVoiceSmall"}
}
}
}
Or the OpenAI-compatible path:
pip install funasr fastapi uvicorn python-multipart
funasr-server --model sensevoice --device cuda
# POST /v1/audio/transcriptions
The model does VAD, recognition, punctuation, and optional timestamps in one call. On a decent GPU it is comfortably faster than realtime for file-based work. For an agent that needs to “listen to this recording and summarize,” the problem is solved.
Real-time overlays live in a different regime
An interview or call assistant has different constraints:
- Two independent sources (your mic + the remote party via loopback).
- Continuous 20–40 ms chunks, not whole files.
- End-to-end latency budget that is usually under 300–500 ms before the overlay feels laggy.
- The model never sees clean audio if the capture path is wrong.
The dominant term is still the Windows audio path. Shared-mode WASAPI goes through the audio engine (audiodg.exe + audioeng.dll), APOs, and a fixed period. Exclusive mode (or a carefully tuned shared mode) is the only reliable way to keep the buffer period in the single-digit to low-teens milliseconds.
A practical dual-track shape that actually ships:
- Mic stream:
IAudioClienton the capture endpoint, preferably exclusive or with a short period. - Loopback stream:
IAudioClienton the render endpoint withAUDCLNT_STREAMFLAGS_LOOPBACK. - Each callback only copies samples into a lock-free SPSC ring (power-of-two capacity). No model work in the audio thread.
- Consumer threads resample to 16 kHz mono, run a cheap VAD, and feed a streaming session (or chunk the FunASR model if you are willing to accept higher latency).
Typical additive costs once samples leave the callback:
| Stage | Typical cost |
|---|---|
| WASAPI shared period | 10 ms (5–8 ms if requested) |
| Callback → ring copy + format convert | 1–3 ms |
| Resample to 16 kHz mono + lightweight VAD | 2–5 ms |
| Local model first partial (or network RTT) | 40–150 ms |
| Overlay paint | <16 ms |
FunASR (or any local model) only affects the last model stage. If your capture path is a 40 ms Bluetooth shared-mode period plus engine enhancements, the recognizer is already starting late.
Wiring the two halves together
You can still use the FunASR MCP or OpenAI-compatible server for the model side. The difference is the feeder:
- File mode: agent or script drops a WAV and calls the tool. Great for post-call notes.
- Streaming mode: your capture process writes short PCM chunks (or uses a local streaming endpoint if the model supports it) and pushes partials to the overlay.
A minimal consumer sketch (Python, after the rings are filled):
import numpy as np
from funasr import AutoModel
model = AutoModel(model="iic/SenseVoiceSmall", vad_model="fsmn-vad", device="cuda")
def on_chunk(pcm_f32_16k: np.ndarray):
# FunASR generate expects a path or array depending on version;
# many people write a short temp WAV or use the streaming API when available.
result = model.generate(input=pcm_f32_16k, batch_size=1, language="auto")
text = result[0]["text"]
# push to overlay / translation layer
return text
Tradeoffs are the usual ones:
- File-oriented MCP is simpler and more accurate on long context; it is not low-latency.
- Streaming local models (or chunked calls) cut latency but raise WER on short fragments and require careful VAD + overlap.
- Exclusive mode gives the best period numbers and fights with other audio apps; shared mode is friendlier and adds engine latency.
- GPU vs CPU: SenseVoiceSmall is usable on CPU for short chunks; larger models want CUDA or a fast NPU.
What actually moves the needle
Agent integrations (MCP, OpenAI-compatible servers, tool-calling) make local ASR easy to call. They do not shrink the WASAPI period, remove APO processing, or give you clean dual-track separation. Those remain Windows-specific engineering problems.
If you are building a real-time translation or interview overlay, spend the first week on the capture path (two independent clients, lock-free rings, exclusive or short-period shared, VAD before the model). Only then plug in FunASR, Whisper, or whatever local model is currently fastest on your hardware. The MCP server is a convenient way to expose that model to agents for offline analysis; the live path still has to be hand-rolled.
I’m building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project’s code and logs.