Fun-ASR encodes the full audio context before any language-model decoding. That work used to happen inside the LM stage — serialized per request, repeated for identical audio. This change moves encoding upstream of LM admission, then arms it with batching, single-flight deduplication, and a bounded CPU LRU cache.
Fun-ASR is full-context: the whole utterance is encoded before the LM generates a single token. On current main, that encode runs after a request is admitted to the LM stage — one request at a time, inside the model runner.
A dedicated service completes audio encoding before LM admission. Requests reach the LM stage with LM-ready embeddings already attached — the LM never touches raw audio features.
Concurrent audio requests. Each request's metadata now carries an audio_fingerprint and num_audio_tokens.
Batches concurrently arriving encodes, dedups identical in-flight audio, serves repeats from cache — on a dedicated CUDA stream.
Only complete encodings pass. Cached embeddings are validated for shape, token count, and dtype before reuse; failures never enter the LM stage.
Precomputed embeddings attach directly to the multimodal request; the original CPU feature tensor is released after encoding.
peak GPU mem −1.1 GiBConcurrently arriving encoder requests encode together. If a batched encode fails, the service degrades to per-item encoding and propagates unrecoverable failures only to the affected requests.
Concurrent requests for identical audio share one encoder execution. In the benchmark this merged 421 of 1,088 requests — reuse that exists even with a completely cold cache.
Configurable entry and byte limits. Stores only complete encoder/adaptor outputs — deliberately no prefix reuse of partial audio windows. StageOutputCache was made thread-safe for concurrent access.
Namespaces and keys make a stale or mismatched hit structurally impossible; every hit is re-validated before use.
_get_audio_feature_uncached() is extracted so the regular and pre-LM paths run the same underlying encoder code — no behavioral fork. New encoder queue, batching, latency, failure, and cache statistics ship with the service.
Full Seed-TTS English set — 1,088 samples, 666 unique audio files — withFunAudioLLM/Fun-ASR-Nano-2512-hf on one NVIDIA H100 80GB. The two passes share one fresh server: cold-start on an empty cache, then warm reusing it.
| Metric | Main | New · cold | New · warm |
|---|---|---|---|
| Throughput (samples/s) | 25.962 | 34.738 | 86.789 |
| Wall time (s) | 41.907 | 31.320 | 12.536 |
| Mean latency (s) | 1.229 | 0.914 | 0.362 |
| p95 latency (s) | 1.802 | 1.114 | 0.504 |
| p99 latency (s) | 3.015 | 1.896 | 0.800 |
| Mean RTF | 0.26585 | 0.19965 | 0.07828 |
| Corpus WER | 0.01710 | 0.01710 | 0.01710 |
| Skipped requests | 0 | 0 | 0 |
// cold start: +33.8% throughput, −25.6% mean latency — before the cache has anything in it.
// warm: +234.3% (3.343×), wall time 41.9s → 12.5s. WER identical everywhere.
A follow-up run pins main (f916b86c) against the change (e6f49902), plus astrict no-hit control: 666 SHA-256-verified unique files, telemetry-confirmedhits=0, merged=0. This separates caching from everything else.
// batching alone doubles effective utilization — no cache hits required.
// exactly-idle GPU time: ~0.70s → ~0.41s; peak memory −1,099 MiB; mean power +17.6W (the GPU is finally working).
| No-hit control | Main | Change | Δ |
|---|---|---|---|
| Throughput (samples/s) | 32.178 | 32.231 | +0.17% |
| Mean latency (s) | 0.991 | 0.974 | −1.7% |
| p95 latency (s) | 1.543 | 1.062 | −31.2% |
| p99 latency (s) | 2.177 | 1.105 | −49.2% |
| p95 RTF | 0.3326 | 0.2998 | −9.9% |
| Peak GPU mem (MiB) | 73,062.9 | 71,963.5 | −1,099.4 |
| Corpus WER | 0.017715 | 0.017715 | identical |
// 666/666 evaluated, 0 skipped, per-sample WER distribution identical.
Cold-start pass, 1,088 requests, cache empty at t=0. The 666 misses match the 666 unique files exactly — and on top of that, single-flight merged 421 concurrent duplicates that arrived before the first encoding of their audio had finished.
// once populated, every request reuses a complete cached encoder output.
// caveat: the service's narrow hits/(hits+misses) metric read 0.15% on the cold pass
// because it excludes single-flight merges — 38.79% effective reuse is the number that matters.
Main and the change — cold and warm alike — produced bit-identical quality on the full dataset, and incomplete or failed encodings are structurally barred from the LM stage.
"The GPU wasn't slow — it was waiting in line.
Encode once, upstream, and admit only what's complete."