Gemma4 with audio input: 16.8 tok/s on Macbook M2 Max 64GB

Here's the setup I decided on for embedding gemma4-12b into a Tauri2 desktop app:

- Native Rust FFI into llama.cpp via llama-cpp-2

(Metal enabled)

- Model: gemma-4-12b-it-Q5_K_S

quantized by Unsloth, Q5_K - Small

- Audio input is a 607 KB 16-bit mono 16 kHz PCM WAV.

- Prompt path: Gemma chat template + llama.cpp mtmd (multimodal) audio marker, with prompt of "Transcribe this audio exactly."

- Benchmark input: 503 multimodal tokens/positions, including 486 audio tokens.

I'm not very well versed on model benchmarking, but it appears to be giving 16.8 tok/sec first-inference performance, model already loaded. The total-path speed breaks down as 2s for audio/prefill plus 3.7s for decode, with decode alone at 26 tok/s.

Does that seem like a reasonable level of performance? Any suggestions on how to speed up the inference?

Other approaches I considered:

- mlx-swift-lm

, but I need audio support, and it doesn't seem to be supported (filed issue #393)

- llama-server

in a sidecar, but that seemed harder to manage the lifecycle

- crabnebula-dev/tauri-plugin-llm

, but I don't think it supports gemma4 (filed issue #22)

The inference and benchmark test code is available in this prototyping-only repo: https://github.com/tleyden/tauri2-local-llm