pandi-call — voice agent in Element Calls

Pandi joins a MatrixRTC (Element Call) call as a real participant, listens, answers, and can be interrupted mid-sentence. Audio never touches hardware: everything runs on PipeWire null sinks.

remote voices ──▶ hermes_ear (sink) ──▶ hermes_ear.monitor ──▶ VAD ──▶ Qwen3-ASR (GPU)
                                                                        │
   call ◀── hermes_mic (source) ◀── hermes_speaker (sink) ◀── Kokoro/Piper ◀── cloud LLM
                 ▲
                 └── Chromium (Element Call SPA) publishes the mic track

Barge-in needs no echo cancellation: our own TTS goes to hermes_speaker, the capture stream is hermes_ear.monitor, and Chromium plays only remote voices into hermes_ear. So anything the VAD hears on hermes_ear is another person.

Build / install

cd ~/coding/projects/pandi-call-rs
cargo build --release && install -m755 target/release/pandi-call ~/bin/pandi-call

Running as a user service (already enabled):

systemctl --user status pandi-call.service
systemctl --user restart pandi-call.service
journalctl --user -u pandi-call.service -f

Commands

command	what it does
pandi-call watch	the daemon: watches call membership in all joined rooms, joins, listens, leaves
pandi-call join <roomId>	join a call now (starts Xvfb + Chromium if needed)
pandi-call leave	hang up
pandi-call status	in_call true/false
pandi-call listen	duplex loop only (assumes the bot is already in a call)
pandi-call say "<text>"	TTS into the current call — proves the audio path
pandi-call ask "<text>"	LLM + TTS into the call, no microphone
pandi-call once <file>	offline: file → ASR → LLM → TTS, prints timings
pandi-call selftest	WHISPER, PIPER, llama-server, PipeWire devices, Chromium
pandi-call tool <name> '<json>'	run one tool directly (home / delegate) — for testing
pandi-call trace	debug: boots the SPA with network/console logging

PANDI_CALL_DEBUG=1 enables progress logging ([join], [cdp], [step], [tool]).

Doing things: home and delegate

The voice model gets two tools; both are executed by the Rust side, so a spoken answer is never the model's imagination.

• home — Home Assistant REST, 140–800 ms. It resolves the device from the name the human actually said, calls the service, then re-reads the entity and speaks that state: "Deckenventilator ist aus.", "Stehlampe ist an, auf 40 Prozent.". Unknown device → "Ein Gerät namens X kenne ich nicht." instead of a generic error. Only switchable domains are eligible (light, switch, fan, cover, climate, media_player, script, scene, input_boolean, vacuum), so "Temperatur" can't be toggled.

  Matching is built for speech, not for typing — ASR mangles names (light.corsair is called "Hänge Bett" in HA and comes back as "Hengebet", "Hänge bit", "Hängebettlicht"):

  score	rule	example
  100	equal after normalising (lowercase, ä→a ö→o ü→u ß→s, punctuation dropped)	"hänge bett" = "Hänge Bett"
  80	name starts with query	"Decken" → Deckenventilator
  60	substring, either direction	"Ventilator", "Hängebettlicht"
  55–60	Levenshtein similarity ≥ 0.78	"Nachttischlampe" → Nachtischlampe
  45	every query word occurs in the name	"Schlafzimmer Hängebett"
  40	longest common run ≥ 4 chars and ≥ 40 % of the shorter string	"Bettlicht" → Hänge Bett

  Ties: lower score loses, then the shorter name wins ("Ventilator" → Deckenventilator, not "TigerTag Auxiliary Fan Ventilator"), then alphabetical id. The 40 % clause on the last rule is what keeps "Fluxkompensator" away from "Deckenventilator" (both end in "ator").

  action is an, aus, toggle, set (with value in percent), or heller / dunkler — the relative ones need no guessed number, they move ±30 points from the read level. After switching, the result is polled until state and level match (30 × 150 ms for level and colour changes, 8 for state only) because Tuya reports brightness ~0.5 s after the state; without that Pandi said "auf 100 Prozent" while the lamp sat at 70.

  color sets a hue or a white tone: rot, blau, grün, lila, orange, gelb, türkis, pink, warmweiß, neutralweiß, kaltweiß — plus any English CSS name, passed through. German words become hs_color locally and white tones become color_temp_kelvin, deliberately not color_name: a hue we chose ourselves is a hue we can verify. Tuya answers 200 up to 3 s before the lamp is actually in the new colour, so the sentence is only spoken once hs_color matches (±15°, ±30 % saturation) or color_temp_kelvin is within 200 K. Without that check Pandi reported "Farbe türkis" while the lamp was still red. set without value does not dim to 50 % when a colour was given — "mach es rot" changes the colour only.

• delegate — everything that needs real work: hermes -p call chat -Q -q "<task>" in the dedicated call profile (own SOUL.md: spoken one-liners, units written out), as a child process with kill_on_drop. Model/provider/profile come from PANDI_CALL_EXEC_MODEL / PANDI_CALL_EXEC_PROVIDER / PANDI_CALL_EXEC_PROFILE (default deepseek-v4.1-flash:cloud on ollama-cloud, profile call), --reasoning none keeps it at 10–25 s. Two delegates may run at once (verified: parallel hermes -p call chat both exit 0).

  A missing tool is not an answer. Both prompts say so: the live system prompt ("Raten ist verboten, Absagen auch") and the call profile's SOUL.md, which explicitly allows the delegate to build the tool — src/tools.rs in this repo, cargo build --release and cargo test green, then systemctl --user restart pandi-call.service (the restart costs Pandi a few seconds of call presence, which is why it comes after a green build, never before). "Schalte den Fluxkompensator an" therefore ends in work, not in a refusal.

Speaking is decoupled from execution: the acknowledgement ("Mache ich." / "Ich kümmere mich darum.") is synthesised while the work happens. home finishes inside the turn; delegate returns immediately and its result arrives through an announce channel, spoken as soon as no turn is running — the human can keep talking in between. Agent answers are list-shaped by nature ("- Temperatur: 4 bis 10 °C"), so they pass one extra non-streamed LLM call that turns them into a single spoken sentence before TTS.

Measured in a live call, 2026-11-03: ASR 211–245 ms, first token 785 ms, ack audible 967 ms after the end of speech; a weather question was announced 14 s later as "Morgen in Erlangen wird es stark bewölkt bei vier bis zehn Grad, und es bleibt trocken."

Latency (measured on this host, 2026-11-03)

stage	ms from end of speech
ASR — Qwen3-ASR-1.7B Q8, RX580/Vulkan, llama-server :8082	420–660 (0.6B: 210–230)
first LLM token — deepseek-v4.1-flash:cloud via Ollama proxy	345–490
TTS — Kokoro-82M German "Martin", warm (:8881)	420–1180 per sentence
first audio out	≈1400–1800 (median over 67 turns: 1621)

ASR model choice is a trade, measured on real call dumps (/tmp/pandi-turn-*.wav):

clip	0.6B	1.7B
"…heißt Hängebett."	"Einst davon heißt Hengebet."	"Eins davon heißt Hängebett."
"Im Schlafzimmer unter Beleuchtung"	"In Schlafzimmer unter Beleuchtung"	"Im Schlafzimmer unter Beleuchtung."
"10 mm Regen morgen"	"Regenmorgen"	"Regen morgen"

Also tested and rejected on evidence: NVIDIA Parakeet TDT 0.6B v3 (ONNX int8, CPU, 25 European languages including German, 150–640 ms per clip). Ten German sentences spoken by Phonon, then degraded to telephone band with pink noise, both models on identical audio:

model	mean WER	mean latency
Qwen3-ASR 1.7B (current, RX580)	0.048	384 ms
Parakeet TDT 0.6B v3 (CPU)	0.072	248 ms

Parakeet is 1.4× faster and needs no VRAM, but it drops verb endings ("Pflanz" → "Pflanzt", "Mach" → "Macht") and mangles numbers — "fünfzehn Uhr dreißig" came back as "15 Uhr dreißig Uhr", which for a device agent that sets timers is disqualifying. Keep Qwen3. Benchmark reproducible with ~/.hermes/cache/scratch/asr-bench/bench.py (throwaway venv there; a fair WER needs real speech, the TTS-then-degrade trick only makes the comparison fair between models, not absolute).

1.7B costs ~+200 ms and gets word boundaries and compound nouns right; the 0.6B file is still on disk — revert by editing ExecStart in ~/.config/systemd/user/llama-asr.service.

Why cloud for the brain and GPU for the ears: the 8B on the RX580 needed 900–1300 ms to the first token, the cloud flash model needs 345–490 ms and answers better. The freed 8 GB of VRAM now holds the ASR model, which cut ASR from 800–1050 ms (whisper small, CPU) to 210–660 ms with correct proper nouns and numbers. reasoning_effort: none in the request is what removes the 700–1300 ms of silent thinking.

TTS: Phonon 40M German "Henrik" (default since 2026-11-04)

phonon-tts.service (~/coding/projects/phonon-tts, ptts runtime, GGUF q8) serves an OpenAI-compatible POST /v1/audio/speech on :8882, voice Henrik. speech::Tts speaks both protocols; the target comes from PANDI_CALL_TTS in .env and is read on every turn (there used to be a hardcoded constant here — Pandi kept speaking Piper while .env already said Phonon):

PANDI_CALL_TTS=http://127.0.0.1:8882/v1   # Phonon (current)
PANDI_CALL_TTS=http://127.0.0.1:8881/v1   # Kokoro "Martin" (fallback, container)
PANDI_CALL_TTS=127.0.0.1:10200            # Wyoming (Piper, the oldest path)

option	result (same sentence, this host)
Phonon-40M "Henrik" (current, CPU, 4 threads)	81–416 ms for 1.3–3.8 s audio, RTF 0.14; DC offset 0.000, silence at −77 dB
Kokoro-82M German "Martin"	420 / 1180 / 770 ms for 2.0 / 5.9 / 4.5 s audio — audibly grainy, DC offset +0.049 (that is the "noise" Emeric heard)
Piper de_DE-thorsten-high (previous)	480 / 990 / 640 ms for the same sentences
Qwen3-TTS-12Hz-1.7B Q8 on the RX580 (llama-tts)	16.2 s generation + 1.6 s vocoder for 6.3 s audio — 0.35× realtime, unusable
XTTS-v2 / Chatterbox / VibeVoice	PyTorch models; ROCm is dead on this RX580 and CPU is 5–30× realtime
Photon / Voxtral TTS	Photon is a CUDA inference engine only (same Kokoro weights, so same grain); Voxtral needs 16 GB VRAM
cloud (ElevenLabs Flash, Azure, Google Chirp)	audibly better, costs per character, needs a key — none in the vault

Thread count is not "more is better": 1/2/4/8/16 threads give 937/680/527/1316/2865 ms on 9.76 s of audio. Past 4 the synthesis segments start starving each other.

Intelligibility, measured by running our own Qwen3-ASR over both outputs and diffing against the input text: Kokoro 0 errors, Piper turns "Deckenventilator" into "dicker Ventilator".

Side effect worth knowing: the service normalises units before synthesis ("12,80 EUR" → "zwölf Euro achtzig", "zzgl." → "zuzüglich", "10 mm in 2 h" → "zehn Millimeter … zwei Stunden"), so spoken units no longer depend on the LLM following the prompt.

Voice comparison files in ~/tts-vergleich/: phonon-1..3.wav (Henrik), phonon-1-annika.wav, phonon-1-leon.wav, against vergleich-piper-thorsten-high.wav and vergleich-kokoro-martin.wav (same sentence).

First sentence is synthesised while the rest of the answer is still being generated; sentences are pipelined, so the gap between sentences is small.

How the call side works

Element Call (v0.26.1, served from https://call.mrschnirschuh.eu) runs in a real Chromium under Xvfb. It is driven over CDP with three primitives only: set localStorage, navigate, click [data-testid=lobby_joinCall].

Session handling is the fragile part. What the app tolerates:

• localStorage["matrix-auth-store"] must be a JSON string – writing a JS object yields "[object Object]", the app cannot parse it, silently drops the session and falls back to guest registration (403, "Lade …" forever).
• The app reads the session only at boot, so after writing it the page must be reloaded.
• A reload leaves the current CDP session unable to evaluate JS, so every reload has to be the last action of its process – do_join therefore runs one action per process (PANDI_CALL_STEPS, default 8).
• A reused Chrome profile can block on a half-open IndexedDB handle from a previous tab and hang in "Lade …" indefinitely. reset_browser() (kill Chrome + wipe profile) before a join is the only reliable cure; a fresh profile boots in ~10 s.
• Rooms need org.matrix.msc3401.call.member at power level 0 (in m.room.power_levels.events), otherwise only admins can join calls. Hermes-created rooms had state_default: 50, so the room Pandi Voice (!kZDgFmAoGcY4aq0wEV0KXp_HZRrc4KZK916weXbvgW8) got that exception added.

Beitritt: gemessene Zeiten

Die Zeit steckt nicht in festen Sleeps, sondern im SPA-Start:

Lage	Dauer
warmer Browser, Tab schon auf der Call-URL	0,85 s
Tab bootet, Session schon drin	2–3 s
Kaltstart (Chromium frisch, Profil neu)	12–25 s

Deshalb drei Regeln, die im Code stehen:

1. Ein leerer Zustand ("") heißt „die Seite bootet", nicht „Session kaputt" — sonst wird das Profil für ein Problem weggeworfen, das keines ist (~25 s Strafe).
2. home = eingeloggt, falscher Raum: navigieren, nicht neu aufsetzen.
3. Profil + Session erst ab Schritt 6 der Schleife verwerfen (vorher: 3).

Die Schritt-Zeiten stehen mit PANDI_CALL_DEBUG=1 im Log ([join] im Call nach N ms, Schritt N/8 nach N ms ohne Erfolg).

Was "im Call" heißt (und was nicht)

Der Auflegen-Knopf im DOM ist kein Beweis für Medien. Drei Prüfungen, in dieser Reihenfolge, und jede einzelne ist schon einmal falsch gewesen:

1. incall_leave da und lobby_joinCall weg → im Call (nicht in der Lobby).
2. roomHeader_participants_count > 1 → wirklich ein Mensch im Call. Matrix- Mitgliedschaften bleiben bis zu vier Stunden gültig, auch wenn der Tab nur geschlossen wurde; der Zähler in der Kopfzeile stimmt dagegen in Echtzeit.
3. media_ok() — ein Remote-<audio> mit srcObject muss currentTime weitertakten. Steht die Uhr, ist die RTC-Session tot, obwohl die UI weiter "im Anruf" zeigt.

watch tritt bei, wenn (2) zutrifft, und verlässt den Call nach 45 s Alleinsein; der veraltete Mitgliedschafts-Zeitpunkt wird gemerkt, damit dieselbe Leiche nicht sofort wieder beigetreten wird.

Der Weg von der Ferne in den Rechner lässt sich ohne zweiten Menschen prüfen: pw-cat --playback --raw --target=hermes_ear ... < ton.raw einspeisen, und ein Ton aus der Seite selbst (AudioContext per CDP) beweist Chromium → hermes_ear. Bleibt hermes_ear.monitor bei spielendem Remote-Track stumm, sendet die Gegenseite Stille (Mikrofon stumm) — dafür sagt Pandi nach 25 s ohne verstandene Sprache einen Satz in den Call, statt es nur zu loggen.

Wyoming protocol (wyoming >= 1.10)

Framing changed: the JSON line carries data_length and payload_length, and the blobs follow the line without separators. Piper answers audio-start → audio-chunk… → audio-stop, whisper answers transcript. Reading with a fixed "data in the line" assumption fails with trailing characters on the first audio chunk.

pw-cat playback of raw PCM needs --raw and an explicit - argument; without them it exits immediately and writes fail with Broken pipe.

Files

• src/main.rs — CLI, turn pipeline (streaming LLM → sentences → Piper → pw-cat), barge-in
• src/cdp.rs — Chromium/CDP: browser lifecycle, session injection, lobby click, media health
• src/audio.rs — parec capture with VAD, pw-cat streaming playback
• src/speech.rs — ASR over HTTP (llama-server /v1/audio/transcriptions), Wyoming client for piper
• src/llm.rs — OpenAI-format SSE (cloud or local) + sentence splitter
• src/matrix.rs — session, room state, call membership
• ~/.config/systemd/user/llama-asr.service — Qwen3-ASR at :8082 (GPU)
• ~/.config/matrix-call/session.json — bot session (device XXCYKAWAOQ)
• ~/.local/state/pandi-call/metrics.jsonl — one line per turn (asr, first token, first audio)

Configuration (env, .env works too)

variable	default
PANDI_CALL_LLM_URL	http://127.0.0.1:8081/v1/chat/completions
PANDI_CALL_LLM_MODEL	deepseek-v4.1-flash:cloud
PANDI_CALL_LLM_KEY	falls back to OLLAMA_API_KEY
PANDI_CALL_ASR_URL / _MODEL	http://127.0.0.1:8082 / asr
PANDI_CALL_CAPTURE / _PLAYBACK	hermes_ear.monitor / hermes_speaker
PANDI_CALL_ROOMS	empty = every room the bot is in