How Penwright cleans up your dictation
Dictation is a typing replacement, not a writing tool. The moment cleanup feels slower than typing, the trick stops working. We spend a lot of time pushing the latency down, which means we spend a lot of time choosing not to call a large language model.
Here's the design we settled on, and why.
Three tiers, smallest first
Every dictation you speak goes through a pipeline of up to three tiers. The first one always runs. The second runs by default. The third only runs when you ask for "Polish" mode. Each tier is allowed to escalate to the next; each tier is also allowed to fail silently and let the previous tier's output stand.
Tier 1, the rules engine
The first stage is a plain Rust function. Under fifty milliseconds, it does the boring jobs that don't need a model:
- Strip filler words, um, uh, like, you know, plus their German counterparts ähm and äh.
- Collapse repetitions ("the the", "I I think") when they look unambiguously accidental.
- Recognize voice commands, "new line", "new paragraph", "period", "comma", "question mark", "open quote", "close quote", and replace them with the right characters.
- Apply your personal dictionary so "Stripe" stays capitalized, your colleague's name stays spelled the way you taught it, and your project codenames don't get autocorrected into something else.
- Expand snippet triggers so "intro email" turns into your three-line signature block.
- Fix obvious capitalization, sentence boundaries, the pronoun "I", brand names you've registered.
None of this needs probability. It needs a finite state machine and some good defaults. We get back text that's already 80% of the way to publishable, and we got there in single-digit milliseconds.
Tier 2, a small local model (default)
The rules engine knows what to do; it doesn't know how to read tone. For the last mile, "does this sentence still sound like me", we run a small language model over the rules output. Today we run EuroLLM 1.7B Instruct in Q4_K_M, loaded through llama.cpp with Metal acceleration. It's a ~1 GB on-device model trained by a European research consortium on all 24 official EU languages plus 11 commercially important ones. English and German are both first-class training languages, which matters because Penwright ships in both. On an M-series Mac that's a sub-second round trip, masked by the floating bar's "Cleaning up…" state so it doesn't feel like waiting.
The system prompt is short and protective: preserve the speaker's wording, don't rewrite for style, don't answer the user, don't summarize. The model's job is to nudge, not to take over.
We landed on 1.7B after testing 3B and 7B locally. 7B is sharper on edge cases but doubles latency without doubling perceived quality. 3B is what we used to ship, it works, but its German is markedly weaker than a model trained with German as a first-class language. EuroLLM at 1.7B is the smallest open model that handles both languages natively at a footprint that fits on every Mac, including 8 GB models.
Tier 3. Polish, opt-in
Sometimes you want more than a nudge. The "Polish" mode in Settings turns Tier 2 into a heavier rewrite, the same EuroLLM 1.7B with a different prompt that's allowed to restructure sentences and clean up grammar more aggressively. We deliberately don't expose a 7B or 13B option. Above 3B, the latency cost doesn't pay rent on this task. The quality delta for "remove filler, fix punctuation, keep my voice" is small; the wall-clock cost of waiting two extra seconds is not.
Why we don't escalate further
The fashionable thing right now would be to dump every sentence into a frontier model and let it work out what you meant. That sounds great in a demo. In practice, three things go wrong:
- The round trip is too slow. Hosted frontier models add one to three seconds per request. Add network jitter and a busy hour, you're past five.
- The privacy story collapses. As soon as your voice transcript leaves your machine, you're trusting a vendor's logging policy, retention defaults, and breach posture.
- The model rewrites you. Bigger models reach for fluency. They smooth your sentences into shapes you didn't actually use. For a typing replacement, that's a feature most users notice immediately and dislike.
Local + small + tier-routed isn't a compromise. It's a different bet about what dictation should feel like. Pressing a hotkey, speaking a sentence, and seeing it appear in your editor before you finish the next thought, that's the experience we're tuning for. Nothing else.
Graceful degradation, everywhere
Pipelines this layered fail in interesting ways. The rule we built around it: at every tier, if anything throws, we drop to the previous tier's output and keep going. Model didn't load? You get the rules output. Rules engine threw on a malformed regex? You get the raw Whisper transcript. We never leave you staring at a frozen state machine because of a bug deeper in the stack, you always get text on screen.
That's the design. It's opinionated. It's deliberately smaller than the competition. And it's the reason your transcript never leaves your Mac, which is, to us, the only promise that matters.