How Penwright cleans up your dictation
Dictation is a typing replacement, not a writing tool. The moment cleanup feels slower than typing, the trick stops working. We spend a lot of time pushing the latency down, which means we spend a lot of time choosing not to call a large language model.
Here's the design we settled on, and why.
Three tiers, smallest first
Every dictation you speak goes through a pipeline of up to three tiers. The first one always runs. The second runs by default. The third only runs when you ask for "Polish" mode. Each tier is allowed to escalate to the next; each tier is also allowed to fail silently and let the previous tier's output stand.
Tier 1, the rules engine
The first stage is a plain Rust function. Under fifty milliseconds, it does the boring jobs that don't need a model:
- Strip filler words, um, uh, like, you know, plus their German counterparts ähm and äh.
- Collapse repetitions ("the the", "I I think") when they look unambiguously accidental.
- Recognize voice commands, "new line", "new paragraph", "period", "comma", "question mark", "open quote", "close quote", and replace them with the right characters.
- Apply your personal dictionary so "Stripe" stays capitalized, your colleague's name stays spelled the way you taught it, and your project codenames don't get autocorrected into something else.
- Expand snippet triggers so "intro email" turns into your three-line signature block.
- Fix obvious capitalization, sentence boundaries, the pronoun "I", brand names you've registered.
None of this needs probability. It needs a finite state machine and some good defaults. We get back text that's already 80% of the way to publishable, and we got there in single-digit milliseconds.
Tier 2, a small local model (default)
The rules engine knows what to do; it doesn't know how to read tone. For the last mile, "does this sentence still sound like me", we run a small instruct model over the rules output, loaded through llama.cpp with Metal acceleration. It is a 4B model, quantised, and it is the same one on every Mac Penwright supports. On an M-series machine that is a sub-second round trip, masked by the floating bar's "Cleaning up…" state so it does not feel like waiting.
The system prompt is short and protective: preserve the speaker's wording, don't rewrite for style, don't answer the user, don't summarize. The model's job is to nudge, not to take over.
The size is a latency decision, not a budget one. We have run this stage on everything from a 1.7B up to a 14B. Bigger models are sharper on edge cases and slower on every single dictation, and the edge cases are rare while the latency is constant. A 4B was where accuracy stopped improving faster than the wait grew, in English and in German, on a Mac that might only have 8 GB of memory. That is the whole argument.
Tier 3. Polish, opt-in
Sometimes you want more than a nudge. Ask for a rewrite and Tier 2 runs again with a different prompt, one that is allowed to restructure sentences and fix grammar properly. It is opt-in for a reason: rewriting is the one thing a dictation tool can do that makes people distrust it. A prompt you dictate into Claude or ChatGPT is never rewritten at all, no matter what is switched on.
Why we don't escalate further
The fashionable thing right now would be to dump every sentence into a frontier model and let it work out what you meant. That sounds great in a demo. In practice, three things go wrong:
- The round trip is too slow. Hosted frontier models add one to three seconds per request. Add network jitter and a busy hour, you're past five.
- The privacy story collapses. As soon as your voice transcript leaves your machine, you're trusting a vendor's logging policy, retention defaults, and breach posture.
- The model rewrites you. Bigger models reach for fluency. They smooth your sentences into shapes you didn't actually use. For a typing replacement, that's a feature most users notice immediately and dislike.
Local + small + tier-routed isn't a compromise. It's a different bet about what dictation should feel like. Pressing a hotkey, speaking a sentence, and seeing it appear in your editor before you finish the next thought, that's the experience we're tuning for. Nothing else.
Except when you are not waiting
Everything above is about the keystroke path, where a second of latency is the entire product. Not all of the work is on that path. When Penwright writes up an hour-long meeting, you are not watching a cursor; you have left the room. There is no latency ceiling to respect, so the argument reverses.
For that work the app uses the strongest model your Mac's memory can actually hold, fetched quietly in the background: an 8B from 16 GB, a 12B from 24 GB, a 30B mixture-of-experts from 48 GB. It is picked by installed memory, and checked again against free memory before a cold load, because a model that swaps is worse than a smaller model that fits. Dictation keeps its fast 4B throughout.
So the rule is not "small models are better". The rule is that the size should be decided by whether a human is waiting on the result, and by the machine in front of you, rather than by which plan you are on. Nothing about that changes where it runs.
Graceful degradation, everywhere
Pipelines this layered fail in interesting ways. The rule we built around it: at every tier, if anything throws, we drop to the previous tier's output and keep going. Model didn't load? You get the rules output. Rules engine threw on a malformed regex? You get the raw Whisper transcript. We never leave you staring at a frozen state machine because of a bug deeper in the stack, you always get text on screen.
That's the design. It's opinionated. It's deliberately smaller than the competition. And it's the reason your transcript never leaves your Mac, which is, to us, the only promise that matters.