Engineering · 8 min read · May 2026

Why we built Penwright local-first

When we started building a dictation tool, the obvious shape was a thin client over a fast cloud transcription API. We picked the harder path. Here’s why — and what we learned doing it.

Voice is the fastest input humans have. We talk at 150 words per minute comfortably, and burst past 200 when we’re excited. Typing peaks somewhere around 60 for most of us, and that’s on a good keyboard with both hands free. The gap is wide enough that it shouldn’t exist — the friction isn’t speech, it’s the tooling around it.

Almost every dictation product solves that by piping your microphone audio to a cloud endpoint, getting a transcript back, and pasting it. It’s the path of least resistance. The infrastructure is solved: you call Whisper-API or AssemblyAI, you bill the customer, you ship. You also create three problems we didn’t want to live with.

The three problems with cloud dictation

The first is privacy. We didn’t want to be the company that keeps a transcript of your investor update, your hiring debrief, your therapist appointment, your half-formed essay about a co-worker. Even with good intentions and a tight retention policy, that data exists somewhere we don’t fully control — in backups, in logs, on someone else’s S3 bucket, in some intern’s CSV export. The only way to be honestly private is to never receive the data in the first place.

The second is latency. A round trip to a cloud transcription endpoint, even a good one, costs you 3–5 seconds. That doesn’t sound like much until you live it. You speak a sentence, then you wait. You speak another, then you wait. Voice stops feeling like the fast input it is. It feels like a slow API call you happen to trigger with your mouth. Local Whisper on Apple silicon with Metal acceleration is ~1 second end-to-end for a normal sentence. You speak, you breathe, the text is there.

The third is resilience. The most reliable dictation tool you’ll ever own is one that doesn’t need the internet. Train, airplane, café, bad hotel Wi-Fi, dead-zone co-working space — voice should still work. If your dictation app depends on us being up, on your DNS resolving, on a Cloudflare outage not happening, then we’ve made you fragile in exchange for a faster ship date.

The engineering bet: Whisper + llama on Metal

The bet that made this possible is that Apple silicon and the open C++ ports of the leading models are good enough — today, on hardware most of our target users already own. We use whisper.cpp for transcription, exposing it through Rust bindings with Metal kernels turned on. We use llama.cpp(specifically EuroLLM 1.7B Instruct, an EU-trained open model with native English and German) for the cleanup pass — the one that turns “um hey karen quick one can you send me the q3 numbers” into a sentence with proper punctuation and case.

Both run on-device, both use Metal where it pays off, and together they fit comfortably under the 1-second budget we set for the round trip from finishing your sentence to the text appearing under your cursor. The cleanup pass is the riskier of the two — it’s the one where a smarter cloud model would visibly do better — but in exchange we get the privacy promise without an asterisk.

The tradeoffs we made on purpose

Going local-first cost us. The download is bigger. The first-run experience is slower — we have to fetch a transcription model and a cleanup model, both of which are hundreds of megabytes. The cleanup model isn’t GPT-4. We don’t support Windows yet, because we don’t have a Metal-equivalent story for it. We don’t support Intel Macs at full quality, because the performance ceiling on Intel is too low for the snappy round trip we want users to feel.

Each of those is a real cost. None of them is worse than the alternative — building a product whose central feature is shipping every word you speak to someone else’s servers.

What the cloud is still for

We do run a small backend — for accounts, licenses, and Stripe billing. Those are the obvious places for a server. They have nothing to do with your transcripts. The local-first promise is encoded as code, not as a marketing line: the routes that could ever touch your audio or text are gated behind a build flag that’s off in production. If you’re curious, that flag lives in apps/api/src/index.ts, and the production privacy contract is literally that one branch.

What we want this to mean

Local-first isn’t a premium feature. It’s a default we think more software should adopt — especially software that touches the most intimate input modality we have. We want Penwright to be the proof point: that you can build a polished, fast, profitable productivity tool without making your users into the product.