A local model on the call path: grammar-constrained, resident, signed
The AI receptionist I launched for a client in October 2026 has one language model on the live call path, and it is small: Qwen3-1.7B, running on llama.cpp on the client's own host. Every sentence a caller says goes through it. It never writes a sentence a caller hears and never states a business fact. It answers one narrow question at a time, and its answer is one word from a closed list. Getting a model that small to be trustworthy on a phone line came down to three properties, and I learned each one from a measurement or an outage rather than from a reading list.
Resident: ~4,000 ms to ~330 ms by not reloading the model
The first integration ran the command-line tool once per utterance. Measured on the production host on 19 September 2026, intent classification took about 4,000 ms. Almost all of it was reloading a 1 GB model for every sentence a caller spoke. Switching to the server build, with the model held in memory, brought the same classification to about 330 ms (764 ms on the very first call). Twelvefold, and the difference between a receptionist you can talk to and one you cannot.
For honesty about the number: 330 ms was my own measurement of a smaller 4-bit quantisation on eight threads, recorded in a source-code comment on 19 September. Production has run a larger 8-bit model on six threads since 22 September. A benchmark of that configuration on 16 October, run on the host through the receptionist's own client over a 43-utterance set, gave about 470 ms median and 583 ms at the 90th percentile of server-side classification time. That is a controlled benchmark, not a live-call measurement, and I have not measured end-to-end latency as a caller experiences it: in the tester calls recorded on 15 October, a model turn took 0.9 to 2.9 seconds. These figures are real; they are measurements of different configurations and scopes, and I keep them apart.
The second latency lesson was about the shape of the question. An early version asked for the intent plus nine extracted fields as one JSON object. On this hardware that cost 5 to 10 seconds per turn, nearly all of it generating about 160 tokens of JSON, against roughly 400 ms for a three-token intent. It also produced the worst failure of that run: asked to extract fields from "the first one please", the model returned a complete invented customer with a name, a street address and a ZIP code. Small models fill schemas. So the model now answers exactly one narrow question per call, field extraction is attempted deterministically first, and the model is asked only for the single named field the scanner could not resolve.
Constrained: a grammar, not a hope
Two findings shaped how the model is allowed to answer. The first was that Qwen3 is a reasoning model and defaults to thinking aloud; without an explicit instruction not to, it spent its entire token budget narrating and emitted no answer at all (zero usable outputs at 32 and 64 tokens, measured). The second was that before any constraint, a blocked-sink utterance came back labelled clogged, a perfectly reasonable English word and not a thing the system can act on.
So the output is constrained by a grammar. The nine permitted labels are compiled into a llama.cpp GBNF rule of the form root ::= "book_visit" | "emergency" | "ask_hours" | …, and the server cannot emit anything else. The parser still rejects anything outside the list, and anything unrecognised, malformed or timed out becomes unclear, which routes to a clarifying question and then to a human. The model's self-reported confidence is carried as a weak hint only: it is a number a language model produced, with no calibration behind it.
The prompt was tuned once, with five worked examples, and then frozen. Definitions alone scored 12 of 15 on a label set and made two absurd mistakes ("can I speak to a person" came back ask_hours); with the examples the tuned set scored 17 of 17. A held-out set, written before the tuning, scored 7 of 18 on labels alone. That gap is the whole point of having a held-out set. What a caller experiences on the same held-out set is 18 of 18 acceptable, with no harmful routing and no invented details, because a redirecting label needs the caller's own words to corroborate it before it is acted on. When later real calls showed the model labelling bare names as a request for a person, I ran a controlled evaluation of three prompt variants on 60 unseen sentences rather than patching phrases: every variant that fixed one class of error broke another, so the prompt stayed and the protection went into deterministic code.
Signed: the outage two days before launch
On 16 October 2026, a routine deploy reported that the receptionist would not start. The website and the background worker were healthy; the model server was not. The cause was a host security policy that had begun blocking the unsigned model-server binary. The file was byte-identical to the one that had run for weeks. Rolling back would not have helped, because every release used the same binary. For about two hours the phone webhook answered 502 and calls went to the fallback line. The line was still testers-only, so no customer was affected, and the model came back by itself once the policy's external verdict refreshed. But a system that only starts when a third party's cache is warm is not a system you launch on.
There were two ways out. Weakening the host's security policy to make the binary start would have traded security for convenience; I rejected it. The other was a signed runtime. The upstream release is signed and checksum-verified, and it bundles its own model server. So the receptionist now launches that signed server directly, with the same flags, the same model file, the same raw prompt and the same grammar; one environment variable changed. The upstream daemon was measured and not chosen: it can only constrain output with a JSON-string schema rather than GBNF, and on the same model 4 of 43 intents differed, classification was about 30% slower, and private memory was four to five times higher.
"Same model, same prompt" is a claim, so it was tested. A parity harness runs every check against every runtime through the receptionist's own client and leads with differences between runtimes before it reports accuracy: 43 intents (the held-out set plus that round's real-call phrases), 10 single-field extractions, and 6 scripted conversations of 27 turns through the real booking flow against the test database. Result on the host: zero differences and identical transcripts. The eight intent misses were the same on both runtimes, and all of them are phrases the deterministic code handles before the model is consulted.
The outage also changed what "healthy" means. The receptionist now reports itself ready only after its model has answered a real classification, at start and every three minutes between calls; before, "model up" meant a server handle existed. A dead runner is restarted by process id, and the deploy tooling ends in one of three verdicts: promoted, partial deployment (website up, receptionist not healthy, never called promoted whatever the test count), or failed. Both recovery paths were exercised by killing the processes: 1 minute 47 seconds to a working model in each case.
What the three properties have in common
Resident is about latency, constrained is about safety, and signed is about whether the thing starts at all. None of them is exotic, and none of them came from the model card. Each came from running the real component on the real host, measuring, and treating the result as the fact. A 1.7-billion-parameter model is a weak classifier with a strong tendency to improvise. It can carry a phone line only when the code around it decides what it may say, when it may be asked, and whether it is even allowed to run.