Production & Agent Infrastructure

Summary

Three systems I run, built AI-assisted under my direction and recorded with their evidence: the production deployment of an AI receptionist (a gated and staged release path with rollback archives, heartbeat monitoring that alerts on silence, restore-tested backups, automatic recovery of crashed processes); a scoped long-term memory system for the coding agent (an MCP gateway where scope is a required parameter, re-indexed on any edit by a launchd watcher, health-checked nightly); and guard rails on the agent’s shell (a PreToolUse hook with a regression suite). Each rule in the release path traces to an incident that happened first. Operational details of the client’s environment are deliberately left out of this page.

The problem and who it serves

A real client’s phone line and booking site had to stay up unattended, ship releases that cannot be a failing build, tell a human when something silently stops, and survive an unplanned restart. The coding agent I build with needed the same discipline: memory that cannot leak across domains, and shell guard rails that do not depend on a rule being remembered.

Me as the operator; the receptionist’s client and their customers, who depend on the service staying up; and my own agent sessions, which depend on the memory and guard systems. The client is unnamed here pending consent for public use.

My role

Owner and operator. I hold the access, approve every deploy and account action, made the recorded engineering trade-offs, performed the console steps and received the drill notifications that count as proof. Implementation was AI-assisted (Claude Code) under that direction: the scripts, gateway, watcher, hook and runbooks were generated to my specification and reviewed by me, and the agent did much of the day-to-day reading of system state and ran the gated deploys with my approval. The record marks which readings were the agent’s and which were mine.

Architecture

The release directory is replaced wholesale on deploy and the data directory is never touched by a deploy or a rollback; the installer refuses to run if the two overlap, because an earlier plan had put the database inside the application root. Hostnames, providers, schedules and thresholds are intentionally omitted from the diagram.

Decisions and trade-offs

• Gate, stage, prove, keep the previous release. Every rule follows an incident: a deploy once went live with a failing test because the last command in a pipe succeeded (now the suite’s own “fail 0” line is required); a restart once stopped nothing because it matched the wrong path (now one fresh process per service must report the release commit); a stale tree once rolled a release back silently (now the provisioner refuses a downgrade unless told). A deployment with a healthy website but an unhealthy receptionist exits with a failure code even though tests passed.
• Monitoring that survives the machine. A monitor on the machine cannot report that the machine is off, and an external service calling in would need a public address. So the machine calls out on a schedule and silence is the alarm. The drill record matters: monitoring was described as “prepared, not working” until a deliberately missed heartbeat produced a real alert.
• Recovery by repeat trigger, not restart-on-failure. Killing the worker on 2026-09-27 showed that the scheduler’s built-in restart does not cover a process that dies under a running task. Each service task now also re-fires on a short schedule.
• Scope as a schema parameter, not a rule. The memory server with an optional filter was removed. The gateway requires a domain, queries each in-scope domain separately so out-of-scope notes are never read into the process, never adds the restricted domain implicitly, and never widens an empty result. Trade-off: there is no “search everything”.
• Guard rails must never block legitimate work. A guard that blocks real work when confused is worse than none, so every rule is narrow and independently tested, reads are never refused, and heredoc bodies are stripped so writing about a rule does not trip it. The suite has 16 cases.

Verified results

Rows citing a round record, launch record, monitoring setup record or runbook are documented, not verified: read from the project’s own records on 2026-10-24, not re-run for this page. The memory and guard suites were run on 2026-10-24.

Limitations and remaining work

• No uptime figure over a defined window; a 90-day availability record is the next piece of evidence worth collecting.
• Recovery, monitoring and backup evidence comes from the project’s own records and my own drills, not from an independent audit, and is documented rather than re-run for this page.
• Memory suite: 2 of 18 ranking expectations fail today; every scope and leak check passes.
• Operational details of the client’s environment are deliberately not published here, and the page does not describe its current setup.