PeakClips Pipeline
Operated locally as a daemon (Jun–Sep 2026) · collaborative · private repository
Summary
A short-form video pipeline whose engineering was driven by measurement: a committed proof harness decides whether a faster path is allowed to ship, a delivery ledger makes crashes safe, and the operator surface is hardened like a public app. The business side (distribution and audience) did not succeed, and this page says so.
The problem and who it serves
Turning long source videos into publishable short clips by hand is slow, and a pipeline that does it automatically must be fast on 4K sources, recover from crashes without posting twice, and never let an unreviewed clip reach a platform.
The two operators (me and a collaborator) reviewing clips through a chat bot and a web dashboard; downstream, social-platform audiences that in practice were never reached at scale.
My role
Primary engineer: about 87% of non-merge commits, every merge to main after the early weeks, 75 pull requests with a PR-only flow. I built the pipeline core, the performance work and its proof harnesses, the recovery and delivery ledger, the dashboard and its authentication, the safety gates and the test suite. The collaborator built the platform publishers, the multi-platform router, the copyright heuristic meter and the split mode. Implementation was AI-assisted under my direction.
Architecture
Every state transition is persisted before the side effect it drives: cards are recorded as pending before the first send and as delivered after each one, so a crash mid-burst resumes in order and never duplicates. Dashboard actions are not a second code path; they are injected into the same gated action queue the bot uses.
Decisions and trade-offs
- Proxy analysis, proven by construction. Decoding 4K60 AV1 per span cost ~25 minutes per source. Options measured: pipe decode (2× slower than OpenCV), hardware AV1 decode (silent software fallback), YOLO on the GPU (not the bottleneck), a 480p proxy. The proxy won, with one rule: every proxy “dead” verdict is re-judged on the full-resolution file, so the fast path can only ever be as strict as the old one.
- Reject the 36× cheaper encoder. Local SSIM favoured the hardware encoder for final renders. I required post-upload proof: after platform transcoding, captions lost 0.005 SSIM against the software encoder. Rejected; the lesson is recorded in config and the safety policy.
- Reject the tracking proxy. A crop-drift harness with explicit pass bars failed on real footage and the gain was ~1.1×; the flag stays off and the code is dormant.
- One approval path. Dashboard writes go through the bot’s approval machinery (serial queue, dedupe, confirm-to-go-live); forged posts to held lanes are refused server-side.
- Fix the hang properly. A wedged worker blocked the gate pool forever on shutdown; the fix uses a background join with a hard deadline and kills orphans snapshotted at spawn, proven with a real stuck subprocess rather than a mock.
Security and reliability
- Dashboard auth: PBKDF2-HMAC-SHA256 at 240,000 iterations, salted, constant-time compare; signed HttpOnly 12-hour session cookie; per-session CSRF on every POST; escalating, mutex-guarded lockout; body-size cap; localhost-only binding.
- A dashboard XSS was found in a bug-hunt and pinned by a test that asserts every text interpolation is escaped (123 call sites).
- A platform-gating hole (prechecks guarded only one platform) was closed and tested across all short-form lanes.
- Recovery exercised for real: four daemon boots requeued an interrupted job after a sleep or restart.
Verified results
| Measure | Result | Evidence |
|---|---|---|
| Dead-air gate, one 24-min 4K60 source, 31 windows | 590.1 s → 17 s (16.7 s measured); 30/31 identical verdicts, 0 missed drops, 1 borderline window escalated | tools/diff_gate.py output, 2026-07-05; harness committed in PR #38 |
| Encoder decision | hardware encoder rejected: captions −0.005 SSIM after platform transcoding | pipeline config lines 618–626, commit b4a95ed, 2026-07-06 |
| Tests | ~80+ test files for the pipeline (132 repo-wide with a sibling product); recovery, idempotency, dashboard-auth and gate suites | Repository; no CI (local runner) |
| Operation | 364 clips produced 2026-06-21 to 2026-09-04; daemon under launchd | State database, read-only |
| Process | 283 commits, 75 pull requests, owner-only merges | Git history |
Limitations and remaining work
- No CI; tests run through a local runner. The 590 → 17 s result is for one gate on one source class; it was not proven on a second source class.
- As a business the project did not reach an audience: the main distribution channel was terminated by the platform, and no verified live audience exists on the surviving accounts. The copyright precheck is operator-overridden in production by a deliberate product decision.
- The repository is private and shared with a collaborator, so figures here are from the audit of 2026-10-21 rather than public links.