Technical Architecture, Operator Experience, and Second-Order Effects
Process Factory is a config-driven, block-based DAG execution engine designed to run multi-step pipelines with dependency resolution, human gates, threshold checks, and LLM-powered block execution. Each pipeline is defined as a JSON template of typed blocks. The engine evaluates readiness by inspecting block states and dependency graphs — a stateless evaluation that requires no persistent memory beyond the run's current state stored in a single Supabase JSONB column.
After every block execution, the engine runs a 5-dimension structural audit (D1–D5) that scores completeness, specificity, downstream readiness, human auditability, and second-order effects. This audit is a pure function: no LLM calls, no network requests, sub-100ms execution time. It operates on heuristics — artifact counts against golden examples, generic phrase detection across 16 patterns, JSON structural validation, placeholder scanning, and upstream artifact consumption tracing.
The key architectural decision: the audit loop is infrastructure, not process. It runs automatically, stores results in block state, generates prioritized fix suggestions, computes quality deltas across runs, and flags score inflation. Quality improvement happens as a side effect of execution — every block run makes the next one measurably better or surfaces exactly why it did not.
Six components. Stateless scheduler. Atomic state persistence.
Each dimension checks a distinct failure mode. Weights: D1 (25%), D2 (25%), D3 (20%), D4 (10%), D5 (20%).
The core innovation: every execution produces audit data, every audit produces fix suggestions, every fix is measurable, and every measurement feeds the next execution.
What makes it recursive: Auto-audit runs after every execution. Fix suggestions reference specific artifacts and JSON paths, not generic advice. Delta tracking means improvement is measurable to one decimal place. The calibration_warning flag fires when the LLM's self-assessment diverges from structural reality.
3-panel workbench. Every decision point surfaces the data needed to decide.
Pipeline View: 23 blocks in 10 phases, color-coded by status. SummaryBar shows status counts + run-level quality composite + weakest dimension + calibration warnings. Auto-advance runs non-gate blocks automatically.
| Panel | Contents |
|---|---|
| Left: Guide | Block summary, instructions, quality checklist |
| Center: Inputs | Form fields, file upload, upstream artifacts |
| Center: Outputs | Artifacts, audit badge (A:7.2), D1-D5 panel with progress bars, issues, suggestions, deltas |
| Center: Logic | Model, timeout, threshold, gate, retry config |
| Right: Chat | Block-scoped advisory + factory-level META chat |
| Right: Gaps | Artifact-aware gap reconciliation with LLM evaluation |
| Right: QA | Quality gate: approve/reject with feedback |
Post-Run: Reflection prompt with 4 questions + mic input (Web Speech API). Stored as Process Memory.
The factory audits itself as a factory run.
meta-audit-factory.json defines a 5-block pipeline:
| Block | Function |
|---|---|
run-aggregate | Compute per-block and run-level D1-D5 composites. Threshold: 5.0 minimum. |
specificity-deep-check | Expert name density, generic phrase ratio, content depth analysis. |
cross-block-consistency | Verify artifact references resolve. Detect orphaned dependencies. |
delta-report | Compare scores against previous run. Per-dimension delta tables. |
audit-scorecard-publisher | Compile into published HTML scorecard. Human gate before publish. |
The meta-audit run's own 5 blocks are auto-audited by the same D1-D5 system. The audit of the audit is recursive.
Real data from the Samuel Ngu expert clone pipeline — 23 blocks, 10 phases.
Audit v1 (baseline)
Fix Cycle: frameworks.json 17% → 100% prompt templates. Resources rewritten. L2 extractors wired to proto-* artifacts.
Audit v2 (post-fix)
Session-Audit (Engine Wiring)
| Pass | Fields | Wired | Orphans |
|---|---|---|---|
| Initial | 13 | 12 | 1 |
| Interval 1 | 28 | 25 | 3 → fixed |
| Interval 2 | 28 | 28 | 0 |
L0 Research: $0.40–$1.30/expert. After proto-* wiring, investment compounds into L2 extraction. D5 drops 4 points if L2 blocks don't reference upstream keys.
Immediate effect → compounding second-order effect.