AI Harness Engineering: Making Agent-Driven Delivery Deterministic

Enterprise teams are shipping AI agents into real codebases faster than their delivery infrastructure can keep up. The pitch is seductive: point a capable model at a legacy system, let it refactor, migrate, and document — and watch months of work compress into days. The reality on the ground is different. Production systems do not tolerate "usually works." They require deterministic outcomes: the same inputs produce the same verified outputs, every time, under audit and rollback constraints that no demo environment ever simulates.

That gap — between probabilistic model output and deterministic production requirements — is where most AI-agent programs stall. Not because the models are too weak, but because the harness around them is missing or under-designed.

The problem

Large language models are probabilistic by construction. Given the same prompt twice, you may get syntactically different but semantically equivalent answers — or you may get a subtly broken migration script that passes a linter but corrupts data at runtime. In a toy repository, that variance is an annoyance. In a platform running fifty ETL pipelines and fourteen production React applications for an investment services client, it is a regression event waiting for a Friday deploy.

The failure modes I see most often in production agent work are not model-quality failures. They trace back to three structural problems:

Context drift. An agent starts a multi-hour remediation session with a clear picture of the codebase, then gradually loses alignment as conversation history grows, intermediate artifacts pile up, and earlier constraints stop being referenced. The model does not forget on purpose — it deprioritizes. Without explicit state management, the agent's understanding of "what we already decided" diverges from what the team decided three checkpoints ago.

Schema misalignment. Agents produce output in shapes the downstream pipeline cannot consume: frontmatter fields that fail validation, migration scripts that assume directory layouts that changed two sprints ago, test files that import modules that were renamed in a parallel workstream. The model often "looks right" to a human skimming diffs. Automated validation catches what human review misses at scale.

State degradation. Long-running agent sessions accumulate partial work — half-migrated files, stub implementations marked TODO, configuration changes that work in isolation but conflict when merged. Without checkpointing and rollback paths, the team discovers degradation only when integration tests fail at the end of a sprint, with no clean revert point.

These are harness problems, not intelligence problems. Better models reduce the frequency of bad output, but they do not eliminate the need for verification infrastructure. A stronger model with a weak harness still ships regressions — it just ships them with more confidence.

What a harness actually is

In agent-driven delivery, I use a simple decomposition:

Agent = Model + Harness

The model generates candidates: code, migrations, documentation, test cases. The harness decides what becomes real: what passes validation, what gets merged, what gets rolled back, and what requires a human checkpoint before proceeding.

A harness is not a wrapper library or a single CLI tool. It is the full infrastructure around the model that makes outcomes trustworthy:

Verification gates are explicit checkpoints where output must satisfy machine-checkable criteria before advancing. A gate might require: all modified files pass TypeScript compilation, frontmatter validates against a zod schema, E2E tests pass on a staging branch, or a human approves a diff above a certain size threshold. Gates are sequential by design — you do not skip from "agent produced code" to "merged to main" without intermediate proof.

Schema constraints define the shape of acceptable artifacts before an agent ever touches them. If your content pipeline expects MDX with specific frontmatter fields, the harness validates against that schema at write time, not at deploy time. If your migration scripts must target a specific directory layout, the harness encodes that layout as a constraint the agent cannot override without an explicit exception. Schema alignment prevents the class of bugs where output looks plausible but fails silently downstream.

Test coverage as a safety net is not optional in harness design — it is the mechanism that lets you trust automated remediation at scale. The goal is not 100% coverage on day one. The goal is coverage on the paths that break when agents refactor: critical user journeys, data transformation boundaries, integration points between services. Tests are the rollback signal. When an agent's change breaks a covered path, the harness blocks merge and surfaces the failure immediately, not after QA discovers it manually.

Rollback paths assume failure is normal, not exceptional. Every automated remediation step should have a revert strategy: git revert, feature flag disable, staged rollout with automatic rollback on error rate spikes. Agents that cannot be rolled back are agents you cannot run in production, regardless of how impressive their output looks in isolation.

The harness does not replace engineering judgment. It externalizes the judgment into repeatable, auditable steps so that judgment scales beyond what one person can hold in their head during a twelve-hour agent session.

Case study as harness design

The clearest proof point in my recent work is the AI-powered pipeline and frontend modernization for an investment services platform: fifty ETL Python pipelines on Windows VMs with Apache Airflow, plus fourteen React applications on an aging stack. The business goal was straightforward — modernize execution environment, adopt current React patterns, cut pipeline runtime, reduce manual remediation. The delivery challenge was harder: use AI agents to accelerate remediation without turning a modernization program into a regression factory.

We did not start by asking agents to rewrite code. We started by designing the harness.

Exploration before execution. The first phase used AI to map the codebase: dependency graphs, failure patterns in existing pipelines, test gaps in frontend applications. That exploration output fed structured context documents — not free-form chat history — that subsequent agent sessions loaded as ground truth. Context drift was reduced because each session started from the same validated snapshot, not from whatever the model remembered from yesterday.

Incremental remediation with validation steps. Rather than "migrate all fifty pipelines," we defined remediation in bounded units: one pipeline family, one application cluster, one migration pattern at a time. Each unit had a validation sequence: run existing tests, apply agent-generated changes, re-run tests, compare pipeline execution metrics against baseline, human review of diffs above a size threshold. An agent could not advance to the next unit until the current unit passed all gates. This is slow compared to a demo where an agent rewrites everything overnight. It is fast compared to a manual migration that discovers regressions in production.

Automated remediation with human checkpoints. We progressively automated the remediation loop — reaching roughly 80% automated remediation across the scope — but automation never meant unattended merge. Human checkpoints were explicit: schema validation failures blocked automatically; test failures blocked automatically; large structural changes required human approval before merge even when tests passed. The harness made the default safe. Opting into risk required an explicit human decision, not an accidental omission.

Test coverage as the trust mechanism. Key user journeys in the fourteen React applications gained Playwright coverage before large-scale agent refactors. Pipeline migrations included execution-time benchmarks and data integrity checks. When an agent's change broke a covered path, merge blocked immediately. Near-zero regressions across fourteen applications during the migration was not luck — it was the harness doing its job. The 20% of remediation that still required manual intervention was precisely the 20% where automated validation could not yet provide sufficient confidence.

Infrastructure migration as a staged rollout. Moving pipelines from Windows VM/Airflow to OpenShift (Kubernetes) and adopting React Compiler across applications happened in stages with rollback paths at each stage. Agents generated migration scripts and configuration changes; the harness enforced staging order, smoke tests between stages, and revert procedures if metrics degraded.

The outcome — 80% automated remediation, pipeline execution time cut in half, near-zero regressions, full stack modernization — is often attributed to "using AI well." That attribution misses the point. The outcome is attributable to harness design that made AI output safe to act on at production scale.

Reusable principles

These principles transfer beyond any specific toolchain or model provider. They are durable because they address the structural gap between probabilistic generation and deterministic delivery.

• Design the harness before you scale the agent. Start with verification gates, schema constraints, and rollback paths. Then increase agent autonomy within those boundaries. An agent without a harness is a demo. An agent with a harness is a delivery mechanism.

• Treat context as managed state, not conversation history. Externalize decisions, constraints, and codebase snapshots into structured artifacts that every agent session loads explicitly. Context drift is the silent killer of long-running agent work; structured state is the antidote.

• Make validation sequential and blocking. Do not parallelize "generate" and "ship." Every artifact passes machine-checkable gates before merge. Human review is a gate too — reserve it for changes where automated validation is insufficient, not as a substitute for automated validation on changes where it is sufficient.

• Invest in test coverage on the paths agents will touch. Agents refactor confidently in covered code and dangerously in uncovered code. Coverage is not a quality metric for its own sake — it is the rollback signal that lets you trust automated remediation at scale.

• Assume failure and design rollback first. If you cannot revert an agent's change cleanly, you cannot run that agent in production. Rollback paths are not pessimism — they are the prerequisite for optimism about automation rates.

What I'm exploring next

The harness patterns from the pipeline modernization project are general, but the implementation is still manual in places: checkpoint definitions live in runbooks, schema constraints are enforced at build time but not always at agent write time, and human approval gates depend on engineer attention rather than policy engines.

What I am exploring now:

Tighter integration between agent sessions and schema validation at write time — so an agent cannot produce an artifact that fails frontmatter or API schema checks, rather than discovering failure at prebuild or CI.

Policy-as-code for human checkpoints — explicit rules for when human approval is required (diff size, file paths, blast radius) rather than ad hoc judgment call-by-call.

Multi-agent orchestration with shared state stores — separate agents for exploration, implementation, and verification, coordinated through a harness that prevents any single agent from bypassing gates another agent enforces.

Extracting a minimal, public reference implementation — a small demo repository showing one concrete verification-gate pattern (schema validation before merge, automated rollback on test failure) so the ideas in this essay are reproducible, not just described.

The term "AI harness engineering" entered wide use in early 2026 because teams hit the same wall: models got good enough to be useful in production, but delivery infrastructure lagged. The next phase is not better models alone — it is better harnesses that make model output safe to ship. That is the work I am doing, and the work I think more teams need to prioritize before they scale agent-driven delivery across their critical systems.