Harness Engineering: Agent Loops, State, Recovery and Evals

An agent submits a form, the server saves it, and the response disappears in a network timeout. Should the agent submit again, report failure or declare success? None of those choices is justified until it finds out what happened. Harness engineering provides the state, tools and recovery rules that let it make that decision—and the evidence required to call the task complete.

We will follow a staging-form change through execution, a crash, recovery and verification. The architecture and code blocks are illustrative designs, not tested SDK recipes. After the mechanics, the engineering examples show which patterns other teams have used and where their findings may stop applying.

1. Define the boundaries before choosing a framework

Separate five responsibilities:

1. Model: proposes text or tool calls from the context it receives.
2. Harness: validates those proposals, routes calls, selects context and manages progression.
3. Execution environment: runs commands, opens browsers and holds working files.
4. Durable storage: records task state, attempts, artifacts and external operation identifiers.
5. Application policy: decides identities, permissions, spending and which actions need approval.

LangChain's anatomy article offers a broad definition covering prompts, tools, infrastructure and orchestration. For implementation, drawing the boundaries more explicitly helps answer an operational question: which component remains authoritative when another component crashes?

For a local code edit, the working tree and version history may be sufficient recovery artifacts. For a customer-support agent that sends refunds, a conversation summary is insufficient: you need the payment system's transaction record and a durable operation identifier.

Choose infrastructure based on the failure you must survive:

Framework choice does not determine whether an email was sent twice. The tool contract and recovery procedure do.

2. Make the runtime a state machine

An application should distinguish running, waiting_for_tool, waiting_for_approval, verifying, completed, failed and paused_budget. Otherwise every interruption looks like an agent that has merely stopped talking.

A model's request to finish should be a candidate completion. The harness checks required evidence before marking the task complete. A timeout is a tool event; it should not silently become a successful result or an invitation to repeat an action forever.

Consider this original example: add a required project-budget field to a website enquiry form, demonstrate it in staging, and leave the release for review. A task record could look like this:

{
  "task_id": "budget-field-042",
  "status": "running",
  "workspace_revision": "base-commit-id",
  "scope": ["form UI", "request validation", "staging persistence"],
  "forbidden_actions": ["production deploy", "email real customers"],
  "acceptance": ["empty budget rejected", "budget stored", "old fields preserved"],
  "artifacts": [],
  "pending_operation": null,
  "limits": {"tool_calls": 40, "wall_minutes": 20, "repair_rounds": 2},
  "usage": {"tool_calls": 0, "repair_rounds": 0},
  "next_action": "inspect submission handler"
}

The numbers are example policy settings, not recommended universal limits. Store the model identifier, prompt version, tool-schema version and dependency lockfile alongside the run so a later failure can be compared with an earlier success.

The controller below makes completion a checked state transition. Its single-writer lease gives one worker temporary authority to update the task; a production implementation must renew that lease and prevent an expired worker from continuing to write.

load durable task and acquire a single-writer lease
reconcile any recorded operation whose outcome is unknown
while task is runnable:
    enforce deadline, usage allowance and cancellation
    proposal = model(context assembled from task + recent evidence)
    if proposal requests completion:
        run acceptance checks against current artifacts
        finish only if all required checks pass
        otherwise return the concrete failure or stop at repair limit
    else:
        validate tool input and application permissions
        persist operation intent before dispatch
        execute with a bounded timeout
        persist observed result and update task checkpoint
release lease

This sketch omits transaction and concurrency implementation. Recording an intent before dispatch lets a replacement worker identify the operation to investigate. The remote service can still complete the operation before the local result is saved, which is why the recovery path must handle an unknown outcome.

3. Treat tool contracts as product interfaces

A useful tool describes input, output, authorization, failure and retry semantics. A tool named do_task returning a paragraph hides too much. Prefer an operation such as submit_staging_enquiry with validated fields and a bounded result.

Example result envelope:

{
  "operation_id": "budget-field-042-submit-1",
  "status": "outcome_unknown",
  "resource_id": null,
  "retry_policy": "reconcile_before_retry",
  "error_code": "RESPONSE_TIMEOUT",
  "evidence_ref": "events/tool-call-18.json"
}

These are suggested application fields, not an SDK contract. Here, outcome_unknown means the request may have succeeded: the next action is to inspect its result, not submit a fresh request. The error code lets you group similar failures; the evidence reference preserves the full log without putting it in every model prompt.

Different failures need different responses:

AWS's idempotency explanation establishes why a missing response can leave a caller uncertain and why a caller-provided request identifier can help. For our form example, preserve the identifier across retries and have the receiving service reject a reused key with different input. A local flag alone cannot prevent a duplicate accepted by a remote service. Respect the provider's retention period for deduplication keys.

4. Keep four kinds of memory separate

LangGraph's persistence documentation distinguishes thread checkpoints from cross-thread stores. It also warns that an in-memory checkpointer loses state at process restart. “Has memory” is therefore an incomplete claim: ask what is stored, where, for how long, and under which identity.

For our staging task, checkpoint after recording an operation result and before moving to the next phase. A checkpoint containing only “submission successful” is weak: save the staging record identifier and the artifact revision the test exercised. If a teammate changes that revision, the previous test is no longer evidence for the current deliverable.

Do not promote every agent conclusion into shared memory. Record provenance and expiry for operational facts. “The service uses port 3000” may stop being true tomorrow; a fresh startup result should override an old note. Keep secrets and customer payloads out of reusable project lessons.

5. Assemble context instead of accumulating everything

Durable storage tells the system what survives. Context assembly decides which part of that record the model needs now. Filling the prompt with entire logs makes constraints harder to find and leaves less room for the next tool result.

For each turn, include: the current goal and non-negotiable constraints; the authoritative task state; the relevant source or code slice; the latest observation; and references to older evidence. Keep verbose build output as an artifact, returning a summary plus an explicit indication that more content exists.

Three strategies have different tradeoffs:

For a handoff, preserve exact acceptance criteria, changed files, completed checks, unresolved errors, pending external operations and the next action. Separately label observations and hypotheses: “request returned 504” is observed; “database rejected it” is an unproven explanation.

Exercise compaction deliberately in evaluation. Place an important constraint early, produce enough tool output to trigger the policy, and see whether the agent still honors it afterward. A long-context marketing number does not answer this workload-specific question.

6. A complete failure and recovery walkthrough

The pieces now meet in one designed test: the form is changed correctly, but the response to its valid submission is lost. The harness must recover without creating a second enquiry.

1. Establish baseline. Start the staging application and submit an existing valid fixture. Record its stored fields. If baseline fails, distinguish the pre-existing failure from the requested change.
2. Implement. Add UI options, server-side validation and storage mapping. A frontend-only required attribute cannot establish that the server rejects a direct invalid request.
3. Verify rejection. Send an empty-budget request directly to staging. Check that it is rejected and creates no record.
4. Dispatch valid submission. Record operation budget-field-042-submit-1, then send the request. The service commits the record, but the network drops the response.
5. Crash and resume. The worker dies before observing a result. The replacement reads the pending operation. Its status is unknown, not failed.
6. Reconcile. Query staging by the operation identifier. If the record exists, validate its values and record its ID. If the service authoritatively confirms absence, retry using the same supported deduplication key. If neither can be established, pause this branch for investigation.
7. Validate persistence. Read the stored record through the service and confirm the selected budget and old fields survived. Check that exactly one record was created for this operation.
8. Validate user experience. Exercise missing-field handling and a valid submission in the browser, including narrow-screen layout and keyboard access.
9. Review. Attach the diff, revision, checks and staging evidence. Mark the implementation complete while accurately stating that production deployment remains unperformed.

This walkthrough exposes two independent defects: UI success without persistence, and replay that creates duplicates. Adding more reasoning tokens will not repair an absent reconciliation interface. You must provide the tool or change the task's permissible behavior.

7. Add another agent only for a clear boundary

A separate evaluator can inspect an implementation without sharing the author's entire conversation. A research worker can gather evidence while the implementer works on an unrelated module. Two agents editing the same file rarely provide that independence.

Give each worker an output contract, ownership boundary and time allowance. The coordinator must verify integration, not concatenate optimistic reports. An evaluator should not silently rewrite the same artifacts it is grading. When it proposes a fix, that creates a new candidate revision that needs rechecking.

8. Grade real outcomes and measure consistency

Anthropic's evaluation guide distinguishes code, model and human graders, and contrasts finding one success across attempts with succeeding consistently. Use deterministic checks for observable state and calibrated human or model judgment for qualities that cannot be reduced to exact values.

For our example, the completion gate checks stored budget, preserved fields, no unauthorized deployment, and no duplicate operation. A model judge may flag confusing error copy; it cannot overrule a failed persistence test. Keep required tests outside the agent's editable area or validate their integrity before grading.

Report repeated trials honestly. pass@k asks whether at least one of k attempts succeeds. pass^k asks whether all k attempts succeed. Under an illustrative independent, equal-probability assumption, a system with 80% per-run success has 99.2% probability of at least one success in three tries, but only 51.2% probability all three succeed. Real trials can share infrastructure failures, so inspect correlations rather than assuming this model fits.

Track cost per accepted task, latency, human interventions and critical failures alongside success rate. A change that improves easy cases while introducing occasional unauthorized writes is unacceptable even if its average score rises.

What recent engineering examples actually teach

These examples solve different problems. Read them as evidence for particular design decisions, rather than a single universal architecture.

The change across these reports matters. Anthropic's March article says the context resets needed in its earlier Sonnet 4.5 setup were unnecessary with Opus 4.5. Copying an older workaround into every new agent can increase latency without helping. A useful harness has removable components and an evaluation that tells you when to remove them.

9. Improve the harness without teaching to the test

The public better-harness README describes an outer agent proposing changes to editable surfaces, with train, holdout and optional final scorecard sets. It explicitly calls the visible/private separation research infrastructure rather than a hard isolation boundary.

Reading the worked configuration reveals a useful practical detail: middleware implementation and registration are separate editable surfaces. Changing an unregistered hook has no effect. Its sample reporting tool returns a string; that is not evidence that a real message was delivered. The core implementation evaluates candidate variants and records acceptance decisions. Study these mechanisms before treating the example as production infrastructure.

For your own experiment, freeze the model, task set and budget while changing one component. Compare baseline against a pre-completion verifier, then test verifier plus context changes. Also remove components: an ablation can show that a long instruction block contributes no benefit.

Use tuning examples for iteration, a validation set for selecting candidates, and an untouched final test set for reporting. A “holdout” consulted repeatedly during selection becomes part of that selection process. Refresh it or reserve another independent set. Preserve a known-good version and check formerly passing tasks after every candidate improvement.

10. Enforce limits outside the prompt

Set wall time, tool-call, spend and recursion limits in the controller. Aggregate child-agent usage under the same parent allowance. Reserve enough time for verification; when the allowance expires, save artifacts and report incomplete work rather than manufacture a pass.

Run generated code in a constrained environment, with only the files, network destinations and credentials required for the task. Treat repository documents, retrieved pages and tool output as data, not authority to change permissions. Authorization should be checked by the application on every consequential call, even when a trusted model selected it.

A strong first implementation is modest: one task record, a few typed tools, durable pending-operation tracking, a reproducible test environment and outcome checks. Add an evaluator or optimizer when traces identify a failure those components can resolve. For retrieval-heavy agents, pair this with the end-to-end RAG workflow: evidence selection is one part of the harness, while action recovery and completion remain separate responsibilities.

Continue with Codex workflow tips, Claude Code workflow tips, AI output evaluation, and Microsoft SkillOpt. A skill revision improves one reusable instruction surface; the surrounding runtime still needs truthful state, bounded actions and a reliable completion gate.