Wire validator conversation history and surface validationOutcome in TaskExecutionResult

[lmorchard]

Current state

Two related issues in the validation phase (webAgent.ts:1205-1340):

1. conversationHistory is built but unused

validateTaskCompletion computes a 30-message stringified conversation history via formatConversationHistory (webAgent.ts:1345-1369) and passes it to buildTaskValidationPrompt(task, successCriteria, finalAnswer, conversationHistory) (webAgent.ts:1227-1235).

But the template at prompts.ts:562-593 has no {{ conversationHistory }} placeholder anywhere:

Evaluate if the task result gives the user what they requested.
Be concise in your response.

Today's Date: {{ currentDate }}
Task: {{ task }}
Success Criteria: {{ successCriteria }}
Result: {{ finalAnswer }}
...

So the helper builds the string and it's silently dropped. The validator only sees the final answer in isolation — it cannot tell whether a partial result was "the agent gave up early" or "the site genuinely didn't have more data."

2. Force-accept reports success: true indistinguishably from real acceptance

When validationAttempts >= maxValidationAttempts (default 3), validateTaskCompletion force-accepts the answer (webAgent.ts:1301-1309):

if (executionState.validationAttempts >= this.maxValidationAttempts) {
  this.logger.warn(
    `Accepting answer after ${this.maxValidationAttempts} validation attempts...`,
  );
  this.eventEmitter.emit(WebAgentEventType.AGENT_STATUS, {
    iterationId: this.currentIterationId,
    message: `Accepting answer after ${this.maxValidationAttempts} validation attempts...`,
  });
  return { isAccepted: true, ... };
}

The caller receives TaskExecutionResult { success: true, finalAnswer: ... } regardless of whether the validator approved on attempt 1 or threw "failed" three times and was overridden. Only an AGENT_STATUS event reveals the difference, and downstream consumers (CLI output, server response, telemetry) don't surface it.

The gap

Validator without context: a "looks fine in isolation but agent gave up early" failure mode is invisible to the current validator. With conversation context, the validator could spot when the agent's last few actions show it was hitting walls (errors, blocked navigation) rather than genuinely completing the task.

Caller cannot distinguish real success from force-accepted weak result: this matters for eval-judge reporting (a task graded as "passed" by Pilo's own validator should be distinguishable from one Pilo force-accepted after retries), and for telemetry (the force-accept rate is a measurable quality signal).

Proposed scope

Decision 1: wire conversationHistory into the template, OR delete the dead code

Two reasonable choices — recommend one, sketch both:

Option A (recommended): wire it up. Add {{ conversationHistory }} to the validation template at prompts.ts:562-593. Adjust the template to instruct the validator to consider the trajectory, not just the final answer.

Evaluate if the task result gives the user what they requested.
Be concise in your response.

Today's Date: {{ currentDate }}
Task: {{ task }}
Success Criteria: {{ successCriteria }}
Result: {{ finalAnswer }}

Recent agent actions and observations:
{{ conversationHistory }}

Evaluation approach:
1. Compare the result against the success criteria defined above
2. Check if all required information is included
3. Verify the answer meets the specified format and detail level
4. Review the recent agent trajectory:
   - Was the agent making genuine progress, or repeating failed attempts?
   - Did the agent run into blockers (errors, walls) that the final answer doesn't acknowledge?
   - Did the agent claim success but actually skip required steps?
...

Option B: delete the dead code. If the design intent is "validator should be a stateless score-the-deliverable check, not influenced by agent reasoning," delete formatConversationHistory (webAgent.ts:1345-1369) and the conversationHistory param.

Recommend A — the helper exists and the gap (validator without context) is real. B is acceptable if there's a deliberate design reason for keeping it isolated; current state of "build it but drop it" is just a bug.

Decision 2: surface validationOutcome in TaskExecutionResult

Extend the type (webAgent.ts:128-143):

export interface TaskExecutionResult {
  success: boolean;
  finalAnswer: string | null;
  validationOutcome?: "accepted" | "force-accepted" | "rejected" | "skipped";
  error?: TaskError;
  stats: { ... };
}

• "accepted" — validator returned complete or excellent on some attempt
• "force-accepted" — maxValidationAttempts exhausted, answer surfaced anyway
• "rejected" — would be reached only if a future change makes rejection terminal (today never happens)
• "skipped" — validation didn't run (e.g., task failed before done())

Update CLI / server / SSE telemetry to surface this where it makes sense. Eval-judge consumption (the parent project) gets a real signal for "the agent itself thinks this answer is shaky."

Implementation notes

• The conversation history is currently this.messages.slice(-30). Consider whether 30 is right — too few and you miss the early task setup; too many and the validator gets distracted by old snapshots. 20 with the very first user message (the task+plan) appended is a reasonable shape.
• Watch for prompt injection: formatConversationHistory stringifies whatever's in messages. Page snapshots inside have already been clipped via truncateOldExternalContent, but the validator should still wrap the rendered history in an external-content marker.
• Tests: extend task validation describe block in webAgent.test.ts:2017+ to cover the new validationOutcome field on the result object across the four states.

Acceptance criteria

• Either: conversationHistory is wired into the validator template AND has at least one test asserting the validator considers trajectory; OR the dead code is removed and a comment explains the design decision.
• TaskExecutionResult exposes validationOutcome with the four values above.
• The CLI output (packages/cli/) and JSON logger surface validationOutcome where they currently surface success.
• Tests cover all four validationOutcome paths.

Effort estimate

4-6 hours. Half is the validator wiring; half is the result-type plumbing through CLI/server/extension consumers.

Related issues

Independent of other Tier 1 work, but the pre-done verification checklist issue reduces the rate at which force-accept fires (the validator rejects less often), so the two are complementary.

Files likely affected

• packages/core/src/webAgent.ts (TaskExecutionResult interface, validateTaskCompletion, formatConversationHistory)
• packages/core/src/prompts.ts (taskValidationTemplate)
• packages/cli/ (any output that displays success)
• packages/server/ (response shape)
• packages/core/test/webAgent.test.ts (validation test block)