Agent Test - Implementation Notes

Overview

tdx agent test provides automated testing of LLM agents using YAML test definitions. Tests are run against agents, and responses are evaluated by a judge agent (Claude Sonnet 4.5).

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        tdx agent test                           │
├─────────────────────────────────────────────────────────────────┤
│  1. Parse test.yml                                              │
│  2. Resolve agent (project/agent from folder structure)         │
│  3. Setup evaluator agent (create/update in tdx_default)        │
│  4. Run all test rounds (send messages, wait for completion)    │
│  5. Fetch chat history once after all rounds complete           │
│  6. Extract responses for each round from history               │
│  7. Evaluate each round against criteria using judge agent      │
│  8. Output results (JSON/JSONL/table/TSV to stdout or file)     │
└─────────────────────────────────────────────────────────────────┘

Key Components

Test Execution (src/sdk/agent/test/executor.ts)

Execution flow:

1. Run rounds sequentially - send user input, consume stream to wait for completion
2. After ALL rounds complete, fetch chat history once via llm.history(chatId)
3. Parse history to extract responses for each round

History structure:

{
  "data": [
    { "input": "user message 1", "at": "..." },
    { "content": "assistant response", "at": "..." },
    { "tool": { "functionName": "...", "content": "..." }, "at": "..." },
    { "content": "more response", "at": "..." },
    { "input": "user message 2", "at": "..." },
    { "content": "response to message 2", "at": "..." }
  ]
}

Each input event marks a new round. Everything between one input and the next belongs to that round's response.

Evaluation (src/sdk/agent/test/evaluator.ts)

Evaluator agent:

• Auto-created in user's default project (tdx_default_<username>)
• Model: Claude Sonnet 4.5
• Version tracked via SHA-256 hash of config (auto-updates when prompt changes)

Evaluation process:

1. For each round, pass conversation history + criteria to evaluator
2. Evaluator returns: { pass: boolean, reason: string, plan: string, thinking: string }
3. Test fails if any round fails evaluation

Output (src/sdk/agent/test/output.ts)

Formats results as JSON, JSONL, TSV, or table for stdout or file output.

Test File Format

Two formats are supported:

Flat Format (Recommended for Single-Round Tests)

tests:
  - name: greeting_test
    user_input: What's the weather today?
    criteria: Should respond with weather information

  - name: calculation_test
    user_input: What is 2 + 2?
    criteria: Should respond with 4

Rounds Format (For Multi-Round Tests)

tests:
  - name: multi_turn_conversation
    rounds:
      - user_input: |
          First message to agent
        criteria: |
          Agent must do X and Y
      - user_input: |
          Follow-up message
        criteria: |
          Agent must remember context and do Z

Mixed Format

You can mix both formats in the same file:

tests:
  # Simple single-round test (flat format)
  - name: simple_query
    user_input: Hello
    criteria: Should greet back

  # Complex multi-round test (rounds format)
  - name: context_memory
    rounds:
      - user_input: My name is Alice
        criteria: Should acknowledge the name
      - user_input: What's my name?
        criteria: Should remember and respond with Alice

Re-evaluation (--reeval)

The --reeval flag allows re-running evaluation against cached conversations without regenerating them. This speeds up criteria iteration.

Cache location: .cache/tdx/last_agent_test_run.json

Behavior:

• Cached tests are re-evaluated using updated criteria from test.yml
• New tests (not in cache) are executed normally if on the same site

Key Design Decisions

1. Binary results (PASS/FAIL) - More reliable than scored evaluations
2. Strong judge model - Sonnet 4.5 provides consistent, high-quality evaluation
3. History from API - Authoritative source, includes all events
4. Per-round criteria - Each round evaluated independently for precise feedback
5. Cached re-evaluation - Speeds up criteria iteration without re-running conversations