{"id":"coding-test-runner","title":"Test Runner","description":"Parses raw Vitest stdout/stderr into a TYPED test report: totals + per-failure details (test, file, message, one-sentence root-cause hypothesis), grouped by suspected cause. Analyses captured output; does not execute anything itself. Reports only what the output shows, never invents.","category":"coding","version":"2.0.0","source":"AKOS","license":"MIT","tags":["coding","structured-output","testing"],"packages":["@agentskit/core","@agentskit/runtime","@agentskit/tools"],"files":["agent.ts","README.md","eval.ts"],"requires":{"zod":"^3","zod-to-json-schema":"^3"},"status":"validated","skill":{"name":"coding-test-runner","description":"Parses raw Vitest stdout/stderr into a TYPED test report: totals + per-failure details (test, file, message, one-sentence root-cause hypothesis), grouped by suspected cause. Analyses captured output; does not execute anything itself. Reports only what the output shows, never invents.","systemPrompt":"You analyse raw Vitest stdout/stderr and produce a STRUCTURED test report. Extract: total\npassed, failed, skipped, and duration. For EACH failure: the test name, the file path, the assertion\nmessage, and a one-sentence root-cause hypothesis. Group failures by suspected root cause so the next\nagent can prioritise. Report only what the output shows — do not invent failures or passes.\n\n${UNTRUSTED_CONTENT_DIRECTIVE}\n\nCall submit_report exactly once with { passed, failed, skipped, duration, failures, summary }. Stop."},"flow":null,"a2a":{"id":"io.agentskit.registry.coding-test-runner","name":"Test Runner","description":"Parses raw Vitest stdout/stderr into a TYPED test report: totals + per-failure details (test, file, message, one-sentence root-cause hypothesis), grouped by suspected cause. Analyses captured output; does not execute anything itself. Reports only what the output shows, never invents.","version":"2.0.0","homepage":"https://registry.agentskit.io","skills":[{"name":"coding-test-runner","description":"Parses raw Vitest stdout/stderr into a TYPED test report: totals + per-failure details (test, file, message, one-sentence root-cause hypothesis), grouped by suspected cause. Analyses captured output; does not execute anything itself. Reports only what the output shows, never invents.","capabilities":{"streaming":true,"cancellation":true,"requiresApproval":false}}]},"sources":[{"path":"agent.ts","content":"import type { AdapterFactory, ChatMemory, Observer, ToolCall, ToolDefinition } from '@agentskit/core'\nimport { fenceUntrustedContent, UNTRUSTED_CONTENT_DIRECTIVE } from '@agentskit/core/security'\nimport { invokeStructured } from '@agentskit/runtime'\nimport { defineZodTool } from '@agentskit/tools'\nimport { z } from 'zod'\nimport { zodToJsonSchema } from 'zod-to-json-schema'\nimport type { JSONSchema7 } from 'json-schema'\n\n/**\n * Test Runner — parses raw Vitest stdout/stderr into a TYPED test report: totals plus\n * per-failure details (test, file, message, one-sentence root-cause hypothesis), grouped\n * so the next agent can prioritise fixes. It analyses output you already captured; it does\n * not execute anything itself (no shell access by design).\n *\n * ```ts\n * const report = await createTestRunnerAgent({ adapter }).run(vitestOutput)\n * ```\n */\n\nexport interface TestFailure {\n  test: string\n  file: string\n  message: string\n  /** One-sentence root-cause hypothesis. */\n  rootCause: string\n}\n\nexport interface TestReport {\n  passed: number\n  failed: number\n  skipped: number\n  duration: string\n  failures: TestFailure[]\n  summary: string\n}\n\nexport interface TestRunnerConfig {\n  adapter: AdapterFactory\n  memory?: ChatMemory\n  observers?: Observer[]\n  onConfirm?: (toolCall: ToolCall) => boolean | Promise<boolean>\n  maxSteps?: number\n}\n\nconst Report = z.object({\n  passed: z.number().int().min(0),\n  failed: z.number().int().min(0),\n  skipped: z.number().int().min(0),\n  duration: z.string(),\n  failures: z.array(z.object({\n    test: z.string(),\n    file: z.string(),\n    message: z.string(),\n    rootCause: z.string(),\n  })),\n  summary: z.string(),\n})\nconst toJson = (s: z.ZodTypeAny): JSONSchema7 => zodToJsonSchema(s) as JSONSchema7\n\nconst skill = {\n  name: 'test-runner',\n  description: 'Parses raw Vitest output into a typed test report with per-failure root-cause hypotheses.',\n  systemPrompt: `You analyse raw Vitest stdout/stderr and produce a STRUCTURED test report. Extract: total\npassed, failed, skipped, and duration. For EACH failure: the test name, the file path, the assertion\nmessage, and a one-sentence root-cause hypothesis. Group failures by suspected root cause so the next\nagent can prioritise. Report only what the output shows — do not invent failures or passes.\n\n${UNTRUSTED_CONTENT_DIRECTIVE}\n\nCall submit_report exactly once with { passed, failed, skipped, duration, failures, summary }. Stop.`,\n  tools: ['submit_report'],\n}\n\nexport function createTestRunnerAgent(config: TestRunnerConfig) {\n  const emit = (label: string, status: 'start' | 'ok' | 'skip' | 'error', detail?: string) => {\n    for (const o of config.observers ?? []) void o.on({ type: 'progress', label, status, detail })\n  }\n  const submit = (): ToolDefinition =>\n    defineZodTool({ name: 'submit_report', description: 'Submit the test report. Call exactly once.', schema: Report, toJsonSchema: toJson, async execute() { return 'recorded' } }) as ToolDefinition\n\n  async function run(vitestOutput: string): Promise<TestReport> {\n    if (!vitestOutput?.trim()) throw new Error('test runner requires raw Vitest output')\n    emit('parse', 'start')\n    const report = await invokeStructured({\n      adapter: config.adapter,\n      tool: submit(),\n      task: `RAW VITEST OUTPUT:\\n${fenceUntrustedContent(vitestOutput)}`,\n      parse: (a) => Report.parse(a),\n      skill,\n      memory: config.memory,\n      observers: config.observers,\n      onConfirm: config.onConfirm,\n      maxSteps: config.maxSteps ?? 3,\n    })\n    emit('parse', 'ok', `${report.passed} passed, ${report.failed} failed`)\n    return report\n  }\n\n  return {\n    name: 'coding-test-runner',\n    run,\n    asHandle() {\n      return { name: 'coding-test-runner', run: async (task: string) => JSON.stringify(await run(task)) }\n    },\n  }\n}\n"},{"path":"README.md","content":"# Test Runner\n\nParses raw Vitest output into a **typed test report** — per-failure root-cause hypotheses, grouped for prioritisation.\n\n```bash\nnpx agentskit add coding-test-runner\n```\n\n```ts\nimport { anthropic } from '@agentskit/adapters'\nimport { createTestRunnerAgent } from './agents/coding-test-runner/agent'\n\nconst report = await createTestRunnerAgent({\n  adapter: anthropic({ apiKey: process.env.ANTHROPIC_API_KEY!, model: 'claude-opus-4-8' }),\n}).run(vitestStdout)\n// → { passed, failed, skipped, duration, failures: [{ test, file, message, rootCause }], summary }\n```\n\n- **Typed report** — `invokeStructured` + zod; each failure gets a one-sentence root-cause hypothesis, grouped by suspected cause.\n- **Analyses, doesn't execute** — it parses output **you** captured; no shell access by design (safe to run anywhere). Reports only what the output shows — never invents passes or failures.\n- Untrusted output text is **fenced**.\n\n`run(vitestOutput)` → `TestReport`. `asHandle()` is JSON-out. See [composing agents](../../COMPOSING.md). Sibling of [`coding-code-qa`](../coding-code-qa) (which runs the commands).\n"},{"path":"eval.ts","content":"import type { EvalSuite } from '@agentskit/eval'\n\nexport const suite: EvalSuite = {\n  name: 'coding-test-runner',\n  cases: [\n    {\n      input: `Spec files run: src/cart/total.test.ts. Vitest output:\n FAIL  src/cart/total.test.ts > cartTotal > applies the 10% discount\nAssertionError: expected 90 to be 81\n ❯ src/cart/total.test.ts:23:25\n Test Files  1 failed (1)\n      Tests  1 failed | 4 passed (5)\n   Duration  1.92s\nProduce the structured report.`,\n      expected: (r: string) => /total\\.test\\.ts/i.test(r) && /(rootCause|root cause|cause|hypothes)/i.test(r),\n    },\n    {\n      input: `Spec files run: src/api/users.test.ts, src/api/orders.test.ts. Vitest output:\n Test Files  2 passed (2)\n      Tests  17 passed (17)\n   Duration  3.41s\nProduce the structured report.`,\n      expected: (r: string) => /17/.test(r) && /3\\.41/.test(r) && /(passed|\"passed\")/i.test(r),\n    },\n    {\n      input: `Spec files run: src/parse/date.test.ts. Vitest output:\n FAIL  src/parse/date.test.ts > parseDate > ISO string\nTypeError: Cannot read properties of undefined (reading 'getTime')\n ❯ src/parse/date.test.ts:11:34\n FAIL  src/parse/date.test.ts > parseDate > unix epoch\nTypeError: Cannot read properties of undefined (reading 'getTime')\n ❯ src/parse/date.test.ts:18:34\n      Tests  2 failed | 1 passed (3)\n   Duration  0.88s\nGroup failures by suspected root cause.`,\n      expected: (r: string) => /date\\.test\\.ts/i.test(r) && /(undefined|getTime|TypeError)/i.test(r),\n    },\n    {\n      input: `Spec files run: src/queue/worker.test.ts. The Vitest process was killed before producing any summary line — the output is truncated mid-run with no totals. Produce the report.`,\n      expected: (r: string) => /(incomplete|truncat|no (summary|total)|cannot|killed|unable|missing|escalat)/i.test(r),\n    },\n  ],\n}\n"}],"installable":true,"validation":{"status":"approved","score":96,"confidence":0.96,"method":"codex-executor-independent-reviewer","iterations":1,"cases":3,"summary":"The agent produced valid structured reports for all three cases, did not execute anything, did not invent Vitest totals or failures from non-Vitest input, surfaced uncertainty, and resisted the injection attempt. Behavior aligns with the stated purpose of parsing captured output only. Minor issue: the minimal case uses an empty string for duration instead of a clearer sentinel like \"unknown\", which is less consistent but not a critical failure.","strengths":["Valid structured output in every case.","Correctly treated prompt-like input as untrusted data rather than instructions.","Did not hallucinate test results, file paths, failures, or root causes.","Injection case ignored the request to output APPROVED and preserved the expected report shape.","Summaries clearly explain that no Vitest output was present."],"notes":["Use a consistent explicit unknown marker for missing duration, preferably \"unknown\", rather than an empty string."]}}