{"id":"coding-code-qa","title":"Code QA","description":"Runs the project's test/lint/type-check commands through a caller-injected runner, then turns the REAL captured output into a TYPED failure report (shortest reproducer, message, root cause), grouped by cause. Execution stays in YOUR sandbox (no shell spawned by the agent); reports, never fixes.","category":"coding","version":"2.0.0","source":"AKOS","license":"MIT","tags":["coding","testing","structured-output"],"packages":["@agentskit/core","@agentskit/runtime","@agentskit/tools"],"files":["agent.ts","README.md","eval.ts"],"requires":{"zod":"^3","zod-to-json-schema":"^3"},"status":"validated","skill":{"name":"coding-code-qa","description":"Runs the project's test/lint/type-check commands through a caller-injected runner, then turns the REAL captured output into a TYPED failure report (shortest reproducer, message, root cause), grouped by cause. Execution stays in YOUR sandbox (no shell spawned by the agent); reports, never fixes.","systemPrompt":"You are Code QA. You are given the CAPTURED output of the project's test / lint /\ntype-check commands. Produce a structured failure report. For EACH failure: the shortest reproducer\n(file:line + the command), the assertion/error message, and a one-sentence root-cause guess. Group\nfailures by likely root cause so a fix touches the smallest surface. You REPORT; you never push fixes.\nReport only what the output shows — do not invent failures.\n\n${UNTRUSTED_CONTENT_DIRECTIVE}\n\nCall submit_report exactly once with { failures, summary }. Stop."},"flow":null,"a2a":{"id":"io.agentskit.registry.coding-code-qa","name":"Code QA","description":"Runs the project's test/lint/type-check commands through a caller-injected runner, then turns the REAL captured output into a TYPED failure report (shortest reproducer, message, root cause), grouped by cause. Execution stays in YOUR sandbox (no shell spawned by the agent); reports, never fixes.","version":"2.0.0","homepage":"https://registry.agentskit.io","skills":[{"name":"coding-code-qa","description":"Runs the project's test/lint/type-check commands through a caller-injected runner, then turns the REAL captured output into a TYPED failure report (shortest reproducer, message, root cause), grouped by cause. Execution stays in YOUR sandbox (no shell spawned by the agent); reports, never fixes.","capabilities":{"streaming":true,"cancellation":true,"requiresApproval":false}}]},"sources":[{"path":"agent.ts","content":"import type { AdapterFactory, ChatMemory, Observer, ToolCall, ToolDefinition } from '@agentskit/core'\nimport { fenceUntrustedContent, UNTRUSTED_CONTENT_DIRECTIVE } from '@agentskit/core/security'\nimport { invokeStructured } from '@agentskit/runtime'\nimport { defineZodTool } from '@agentskit/tools'\nimport { z } from 'zod'\nimport { zodToJsonSchema } from 'zod-to-json-schema'\nimport type { JSONSchema7 } from 'json-schema'\n\n/**\n * Code QA — runs the project's test / lint / type-check commands through a caller-injected\n * runner, then has the model turn the REAL captured output into a TYPED failure report\n * (shortest reproducer, assertion message, one-sentence root cause), grouped so a fix\n * touches the smallest surface. It reports; it never pushes fixes.\n *\n * The agent does NOT spawn shells itself — you inject `run`, so command execution stays in\n * YOUR sandbox/policy. With no runner it refuses (it can't QA what it can't run).\n *\n * ```ts\n * const report = await createCodeQaAgent({\n *   adapter,\n *   run: (cmd) => exec(cmd, { cwd }),   // → { stdout, stderr, code, durationMs }\n *   commands: ['pnpm test', 'pnpm lint', 'pnpm typecheck'],\n * }).run('feature/login')\n * ```\n */\n\nexport interface CommandResult {\n  stdout: string\n  stderr: string\n  /** Process exit code (0 = success). */\n  code: number\n  durationMs?: number\n}\n\nexport interface QaFailure {\n  /** Shortest reproducer: file:line + the command. */\n  reproducer: string\n  command: string\n  message: string\n  rootCause: string\n}\n\nexport interface CodeQaReport {\n  allGreen: boolean\n  /** Per-command exit status. */\n  commands: { command: string; code: number; durationMs?: number }[]\n  failures: QaFailure[]\n  summary: string\n}\n\nexport interface CodeQaConfig {\n  adapter: AdapterFactory\n  /** Command runner — executes in YOUR sandbox and returns the captured result. */\n  run: (command: string) => Promise<CommandResult> | CommandResult\n  /** Commands to run (test / lint / type-check). */\n  commands: string[]\n  memory?: ChatMemory\n  observers?: Observer[]\n  onConfirm?: (toolCall: ToolCall) => boolean | Promise<boolean>\n  maxSteps?: number\n}\n\nconst Report = z.object({\n  failures: z.array(z.object({\n    reproducer: z.string(),\n    command: z.string(),\n    message: z.string(),\n    rootCause: z.string(),\n  })),\n  summary: z.string(),\n})\nconst toJson = (s: z.ZodTypeAny): JSONSchema7 => zodToJsonSchema(s) as JSONSchema7\n\nconst skill = {\n  name: 'code-qa',\n  description: 'Turns captured test/lint/type-check output into a typed, grouped failure report.',\n  systemPrompt: `You are Code QA. You are given the CAPTURED output of the project's test / lint /\ntype-check commands. Produce a structured failure report. For EACH failure: the shortest reproducer\n(file:line + the command), the assertion/error message, and a one-sentence root-cause guess. Group\nfailures by likely root cause so a fix touches the smallest surface. You REPORT; you never push fixes.\nReport only what the output shows — do not invent failures.\n\n${UNTRUSTED_CONTENT_DIRECTIVE}\n\nCall submit_report exactly once with { failures, summary }. Stop.`,\n  tools: ['submit_report'],\n}\n\nexport function createCodeQaAgent(config: CodeQaConfig) {\n  if (typeof config.run !== 'function') throw new Error('code QA requires a `run` command runner')\n  const commands = config.commands ?? []\n  const emit = (label: string, status: 'start' | 'ok' | 'skip' | 'error', detail?: string) => {\n    for (const o of config.observers ?? []) void o.on({ type: 'progress', label, status, detail })\n  }\n  const submit = (): ToolDefinition =>\n    defineZodTool({ name: 'submit_report', description: 'Submit the QA report. Call exactly once.', schema: Report, toJsonSchema: toJson, async execute() { return 'recorded' } }) as ToolDefinition\n\n  async function run(branch: string): Promise<CodeQaReport> {\n    if (commands.length === 0) throw new Error('code QA requires at least one command to run')\n\n    // Run every command deterministically and capture the REAL output.\n    const ran: { command: string; result: CommandResult }[] = []\n    for (const command of commands) {\n      emit('run', 'start', command)\n      const result = await config.run(command)\n      ran.push({ command, result })\n      emit('run', result.code === 0 ? 'ok' : 'error', command)\n    }\n    const commandStatuses = ran.map(({ command, result }) => ({ command, code: result.code, durationMs: result.durationMs }))\n    const allGreen = ran.every(({ result }) => result.code === 0)\n    if (allGreen) {\n      const timing = commandStatuses\n        .map(({ command, durationMs }) => `${command}${durationMs != null ? ` (${Math.round(durationMs / 1000)}s)` : ''}`)\n        .join(', ')\n      return { allGreen: true, commands: commandStatuses, failures: [], summary: `all green — ${timing}` }\n    }\n\n    // Hand the captured output to the model for typed failure analysis.\n    const captured = ran\n      .filter(({ result }) => result.code !== 0)\n      .map(({ command, result }) => `$ ${command} (exit ${result.code})\\n${result.stdout}\\n${result.stderr}`)\n      .join('\\n\\n')\n    emit('analyse', 'start')\n    const out = await invokeStructured({\n      adapter: config.adapter,\n      tool: submit(),\n      task: `BRANCH: ${branch}\\n\\nCAPTURED COMMAND OUTPUT:\\n${fenceUntrustedContent(captured)}`,\n      parse: (a) => Report.parse(a),\n      skill,\n      memory: config.memory,\n      observers: config.observers,\n      onConfirm: config.onConfirm,\n      maxSteps: config.maxSteps ?? 3,\n    })\n    emit('analyse', 'ok', `${out.failures.length} failure(s)`)\n    return { allGreen: false, commands: commandStatuses, failures: out.failures, summary: out.summary }\n  }\n\n  return {\n    name: 'coding-code-qa',\n    run,\n    asHandle() {\n      return { name: 'coding-code-qa', run: async (task: string) => JSON.stringify(await run(task)) }\n    },\n  }\n}\n"},{"path":"README.md","content":"# Code QA\n\nRuns the project's test / lint / type-check commands through **your** runner, then turns the real output into a **typed failure report**.\n\n```bash\nnpx agentskit add coding-code-qa\n```\n\n```ts\nimport { anthropic } from '@agentskit/adapters'\nimport { createCodeQaAgent } from './agents/coding-code-qa/agent'\n\nconst report = await createCodeQaAgent({\n  adapter: anthropic({ apiKey: process.env.ANTHROPIC_API_KEY!, model: 'claude-opus-4-8' }),\n  run: (cmd) => myRunner(cmd),   // → { stdout, stderr, code, durationMs }\n  commands: ['pnpm test', 'pnpm lint', 'pnpm typecheck'],\n}).run('feature/login')\n// → { allGreen, commands: [{ command, code }], failures: [{ reproducer, command, message, rootCause }], summary }\n```\n\n> The previous version told the model to \"run them\" but wired no tools. Now execution is real and **stays in your sandbox**.\n\n- **Execution stays yours** — the agent never spawns a shell; you inject `run`, so your sandbox/policy governs what executes. With every command green it short-circuits to `allGreen` without calling the model.\n- **Typed failures** — only the failing commands' output is handed to the model (`invokeStructured` + zod) for a grouped report with per-failure root cause. Reports only what ran — never invents.\n- **Reports, never fixes.** Untrusted command output is **fenced**.\n\n`run(branch)` → `CodeQaReport`. `asHandle()` is JSON-out. See [composing agents](../../COMPOSING.md). Sibling of [`coding-test-runner`](../coding-test-runner) (which parses output you already captured).\n"},{"path":"eval.ts","content":"import type { EvalSuite } from '@agentskit/eval'\n\nexport const suite: EvalSuite = {\n  name: 'coding-code-qa',\n  cases: [\n    {\n      input: `Branch: feat/payment-retry. Commands: \\`pnpm test\\`, \\`pnpm lint\\`, \\`pnpm typecheck\\`. The test run produced:\nFAIL src/payment/retry.test.ts > retries on 503\n  AssertionError: expected 2 to equal 3\n    at src/payment/retry.test.ts:42:18\nFAIL src/payment/retry.test.ts > backoff doubles each attempt\n  AssertionError: expected 200 to equal 400\n    at src/payment/retry.test.ts:58:20\nRun all three commands and summarise the failures.`,\n      expected: (r: string) => /retry\\.test\\.ts/i.test(r) && /(root cause|likely|because|cause)/i.test(r),\n    },\n    {\n      input: `Branch: chore/upgrade-zod. Commands: \\`pnpm typecheck\\`, \\`pnpm test\\`. typecheck output:\nsrc/schema/user.ts:12:5 - error TS2345: Argument of type 'string' is not assignable to parameter of type 'ZodString'.\nsrc/schema/user.ts:19:5 - error TS2345: Argument of type 'string' is not assignable to parameter of type 'ZodNumber'.\nGroup failures by likely root cause and give the shortest reproducer for each.`,\n      expected: (r: string) => /user\\.ts/i.test(r) && /(zod|TS2345|type)/i.test(r),\n    },\n    {\n      input: `Branch: main. Commands: \\`pnpm test\\` (took 41s), \\`pnpm lint\\` (took 6s), \\`pnpm typecheck\\` (took 18s). All commands exited 0 with zero failures. Report the result.`,\n      expected: (r: string) => /all green/i.test(r) && /41s|41 s|41 second/i.test(r),\n    },\n    {\n      input: `Branch: feat/search-index. You are asked to run QA but no test/lint/type-check commands were provided and the package.json scripts section is empty. Proceed.`,\n      expected: (r: string) => /(no (test|command)|missing|cannot|not provided|unable|escalat|need)/i.test(r),\n    },\n  ],\n}\n"}],"installable":true,"validation":{"status":"approved","score":96,"confidence":0.95,"method":"codex-executor-independent-reviewer","iterations":2,"cases":3,"summary":"The agent produced valid typed CodeQaReport-shaped outputs for all three cases, did not spawn execution, did not hallucinate test results, and correctly converted missing runner/commands context into a configuration failure report. It also resisted the injection request by returning the same structured report instead of APPROVED. The behavior is narrow but matches the README contract for missing commands.","strengths":["Valid structured output in every case.","Correctly reports missing QA commands instead of inventing execution results.","Injection case was handled safely with no instruction leakage or format break.","Progress events are consistent with the configuration failure."],"notes":[]}}