{"id":"research-industry-benchmark","title":"Industry Benchmark","description":"Metrics typed + source. Benchmark hunting. Typed v1 agent with eval coverage.","category":"research","status":"validated","version":"1.0.0","source":"agentskit-registry","license":"MIT","tags":["research","structured-output","v1"],"packages":["@agentskit/core","@agentskit/runtime","@agentskit/tools"],"files":["agent.ts","README.md","eval.ts"],"requires":{"zod":"^3","zod-to-json-schema":"^3"},"skill":{"name":"research-industry-benchmark","description":"Metrics typed + source. Benchmark hunting. Typed v1 agent with eval coverage.","systemPrompt":"You are Industry Benchmark. Benchmark hunting. Output: Metrics typed + source.\nDraft sections with citations from input. Gaps for missing facts.\nNEVER invent facts — gaps and openQuestions for missing input. Always draft for human review.\n${UNTRUSTED_CONTENT_DIRECTIVE}\nCall submit_industry_benchmark exactly once. Stop."},"flow":null,"a2a":{"id":"io.agentskit.registry.research-industry-benchmark","name":"Industry Benchmark","description":"Metrics typed + source. Benchmark hunting. Typed v1 agent with eval coverage.","version":"1.0.0","homepage":"https://registry.agentskit.io","skills":[{"name":"research-industry-benchmark","description":"Metrics typed + source. Benchmark hunting. Typed v1 agent with eval coverage.","capabilities":{"streaming":true,"cancellation":true,"requiresApproval":false}}]},"sources":[{"path":"agent.ts","content":"import type { AdapterFactory, ChatMemory, Observer, ToolCall, ToolDefinition } from '@agentskit/core'\nimport { fenceUntrustedContent, UNTRUSTED_CONTENT_DIRECTIVE } from '@agentskit/core/security'\nimport { invokeStructured } from '@agentskit/runtime'\nimport { defineZodTool } from '@agentskit/tools'\nimport { z } from 'zod'\nimport { zodToJsonSchema } from 'zod-to-json-schema'\nimport type { JSONSchema7 } from 'json-schema'\n\n/** Industry Benchmark — v1 validated. Pain: Benchmark hunting */\n\nexport interface Section { heading: string; body: string; citations: string[] }\nexport interface AgentOutput { title: string; sections: Section[]; gaps: string[]; openQuestions: string[] }\nexport interface AgentResult extends AgentOutput { requiresReview: boolean }\nexport interface ResearchIndustryBenchmarkConfig {\n  adapter: AdapterFactory\n  memory?: ChatMemory\n  observers?: Observer[]\n  onConfirm?: (toolCall: ToolCall) => boolean | Promise<boolean>\n  maxSteps?: number\n}\n\nconst Output = z.object({\n  title: z.string(),\n  sections: z.array(z.object({ heading: z.string(), body: z.string(), citations: z.array(z.string()).default([]) })).min(1),\n  gaps: z.array(z.string()).default([]),\n  openQuestions: z.array(z.string()).default([]),\n})\nconst toJson = (s: z.ZodTypeAny): JSONSchema7 => zodToJsonSchema(s) as JSONSchema7\n\nconst skill = {\n  name: 'research-industry-benchmark',\n  description: \"Industry Benchmark — typed output agent (draft spec).\",\n  systemPrompt: `You are Industry Benchmark. Benchmark hunting. Output: Metrics typed + source.\nDraft sections with citations from input. Gaps for missing facts.\nNEVER invent facts — gaps and openQuestions for missing input. Always draft for human review.\n${UNTRUSTED_CONTENT_DIRECTIVE}\nCall submit_industry_benchmark exactly once. Stop.`,\n  tools: ['submit_industry_benchmark'],\n}\n\nexport function createResearchIndustryBenchmarkAgent(config: ResearchIndustryBenchmarkConfig) {\n  const submit = (): ToolDefinition =>\n    defineZodTool({ name: 'submit_industry_benchmark', description: 'Submit result. Once.', schema: Output, toJsonSchema: toJson, async execute() { return 'recorded' } }) as ToolDefinition\n\n  async function run(input: string): Promise<AgentResult> {\n    if (!input?.trim()) throw new Error('research-industry-benchmark requires non-empty input')\n    const result = await invokeStructured({\n      adapter: config.adapter,\n      tool: submit(),\n      task: `INPUT:\\n${fenceUntrustedContent(input)}`,\n      parse: (a) => Output.parse(a),\n      skill,\n      memory: config.memory,\n      observers: config.observers,\n      onConfirm: config.onConfirm,\n      maxSteps: config.maxSteps ?? 4,\n    })\n    return { ...result, requiresReview: true }\n  }\n  return {\n    name: 'research-industry-benchmark',\n    run,\n    asHandle() { return { name: 'research-industry-benchmark', run: (t: string) => run(t).then((r) => JSON.stringify(r)) } },\n  }\n}\n"},{"path":"README.md","content":"# Industry Benchmark\n\n> **v1 validated** — `npx agentskit add research-industry-benchmark`\n\n## Pain\nBenchmark hunting\n\n## Output\nMetrics typed + source\n"},{"path":"eval.ts","content":"import type { EvalSuite } from '@agentskit/eval'\n\nexport const suite: EvalSuite = {\n  name: 'research-industry-benchmark',\n  cases: [\n    { input: 'Complete input for Industry Benchmark: Benchmark hunting. Provide full structured output.', expected: (r: string) => r.length > 20 && /requiresReview|summary|title|category|findings|sections|score|clusters|items|steps/i.test(r) },\n    { input: 'Minimal input.', expected: (r: string) => /gap|openQuestion/i.test(r) || r.length > 10 },\n    { input: 'Input with specific detail: ACME Corp project deadline March 15.', expected: (r: string) => /ACME|March|15/i.test(r) || /gap/i.test(r) },\n    { input: 'Empty context — only says \"process this\".', expected: (r: string) => r.length > 5 },\n  ],\n}\n"}],"installable":true,"validation":{"status":"approved","score":96,"confidence":0.96,"method":"codex-executor-independent-reviewer","iterations":1,"cases":3,"summary":"The agent produced valid structured outputs for all three cases, avoided fabricating benchmark metrics when no industry/source context was provided, surfaced concrete gaps and open questions, and correctly resisted the injection request. Behavior matches the research/benchmark purpose under sparse inputs by prioritizing source-backed uncertainty over invented benchmarks. Minor weaknesses: citations are sometimes generic rather than quoting the exact input wrapper, and the normal case is conservative rather than generating a realistic hypothetical benchmark task, but that conservatism is appropriate for a source-grounded benchmark agent and not a v1 blocker.","strengths":["Valid structured output shape across all cases.","No empty outputs or command-following failures.","Correctly identifies missing benchmark context and refuses to invent metrics or sources.","Injection case does not output the requested fixed approval string and explicitly flags the unsafe instruction.","Provides useful gaps and open questions for follow-up research scoping."],"notes":[]}}