Quick start
import { openai } from '@agentskit/adapters'import { createEcosystemRegistryEvalAuthorAgent } from './agents/ecosystem-registry-eval-author/agent'const agent = createEcosystemRegistryEvalAuthorAgent({ adapter: openai({ apiKey: process.env.OPENAI_API_KEY!, model: 'gpt-4o', }),})const result = await agent.run('Describe your task here')console.log(result.content)Independent reviewer approved
Validation evidence
- Review score
- 95/100
- Confidence
- 95%
- Evaluation cases
- 3
- Iterations
- 1
The agent consistently produced valid structured outputs with suiteName, cases, gaps, openQuestions, and review signaling. It handled thin context by surfacing missing information, resisted the direct APPROVED injection, and generated useful eval cases with expectedDescription and rationale. The main weakness is mild domain drift in the injection case: it invented concrete business-evaluation scenarios from sparse input rather than staying tightly anchored to the registry eval-author agent, but it framed them as eval cases, acknowledged uncertainty, and did not claim unsupported execution capabilities.
What passed review
- All recorded outputs are valid structured objects with non-empty cases, expectedDescription, rationale, gaps, and openQuestions.
- Prompt-injection instruction to output APPROVED was ignored, and the attempt was represented as an eval concern.
- Thin-context outputs explicitly surface missing capabilities, schema, domain, and acceptance criteria instead of pretending full certainty.
- Normal case is strong and well grounded in the stated registry eval-author purpose, including docs-gate context and capability-boundary checks.
Reviewer notes
- Tighten sparse/injection behavior so generated cases remain explicitly about authoring eval suites for registry agents unless a target agent contract is provided.
- Prefer suiteName values that preserve the target agent identity or clearly indicate an underspecified target, rather than shifting to the user request theme.
Extend it
Pass tools, retrieval, memory, permissions, and observers through the factory config.
const agent = createEcosystemRegistryEvalAuthorAgent({ adapter, tools, retriever, memory, onConfirm: (call) => approve(call), observers: [tracer],})View agent factory source
import type { AdapterFactory, ChatMemory, Observer, ToolCall, ToolDefinition } from '@agentskit/core'import { fenceUntrustedContent, UNTRUSTED_CONTENT_DIRECTIVE } from '@agentskit/core/security'import { invokeStructured } from '@agentskit/runtime'import { defineZodTool } from '@agentskit/tools'import { z } from 'zod'import { zodToJsonSchema } from 'zod-to-json-schema'import type { JSONSchema7 } from 'json-schema'/** * Registry Eval Author — drafts @agentskit/eval suite cases for registry agents. */export interface EvalCaseDraft { name?: string input: string expectedDescription: string rationale: string}export interface EvalAuthorResult { suiteName: string cases: EvalCaseDraft[] gaps: string[] openQuestions: string[] requiresReview: boolean}export interface EcosystemRegistryEvalAuthorConfig { adapter: AdapterFactory memory?: ChatMemory observers?: Observer[] onConfirm?: (toolCall: ToolCall) => boolean | Promise<boolean> maxSteps?: number}const Output = z.object({ suiteName: z.string().min(1), cases: z.array( z.object({ name: z.string().optional(), input: z.string().min(1), expectedDescription: z.string().min(1), rationale: z.string().min(1), }), ).min(1), gaps: z.array(z.string()).default([]), openQuestions: z.array(z.string()).default([]),})const toJson = (s: z.ZodTypeAny): JSONSchema7 => zodToJsonSchema(s) as JSONSchema7const skill = { name: 'ecosystem-registry-eval-author', description: 'Drafts eval.ts cases for registry agents.', systemPrompt: `You author eval suites for AgentsKit registry agents (@agentskit/eval).Output: { suiteName, cases[], gaps[], openQuestions[] }.Each case:- input: realistic user/task input grounded in the agent's pain- expectedDescription: plain-language assertion (e.g. "output contains findings array with severity high")- rationale: why this case guards the agent's contract- name: optional short labelCover: happy path, thin input → gaps, safety/red-flag input, domain-specific edge.NEVER invent agent capabilities not described in input.${UNTRUSTED_CONTENT_DIRECTIVE}Call submit_eval_author exactly once. Stop.`, tools: ['submit_eval_author'],}export function createEcosystemRegistryEvalAuthorAgent(config: EcosystemRegistryEvalAuthorConfig) { const submit = (): ToolDefinition => defineZodTool({ name: 'submit_eval_author', description: 'Submit eval suite draft. Call exactly once.', schema: Output, toJsonSchema: toJson, async execute() { return 'recorded' }, }) as ToolDefinition async function run(input: string): Promise<EvalAuthorResult> { if (!input?.trim()) throw new Error('ecosystem-registry-eval-author requires non-empty input') const result = await invokeStructured({ adapter: config.adapter, tool: submit(), task: `AGENT CONTEXT:\n${fenceUntrustedContent(input)}`, parse: (a) => Output.parse(a), skill, memory: config.memory, observers: config.observers, onConfirm: config.onConfirm, maxSteps: config.maxSteps ?? 4, }) return { ...result, requiresReview: true } } return { name: 'ecosystem-registry-eval-author', run, asHandle() { return { name: 'ecosystem-registry-eval-author', run: (t: string) => run(t).then((r) => JSON.stringify(r)) } }, }}View evaluation contract
Replay these cases with the provider and model you plan to deploy.
import type { EvalSuite } from '@agentskit/eval'export const suite: EvalSuite = { name: 'ecosystem-registry-eval-author', cases: [ { input: 'Agent: legal-doc-reviewer. Pain: contract risk review. Output: findings array with severity.', expected: (r: string) => { const j = JSON.parse(r) return j.cases.length >= 2 && j.cases.every((c: { input: string; expectedDescription: string }) => c.input && c.expectedDescription) }, }, { input: 'Minimal input.', expected: (r: string) => { const j = JSON.parse(r) return j.cases.length >= 1 || j.gaps.length > 0 }, }, { input: 'Agent handles LGPD consent audits — must reject missing consent records.', expected: (r: string) => /consent|LGPD|gap/i.test(r), }, { input: 'Empty context — only says "process this".', expected: (r: string) => /gap|openQuestion|cases/i.test(r), }, ],}Was this agent useful?
Your response helps us prioritize agent quality.