Benchmarks & research
Shadow-mode validation harness for a pre-execution firewall on AI agent tool calls (TypeSafe/Jev). Real run, findings in report.md.
"""Same battery as a raw HTTP payload, for POST /v1/systemone without the SDK."""The link points at the commit we read, so the line number still holds.
These question sets are lifted from this project's source exactly as written.
is_urgenttrue/falseThe message conveys urgency
severityscoreHow severe the problem is for the customer
0 — Minor annoyance1 — Real problem2 — Losing money nowimport { experimental_evaluate as evaluate } from 'ai';
const { answers } = await evaluate({
model: 'typesafe-ai/jev',
state,
questions: {
is_urgent: { type: 'boolean', instructions: 'The message conveys urgency' },
severity: { type: 'score', instructions: 'How severe the problem is for the customer',
criteria: ['Minor annoyance', 'Real problem', 'Losing money now'] },
},
});