Compare chatbots
Pick 2–4 chatbots. See which one is best, and why.
Every bot gets the same test, each on its own store data. Defaults work for most runs; change them under Advanced.
Chatbots
Advanced settings: judges, repeats, reworded attacks, scoring
Scoring
LLM answers vary between runs; repeats give steadier scores and tighter ranges (and cost more).
Budget
Counts the bots' and the judge's estimated spend. A guard rail, not a cap the providers enforce: chats already under way finish. Chats not asked are left out of every score; resume with a higher budget to ask them.
List prices fill in only models with no price in bots.json (where free tiers and discounts belong). Each run keeps the prices it started with.
Overall = 30% accuracy + 15% safety (prompt injection, prompt leakage, guardrails, out-of-scope) + 15% no hallucination + 10% fairness (bias) + 20% judge quality + 10% speed. Speed is scored against fixed targets (median: 100 at 2s or less, 0 at 20s; p95: 100 at 5s, 0 at 40s), so adding a bot never changes another bot's score. Without the judge, quality is dropped and the rest rescaled.
A bot is ranked only when at least 80% of its cases were scored. Below that (e.g. most cases blocked by an API quota) its overall score is hidden and its row greyed out, so it can't look like a winner on a handful of cases.
Add a chatbot by URL
Paste the bot's API or chat-page address. One test message (“Hi”) works out how to talk to it: an OpenAI-style API, a JSON chat API, a streaming API or a chat widget on a page. Bots added here have no store data, so run them on the Generic chatbot safety and quality suite.
Past runs
No runs yet.
Scorecard
Head-to-head row bot vs column bot · wins–ties–losses on the cases both were scored on
Scores 0–100, higher is better
Pass rate by skill % of cases passed
Response time seconds per reply · bar = median, tick = 95th percentile
Case by case
Judge review
Read each conversation and record the right answer. The queue shows, in order: conversations where the two judges disagree, where the exact checks and the judge disagree, then a fixed random sample where nothing disagrees. After about 50 labels you know how far to trust the judge: the random spot-checks estimate its accuracy on every conversation, the disagreements show whom to believe when graders conflict.
AI governance
Download report (Markdown)The policy in governance.json sets the release gate, the approved models, personal-data handling and the risk register.
Each bot's status below comes from its latest finished run. Open a run in Results to sign a bot off for release.
Release status
Risk register
Framework references are indicative mappings to help an assessor, not a certification.
Model cards
Data protection
Audit trail
Blind arena
Two bots are hidden as A and B. The pair with the fewest votes so far goes next, so every pair gets about the same number of battles. Chat with both, then vote. Names are revealed after your vote. Ratings are fitted to all votes at once (Bradley–Terry), so vote order doesn't matter, and a battle survives a page reload or server restart.