Every AI-image and deepfake detector says it is accurate. The only way to know is to check its answers against what really was AI or fake. Our detector benchmark does that with results people reviewed by hand, and it shows its sample size and its doubt.
Key takeaways
- Scores come only from results reviewed by hand.
- Only teams that agreed to share are counted. It is off by default.
- Every score has a range, not just a rank.
- Under 200 reviewed results, a detector is not scored.
Where the results come from
Teams using Cockpit review what their detectors flagged: was this image really AI-generated, was this video really a deepfake? Those reviewed results are the ground truth. A team's results only count in the public benchmark if the team agreed, and the setting is off until they do. Sample files are always left out, and no team or customer is ever named.
What we score
| Score | What it means |
|---|---|
| Right when it flags | Of the files it called AI or fake, how many really were. High means few false alarms. |
| Catches | Of the files that really were AI or fake, how many it caught. High means few misses. |
| Overall | Both together in one score (F1), with a 95% range. |
Each check is scored on its own: AI image, deepfake and AI video. Each detector we route to, such as Sightengine, Hive and BitMind, gets its own row, over the last 30 or 90 days.
Why a range, and why 200
A detector that was right on 19 of 20 files looks great, but 20 files say very little. The range shows where the true score likely sits given how many files were reviewed: the more reviews, the narrower the range. Below 200 reviewed results in the period, the row says "Not enough reviews yet" instead of a score. Equal scores share a rank.
A detector's score depends on what it is shown. Check the scores for the kind of content you handle, and keep reviewing your own results.
Using the results
All the detectors in the benchmark are available through the Gateway Toolkit on one key, with the threshold set per check and every result shown next to its cost.
See the detector benchmark and how each detector is scored.