Recommended score thresholds
Generated by python scripts/eval/thresholds.py; do not edit by hand. Strictness conservative. Documents under 100 words abstain and are excluded. Sources: wiki_aicleanup.jsonl, fineweb_edu_pre2022.jsonl, arxiv_pre2022.jsonl, wiki_2023.jsonl, gutenberg_essays.jsonl.
A recommended score_threshold is the smallest cutoff with pooled false-positive rate at or under 5% on human-good documents, floored at that set's P95. It is stated only when the pooled clean set has at least 100 documents. P99 needs 300.
How to read this
- Stay under the clean P95 for the profile you write in: that is where documented human prose lands, and a draft above it is unusual enough to be worth a look at the findings.
- Gate CI at the recommended cutoff with
score_threshold (or --fail-on-score); the FPR column is the share of human-good documents that would fail.
- The TPR column is the recall on Wikipedia articles editors flagged as suspected AI-generated. It is low. Those labels are subjective and often about citations or markup rather than prose, and the flagged articles are encyclopedic prose without the blog-style tells the rule packs target. Read a low TPR as "this gate will not catch wild Wikipedia slop", not as "the threshold is wrong".
- Human-good sources: FineWeb-Edu pages from Common Crawl dumps dated 2021 or earlier (pre-LLM by construction), arXiv abstracts through 2021, random Wikipedia articles from the 2023-11 snapshot ("not flagged", not "pre-LLM"), and pre-1929 essays (reported separately, never pooled).
| profile |
n clean |
n slop |
clean P50 |
P90 |
P95 |
P99 |
recommended |
FPR |
TPR |
| blog |
522 |
90 |
7.5 |
10.0 |
11.7 |
15.5 |
25.0 |
0.01 |
0.01 |
| essay |
522 |
90 |
7.6 |
10.2 |
11.6 |
15.7 |
25.0 |
0.01 |
0.01 |
| academic |
522 |
90 |
7.5 |
9.7 |
11.2 |
14.6 |
25.0 |
0.01 |
0.01 |
| marketing |
522 |
90 |
7.5 |
9.6 |
10.9 |
14.4 |
25.0 |
0.01 |
0.01 |
| technical |
522 |
90 |
7.4 |
9.4 |
10.7 |
13.8 |
25.0 |
0.01 |
0.01 |
| social |
522 |
90 |
7.5 |
10.0 |
11.5 |
15.1 |
25.0 |
0.01 |
0.01 |
Cutoff sweep (pooled)
blog
| cutoff |
FPR |
TPR |
| 25 |
0.01 |
0.01 |
| 30 |
0.01 |
0.01 |
| 35 |
0.01 |
0.01 |
| 40 |
0.01 |
0.01 |
| 45 |
0.00 |
0.01 |
| 50 |
0.00 |
0.01 |
| 55 |
0.00 |
0.00 |
| 60 |
0.00 |
0.00 |
| 65 |
0.00 |
0.00 |
| 70 |
0.00 |
0.00 |
| 75 |
0.00 |
0.00 |
essay
| cutoff |
FPR |
TPR |
| 25 |
0.01 |
0.01 |
| 30 |
0.01 |
0.01 |
| 35 |
0.00 |
0.01 |
| 40 |
0.00 |
0.01 |
| 45 |
0.00 |
0.01 |
| 50 |
0.00 |
0.01 |
| 55 |
0.00 |
0.00 |
| 60 |
0.00 |
0.00 |
| 65 |
0.00 |
0.00 |
| 70 |
0.00 |
0.00 |
| 75 |
0.00 |
0.00 |
academic
| cutoff |
FPR |
TPR |
| 25 |
0.01 |
0.01 |
| 30 |
0.01 |
0.01 |
| 35 |
0.00 |
0.01 |
| 40 |
0.00 |
0.01 |
| 45 |
0.00 |
0.00 |
| 50 |
0.00 |
0.00 |
| 55 |
0.00 |
0.00 |
| 60 |
0.00 |
0.00 |
| 65 |
0.00 |
0.00 |
| 70 |
0.00 |
0.00 |
| 75 |
0.00 |
0.00 |
marketing
| cutoff |
FPR |
TPR |
| 25 |
0.01 |
0.01 |
| 30 |
0.00 |
0.01 |
| 35 |
0.00 |
0.01 |
| 40 |
0.00 |
0.01 |
| 45 |
0.00 |
0.00 |
| 50 |
0.00 |
0.00 |
| 55 |
0.00 |
0.00 |
| 60 |
0.00 |
0.00 |
| 65 |
0.00 |
0.00 |
| 70 |
0.00 |
0.00 |
| 75 |
0.00 |
0.00 |
technical
| cutoff |
FPR |
TPR |
| 25 |
0.01 |
0.01 |
| 30 |
0.01 |
0.01 |
| 35 |
0.00 |
0.01 |
| 40 |
0.00 |
0.00 |
| 45 |
0.00 |
0.00 |
| 50 |
0.00 |
0.00 |
| 55 |
0.00 |
0.00 |
| 60 |
0.00 |
0.00 |
| 65 |
0.00 |
0.00 |
| 70 |
0.00 |
0.00 |
| 75 |
0.00 |
0.00 |
social
| cutoff |
FPR |
TPR |
| 25 |
0.01 |
0.01 |
| 30 |
0.01 |
0.01 |
| 35 |
0.00 |
0.01 |
| 40 |
0.00 |
0.01 |
| 45 |
0.00 |
0.01 |
| 50 |
0.00 |
0.01 |
| 55 |
0.00 |
0.00 |
| 60 |
0.00 |
0.00 |
| 65 |
0.00 |
0.00 |
| 70 |
0.00 |
0.00 |
| 75 |
0.00 |
0.00 |
Per subgroup at the recommended cutoff
blog
| subgroup |
pooled |
n clean |
n slop |
clean P50 |
P90 |
P95 |
FPR |
TPR |
| arxiv_pre2022 |
yes |
150 |
0 |
8.2 |
12.5 |
14.4 |
0.02 |
n/a |
| gutenberg |
no |
120 |
0 |
7.5 |
10.2 |
11.3 |
0.00 |
n/a |
| web_edu_pre2022 |
yes |
150 |
0 |
7.6 |
9.3 |
10.6 |
0.00 |
n/a |
| wiki |
yes |
72 |
90 |
7.3 |
8.0 |
n/a |
0.00 |
0.01 |
| wiki_2023 |
yes |
150 |
0 |
7.1 |
7.7 |
8.1 |
0.00 |
n/a |
essay
| subgroup |
pooled |
n clean |
n slop |
clean P50 |
P90 |
P95 |
FPR |
TPR |
| arxiv_pre2022 |
yes |
150 |
0 |
8.6 |
12.9 |
14.6 |
0.02 |
n/a |
| gutenberg |
no |
120 |
0 |
7.5 |
9.9 |
10.8 |
0.00 |
n/a |
| web_edu_pre2022 |
yes |
150 |
0 |
7.7 |
9.3 |
10.4 |
0.00 |
n/a |
| wiki |
yes |
72 |
90 |
7.3 |
8.1 |
n/a |
0.00 |
0.01 |
| wiki_2023 |
yes |
150 |
0 |
7.1 |
7.8 |
8.0 |
0.00 |
n/a |
academic
| subgroup |
pooled |
n clean |
n slop |
clean P50 |
P90 |
P95 |
FPR |
TPR |
| arxiv_pre2022 |
yes |
150 |
0 |
8.5 |
12.5 |
14.0 |
0.02 |
n/a |
| gutenberg |
no |
120 |
0 |
7.5 |
9.4 |
10.2 |
0.00 |
n/a |
| web_edu_pre2022 |
yes |
150 |
0 |
7.7 |
9.1 |
10.1 |
0.00 |
n/a |
| wiki |
yes |
72 |
90 |
7.3 |
8.1 |
n/a |
0.00 |
0.01 |
| wiki_2023 |
yes |
150 |
0 |
7.1 |
7.7 |
7.9 |
0.00 |
n/a |
marketing
| subgroup |
pooled |
n clean |
n slop |
clean P50 |
P90 |
P95 |
FPR |
TPR |
| arxiv_pre2022 |
yes |
150 |
0 |
8.5 |
11.9 |
13.4 |
0.02 |
n/a |
| gutenberg |
no |
120 |
0 |
7.5 |
8.9 |
9.4 |
0.00 |
n/a |
| web_edu_pre2022 |
yes |
150 |
0 |
7.7 |
8.9 |
9.6 |
0.00 |
n/a |
| wiki |
yes |
72 |
90 |
7.2 |
8.1 |
n/a |
0.00 |
0.01 |
| wiki_2023 |
yes |
150 |
0 |
7.1 |
7.7 |
7.9 |
0.00 |
n/a |
technical
| subgroup |
pooled |
n clean |
n slop |
clean P50 |
P90 |
P95 |
FPR |
TPR |
| arxiv_pre2022 |
yes |
150 |
0 |
8.0 |
11.3 |
13.1 |
0.02 |
n/a |
| gutenberg |
no |
120 |
0 |
7.4 |
9.3 |
10.2 |
0.00 |
n/a |
| web_edu_pre2022 |
yes |
150 |
0 |
7.5 |
9.0 |
9.7 |
0.00 |
n/a |
| wiki |
yes |
72 |
90 |
7.2 |
7.8 |
n/a |
0.00 |
0.01 |
| wiki_2023 |
yes |
150 |
0 |
7.1 |
7.7 |
7.8 |
0.00 |
n/a |
social
| subgroup |
pooled |
n clean |
n slop |
clean P50 |
P90 |
P95 |
FPR |
TPR |
| arxiv_pre2022 |
yes |
150 |
0 |
8.6 |
12.7 |
14.4 |
0.02 |
n/a |
| gutenberg |
no |
120 |
0 |
7.5 |
9.3 |
10.2 |
0.00 |
n/a |
| web_edu_pre2022 |
yes |
150 |
0 |
7.7 |
9.1 |
10.1 |
0.00 |
n/a |
| wiki |
yes |
72 |
90 |
7.2 |
8.1 |
n/a |
0.00 |
0.01 |
| wiki_2023 |
yes |
150 |
0 |
7.1 |
7.8 |
8.0 |
0.00 |
n/a |