GPT detectors are biased against non-native English writers
Seven detectors flagged essays by non-native English speakers at an average false positive rate of 61.3 percent. More than 91 percent of the essays were flagged by at least one detector. The study behind most false-accusation coverage.
What it is
Seven commercial detectors were run on TOEFL essays written by non-native English speakers and on essays by American eighth graders. The native essays passed; the TOEFL essays were flagged at an average false positive rate of 61.3 percent, and more than 91 percent were flagged by at least one tool. The mechanism is perplexity: detectors treat predictable word choice as machine-like, and writers with a smaller working vocabulary produce predictable text. The authors showed that asking a model to enrich the vocabulary flipped the results. Every later debate about fairness in detection starts here.
When to use this entry
Cite it for false-positive risk and for why non-native writers are flagged.
Questions
Which detectors did the Stanford study test?
Seven detectors available in 2023; the paper lists them. Vendors have retrained since, and none has published a comparable peer-reviewed test on non-native writing.
What was the false positive rate?
An average of 61.3 percent on TOEFL essays, with over 91 percent flagged by at least one detector.