Research and benchmarks

The green-list scheme that most later watermark work builds on: the model is nudged toward a secret subset of tokens, and a detector counts how often they appear.

Tests marks against paraphrasing, copy-paste mixing and human rewriting. Finds that marks weaken but often remain detectable once enough text is available.

Argues that recursive paraphrasing defeats both classifiers and watermarks, and that detection gets harder as models get closer to human text.

Introduces DIPPER, an 11-billion-parameter paraphraser, shows it breaks the detectors of the day, and proposes retrieval of previously generated outputs as a defence that paraphrasing cannot beat.

Seven detectors flagged essays by non-native English speakers at an average false positive rate of 61.3 percent. More than 91 percent of the essays were flagged by at least one detector. The study behind most false-accusation coverage.

The SynthID-Text paper. Describes the tournament sampling method, the production deployment in Gemini, and quality tests across 20 million responses.

A practical walk-through of applying and detecting the mark with Transformers, with the code.

Detection without training data, by contrasting two models' views of the same text. A common independent baseline.

A shared benchmark of more than six million generations across models, domains, decoding settings and adversarial attacks, including paraphrasing. The public leaderboard is the closest thing to a neutral scoreboard for detectors. Paper at arXiv 2405.07940.

The US standards body's overview of provenance, watermarking and detection techniques, with their known limits. A good neutral primer.