Why duplicate detection misses generated text, and what we use instead

Before we publish a page, a script compares it against its siblings and measures how much text they share. This is the standard approach to catching thin content, and we built ours carefully: eight-word passages, a majority template mask, shingle-based rather than token-based.

Then we ran it against 3,038 pages of competitor content to see what it would catch.

The answer was: the obvious half.

Where it works

Five clusters recycle so much literal text that any duplicate check flags them instantly. The figure below is the share of each page’s text that also appears on a sibling page:

• 93% across 33 pages, one competitor’s model-comparison section
• 82% across 57 integration pages
• 73% across 14 industry pages
• 68% across 44 alternatives pages
• 65% across 21 use-case pages

The 82% case is the clearest. Those 57 pages have a median of 335 words, of which roughly 103 are not shared with a sibling. Strip the template and there is a paragraph left. A word-count threshold catches it. A duplicate check catches it. Nothing subtle is happening.

Where it fails completely

The largest cluster in the corpus, 552 comparison pages built from 187 tool pairs, measures 12% recycled. Our check scores those pages 95% unique. All 552 would publish.

They score that way because each page was written fresh. There is no copied block to find. The sameness is in the argument structure, the fabricated statistic in the opening paragraph, the identical byline, and the single publication date, none of which is a text statistic.

This is not a flaw in our implementation. It is a limit on the entire category. Any measure you can compute from the words is a measure of how the words are arranged, and arrangement is exactly what generation is good at. Generating text that is lexically unique is free. It has been free for years.

What actually separated the two groups

We looked for a signal that tracked the difference, and found one that is almost embarrassing in its simplicity: outbound citations per page.

• Clusters that read as researched: 5 to 12 external sources per page
• Clusters that read as manufactured: 0 to 1

One competitor’s blog averages 12.4 links to sources outside its own domain. Another’s 100-page blog averages 0.15. The 552-page cluster averages 4.1, but almost all of those are internal links to the vendor’s own signup and product pages rather than to anything a reader could check.

Citations are not a proxy for quality. They are a proxy for whether someone went and looked at something, which is the thing we actually care about and the thing text statistics cannot see.

What we changed

Our gate now has two parts, and the older part is no longer the one we trust.

The text statistics stay. They catch the copy-paste failure mode, which is real and which five clusters in this corpus demonstrate. But they are now the secondary check.

The primary check is evidence. A page must cite something, name who verified it, and carry a date on the verification. Our content schema already stored all three on every fact, because we built it that way for other reasons. Turning it into a publication condition was a small change to one function.

The useful property is that it cannot be satisfied by writing better. You can generate prose that scores 95% unique in a second. You cannot generate having driven the product.