How we read 92 megabytes of competitor content without skimming

Several of our posts cite numbers from a corpus of competitor pages. This is where those numbers come from, so you can decide how much weight they carry.

What we collected

Ten companies building in and around product-demo video. For each, we took the public sitemap, fetched every URL it listed, and kept the extracted text of every page that returned content.

• 3,135 pages discovered and attempted
• 3,038 full texts successfully parsed
• 92 MB of prose
• Per-page metadata: title, meta description, publication date, word count, image count, canonical URL, and an extraction quality score

Nothing was sampled. We read whole sites, which is why numbers like “98% of 552 pages” are counts rather than estimates.

How pages were grouped

Individual URLs are not comparable; clusters are. We collapse each URL path into a shape by keeping the first segment and replacing the rest, so /tool-comparison/camtasia-vs-bandicam and /tool-comparison/loom-vs-scribe both become one pattern.

Any shape with eight or more pages is treated as a programmatic cluster. That threshold is arbitrary but low enough to catch small templated sections and high enough that a handful of hand-written pages sharing a prefix does not qualify.

The three measurements

Recycling. For each cluster, every eight-word passage in every page, counted across the whole cluster. A page’s recycling score is the share of its passages that appear on at least one sibling. Eight words is long enough that incidental phrasing does not register and short enough to catch a copied sentence.

This counts minority duplication, which matters: a block copied onto twenty of 552 pages is invisible to a majority-mask approach and is exactly what a search engine deduplicates.

Uniqueness. Five-word passages, compared against a mask of the passages that at least half the cluster shares. What survives is roughly the page’s own content. We report it as a word count by applying the surviving ratio to the page’s length.

Structural sameness. Every heading in the page, lowercased, with proper nouns from the page’s own URL blanked out. Then: what share of a page’s headings do the majority of its siblings also use? This catches the case where the wording differs completely but the skeleton does not.

What this data cannot tell you

This is the most important section and it is short.

There is no traffic, ranking, impression or indexation data anywhere in this corpus. We know what these companies published. We do not know whether any of it worked, and no measurement described above can tell us.

Every conclusion we draw is about construction and evidence. When we say a cluster is thin, we mean it contains little that is not template. We are not saying it does not rank, because we cannot see that.

Two further limits worth naming. Sitemaps are self-reported, so a company that omits URLs from its sitemap is under-counted here. And publication dates come from page metadata, which is set by a CMS and can be rewritten, so a bulk date could in principle reflect a migration rather than a launch.

Reproducing it

Every source is a public sitemap, listed at the bottom of each post that cites this work. The measurements above are described precisely enough to reimplement. If you get different numbers we would genuinely like to know.