Duplicate Content

Duplicate content is the same page, or nearly the same, reachable at more than one URL, on one site or across several. A crawler that finds two copies has to decide which one to keep, and that decision is made for you if you do not make it.

How duplicates arise

Most of it is accidental. The same page answers with and without www, over http and https, with and without a trailing slash, with a tracking parameter appended, or as a printer version. A product listed in two categories gets two paths. A location page copied for a second town keeps every sentence but the town. None of this is written twice on purpose; it is one page the server will hand out under several names.

How a canonical is chosen

Google’s documentation on consolidating duplicate URLs says that when duplicates exist it picks the version it judges best to show, and that a site can state a preference. The methods differ in weight: a redirect is a strong signal that its target should be the canonical, a rel="canonical" annotation is also a strong signal, and listing a URL in the sitemap is a weak one. They can be combined, and they must not contradict each other; a page that redirects to one URL while its canonical tag names another has said two things. The same page adds that a site without any stated preference will usually be fine, which is true for the choice of URL and less true for what accumulates: links to the two copies are split, and an assistant that fetched the copy you did not maintain quotes that one.

Duplication is not scraping

The spam policies describe scraped content separately: content taken from other sites and republished without adding anything, sometimes with synonyms swapped in. That is a policy violation. Two URLs on your own site serving the same page is not; it is a housekeeping problem with a documented fix. The two get confused because both are called duplicate content, and only one of them carries a penalty.

A common question

Is duplicate content on your own site a penalty, or just untidy?

Untidy, in the sense that it carries no penalty. The spam policies deal separately with scraped content, meaning material taken from other sites and republished with nothing added, sometimes with synonyms swapped in, and that is the version that breaks a rule. One server handing out the same page under several names is a housekeeping problem with a documented fix. It still costs something, because links to the two copies are split between them and an assistant that fetched the copy nobody maintains will quote that one.

Sources

Canonical selection and signal strength: Google Search Central, consolidate duplicate URLs. Scraped content: Google Search Central, spam policies. No figures on this page are this site’s own.