Further down the line after you have your website up and running you may employ an seo agency to help improve your search engine ranking. One of the first things a good seo company will do is review your site and carry out header checks. If these checks result in duplicate content it is most likely that they will recommend fixing the issues. By thinking about these options before you build your website can prevent potentially costly developer time later on.
For website development companies and seo companies or anyone with an interest in on-site seo it is important to know that your server is returning appropriate headers when a webpage is requested.
Headers are the response returned by a server to the user agent (typically a browser or a bot) before the page content is returned. This information tells the user agent what to do with the request. The most common response codes are 200 meaning the page is fine, 301 meaning the page has permanently moved to a new url and 404 meaning the page does not exist.
This tool provides a simple way of testing headers, enter a url below and try it out.
Search engines and specifically Google do not like duplicate content. Unfortunately a lot of servers can be quite forgiving when a users requests a url and in a bad case could accept 8 different urls serving up the same content for each as in the example below.
If your website outputs a 200 OK header for multiple versions of the same page this can create problems. Search engines can see this as duplicate content this can go against you. You can also end up sharing link juice between several urls, rather than giving all the benefit to one url.
To help prevent duplicate content issues, ensure your website only provides one url for each page, you will need to decide a few things first:
www or no www it is not so important, but we prefer with because some web users put them in anyway.
Slash or no slash it doesn't really matter.
We would recommend that all urls are always in lowercase, if you don't do this from the start you can create yourself a lot of work later on.
Whatever you choose, once you have decided STICK TO IT.
Make sure the internal linking structure created by your web application outputs urls consistently to match your chosen url format.
Whenever creating backlinks to your site you should always be consistent, making sure the format of the url matches the format your server outputs, for example always link with www. in lower case and without a slash on the end.
Deal with the urls that do not match your pattern to prevent them from being indexed by search engines. This could be done by return a 404 header to say the page does not exist or by returning a 301 redirect header with the correct url or possibly by outputting the canonical tag.
A final note of caution: when writing your 301s into your system try to avoid chaining. Chaining 301s is when multiple 301s are triggered on one request. If a url 301 chain is too long it will not get indexed and will use unecessary server resources. This tool was created to test test for 301 chains.