Using Robots.txt to Stop Pages Appearing in Search Results
Posted by in SEO Added October 3, 2011This is one of my favourite SEO mythbusters.
It is the notion that preventing Google crawling a page in your robots.txt will prevent it appearing in organic search results. It’s rare to come across examples, but today I have, triggering me to make this blog post. In case there is any confusion, Matt Cutts explains it clearly in this Webmaster video. He also runs over the use of the meta noindex tag.
Here is the example. ESPN shop are disallowing all UAs from crawling the domain ESPNshop.co.uk.
However if you do some very strict searches, Google still shows the URL in search results.
I love quick wins. Remember if there is a page you want removed from Google organic search results, either
- Add the meta noindex to the <head> tag
- Remove the URL in Google Webmaster Tools
You can follow any responses to this entry through the RSS 2.0 You can leave a response, or trackback.
