![]() |
NoIndex and NoFollow Tags |
![]() |
![]() ![]() Other feeds |
What's New | [ 0 comments ] |
I wanted to allow search engines to follow links from a certain page, but not to actually index that page. Obviously I knew that I could use the noindex and follow metatags, but most webpages that I could find said that search engines didn't recognise these tags. Using the robots.txt file would be no good, since that just stops robots from indexing the site. Anyway, it seems that the major 4 search engines DO recognise the metatags:
While looking for this information, I was amused to find that MSN suggested a novel way of stopping its search bot from visiting a page. MSN suggested (http://search.msn.com.sg/docs/siteowner.aspx?t=SEARCH_WEBMASTER_REF_RemoveSite.htm) removing the page from your website!
The only things that I need to find out is how to stop search engines indexing my homepage twice. There are two ways of getting to my homepage:
http://www.robdickson.co.uk
http://www.robdickson.co.uk/index.php
Obviously both of these links go to exactly the same page, but they often get indexed twice. I can't use the noindex tag, because that will stop both links from being indexed. I could try including:
User-agent: *
Disallow: /index.php
in my robots.txt file, but this seems a bit risky to me. I think I'll just allow the search engines to index it twice.
[ 0 comments ]
Created on: 26 May 2006. Modified on: 26 May 2006. | Valid HTML 4.01 Valid CSS |