Stopping useless links from being indexed

Posted: Mon Nov 12, 2007 1:19 am

This month I'll be looking at ways in which you can stop useless links getting indexed by search engines.

In particular I am keen to explore the use of the html rel = nofollow tag

You may have often seen me posting about how search engines crawl and index worthless links, in particular links in forums for 'new topic', 'reply' and 'quote'.
Whilst this might seem over the top to some, remember that these three links are associated with a given post or forum ID and could therefore be classed as 'duplicate content' - not something we really want from a SEO point of view.

Before I write a lengthy article on how the rel nofollow tag works I need to recap my research and provide real-world examples of code changes including re-testing some sitemapping tools.

I make this post now so you know whats coming and can stay tuned for the juicy bits.

Site Admin Joined: Jan 06, 2006 Posts: 308

BTW, I have a switch on the upcoming pagination plug-in for news in RavenNuke which allows one to control whether the rel="nofollow" is used or not. Wink

Posted: Tue Nov 13, 2007 10:02 am

That will certainly be interesting Smile

I finally found where users profiles were getting indexed from - my RN Info block but suspect similar blocks will also be allowing the crawling of user profiles. This is one thing I wanted to stop as it prevents any links in their profile from getting crawled as well.
I have already modified my block Smile

Site Admin Joined: Jan 06, 2006 Posts: 308

Are you saying that the links to the profile were physically different? If so, that would be a problem (i.e., "duplicate content"). However, if the links are the same, it will not hurt if there are other pages which also reference the same link.

So, if you still wanted the profiles to be crawled we just need to make sure the links to the profile are identical.

E.g., having 5 pages all reference the following link to a profile is fine:

account-profile-1.html

However, having 5 pages with the following "mix" of links would not:

account-profile-1.html
account-profile-view-1.html

Yes, I made up the links just to make a point.

Posted: Wed Nov 14, 2007 4:29 am

No the user profile links are the same but there is a potential for duplicate content. For example if you do not have 'blocks' tapped, you can have a link indexed from the block which is a none tapped link and then from a news article post you could have an indexable link which is tapped - both links take you to the user profile but are completely different links.

The user profile links is more of a personal thing with me. They are valueless links (to my mind) so I don't want them indexed. If I had several thousand members, that would be several thousand links indexed which although it could be argued that they are 'a link' to my site, to my mind they are links that are not of 'relevant content' value.

I hope to show during this thread that judicial use of the rel nofollow tag can get rid of some 'worthless' indexable links.
'Worthless' in so much as they are of no real benefit when it comes to 'relevant content'. We already know that some script kiddies use Google to search for nukes copyright line so adding a rel nofollow tag there would have merits too.

Posted: Thu Nov 15, 2007 8:20 am

The rel nofollow tag is an unofficial HTML tag which some search engines look for.
Because it is not an 'official' tag, some search engines adhere to it strictly, some don't.

The first search engine to look for the rel nofollow tag was Google back in 2005, closely followed by MSN Yahoo and Ask.com

It is important to remember that different search engines interpret the rel nofollow tag differently, so we cannot use it as a magic pill to cure to all ills BUT we can use it as one of a set of tools in our armoury in tweaking our SEO techniques.
For example, Google and Yahoo;
Will follow a link with the rel nofollow attribute
Will not index the link

Some other search engines will follow the link and index it.

The rel nofollow tag was originally introduced and used on blogs which became popular during 2005 as a means to prevent a drop in page rank due to link backs to sites which were not particularly popular or had very low rankings.

A typical example;
YourSite has a pagerank of 5 and it contains a link to a site that has a pagerank of 1. The search engines might look at that and say, "ok, so YourSite is popular and has a pagerank of 5 so it must have unique and valuable content, so what we'll do is, we'll give the site with a pagerank of 1 a boost, lets now give it a pagerank of 2 as YourSite must consider it a valuable resource because of the link to it."
A bit later on, the search engine crawls the (now) pageranked 2 site and looks at its content and says "hmm, actually this site is pretty crap so we are going to give it a new pagerank of 1. Sites linking to it must be pretty crap too so we'll give YourSite a new rank of 4".
You have now lost some of your pagerank!
OK so this is very over simplified and there are more factors in play than just link banks when it comes to search engine algorythms but it makes the point - and it is the primary reason why the rel nofollow tag was introduced.

Site Admin Joined: Jan 06, 2006 Posts: 308

I agree on the "tapped" vs. "untapped". If you find these, please let me know so that I can get them "fixed".

Thank you sir.