Manipulating Search Engine User Agents

A search engine optimisation specialist can instruct a search engine user agent (webbot/spider/crawler) what to index and what not to index using special external files called robots exclusion protocol files and on page meta tags or no follow attributes. The following The Web Clinic blog describes some of the SEO commands you can use to control what a search engine indexes off your ecommerce website…

It is useful to instruct a search engine not to crawl and index a page that is under construction. Adding the nofollow attribute to a link takes the form “<a href = “” rel = “nofollow” >Health Store</a>” and prevents a link from being followed by search engine spiders.

The “nofollow” attribute can also be used in a robots meta tag placed in the head of a webpage. The following will instruct search engines not to index this page and not to follow any links from this page for use in indexing or weighting…

<meta name=”robots” content=”noindex, nofollow” />

The following will tell a spider not to index this page, but to allow the following of links that can then be indexed and weighted…

<meta name=”robots” content=”noindex, follow”>

The following will instruct the spider to index this page but not to follow any links from it and is most commonly used in message boards…

<meta name=”robots” content=”index, nofollow”>

“Robots exclustion protocol” is used to prevent directories from being indexed in a separate robots.txt file which is located in the site’s root directory.

The following instruction tells the search engines to disallow NO directories for any search engine.

User-agent: *

Conversely, the following command will disallow ALL directories for any search engine.

User-agent: *
Disallow: /

  • Share/Save/Bookmark

Related posts:

  1. Search engine marketing versus search engine optimisation Marketing is an integral part of any business, not just...
  2. Search engine queries Given below are some Search Queries • Dictionary: To see...
  3. Search Engine Marketing: The Latest Trend in the Vogue If you get the total number of web pages on...
  4. Search Engine Optimisation tricks to avoid There are a number of black hat and harmful seo...
  5. Search Engine Marketing Explained Which Search Engine is Important? All major search engines provide...

Tags: ,

2 Responses to “Manipulating Search Engine User Agents”

  1. Hi really enjoyed reading your article and was wondering My idea of making search engine data on our movement total nonsense and unsellable to advertising agencies?

  2. Lesia Duncil says:

    @admin: I just have to say your website is the first I’ve come across this morning that doesn’t have spelling errors every other sentence. Thank you for taking the time to write something that doesn’t look like a 6th grader put together. Sorry, just had to vent.

Leave a Reply

Follow our SEO experts in Twitter
  • What our SEO Experts says

  • Archives

  • SEO Keyword Cloud