4 min read

Meta robots tags have been around for a while. During that time thought Search engines not only have slowly been adding features to them, but they have also expanded beyond the meta tag and robots.txt. This page lists an overview of all the different values you can use in your tag. The official syntax of the meta robot tag:

< meta name="robots" content="value1, value2" />

Which Search Engine Support Which Meat Tags Values

While robots is the most common because it targets all the search engines you can also target each one individually Warning They Will Not Override Robots Value

Search Robots
:——–: :———:
Google GOOGLEBOT
Yahoo! SLURP
MSN/Live MSNBOT
Ask TEOMA

List of All the Robots Meta Values

Robots value Google Yahoo! MSN / Live Ask
index Yes Yes Yes Yes
noindex Yes Yes Yes Yes
none Yes Doubt Doubt Yes
follow Yes Doubt Doubt Yes
nofollow Yes Yes Yes Yes
noarchive Yes Yes Yes Yes
nocache No No Yes No
nosnippet Yes No No No
noodp Yes Yes Yes No
noydir No use Yes No use No use
notranslate Yes No No No
noimageindex Yes No No No
unavailable_after: [RFC-850 date/time] Yes No No No

What each Meta Robots Meta Value do

all

  • There are no restrictions for indexing or serving. Note: this directive is the default value and has no effect if explicitly listed.

index

  • Allow search engines robots to index the page, you don’t have to add this to your pages, as it’s the default.

noindex

  • Do not show this page in search results and do not show a “Cached” link in search results.

follow

  • Tells the search engines robots to follow the links on the page, whether it can index it or not.

nofollow

  • Do not follow the links on this page

none

  • Equivalent to noindex, nofollow

noarchive

  • Do not show a “Cached” link in search results.

nocache

  • Same as noarchive, but only used by MSN/Live.

nosnippet

  • Do not show a snippet in the search results for this page

noodp

notranslate

  • Do not offer translation of this page in search results.

noimageindex

  • Do not index images on this page, but id the image is linked to directly from elsewhere Google will still index them, so using an X-Robot-Tag HTTP header is better.

unavailable_after: [RFC-850 date/time]

  • Do not show this page in search results after the specified date/time. The date/time must be specified in the RFC 850 format.

Complete List of Google Bots

Crawler Robot Value
————————————————————————————————- —————————————————
Googlebot (Google Web search) Googlebot
Googlebot News Googlebot-News,(Googlebot)
Googlebot Images Googlebot-Image,(Googlebot)
Googlebot Video Googlebot-Video,(Googlebot)
Google Smartphone Googlebot
Google Mobile AdSense Mediapartners-Google,Mediapartners,(Googlebot)
Google AdSense Mediapartners-Google,Mediapartners,(Googlebot)
Google AdsBot landing page quality check AdsBot-Google
Google app crawler,(Used to fetch resources for mobile apps, obeys AdsBot-Google robots rules.) AdsBot-Google-Mobile-Apps

Resources