Using Robots.txt with a Sitemap Location

Using Robots.txt with a Sitemap Location

A robots.txt file is a useful way of flagging something to the web robots trawling your content, and it's certainly nothing new. However, we've recently started using them in a rather different way; employing robots.txt validators that won't validate unless there is an XML sitemap specified in conjunction to it.

Why this works…

It's a really good idea to do this, as it ensures that the XML sitemap gets picked up by all web robot crawlers that support the 'sitemap' tag in the robots.txt file. Plenty of search engines do, including Google and MSN, which makes it doubly worthwhile.

An example of using robots.txt with a sitemap location

Let's say, for example, that we're using sitemap for domain located at The robots.txt file would look something like this:

User-agent: *
Disallow: /directory-x
Disallow: /directory-y

It's really important to use a blank link between the user-agent tag and sitemap tag, otherwise you might be inadvertently giving some of the web crawlers some issues.


If you like this post please like, share or subscribe !

This website uses cookies to ensure you get the best experience on our website. OK You can see our Cookie policy here