Sitemap in robots.txt

Are you a SEO-conscioncious webmaster? Of course, you are!

And you already have a nice, standards-compliant XML sitemap! Congrats! (Maybe you used a Joomla component to generate it on the fly - Brilliant!) You even had submitted it to the Google! Great! And you think you can enjoy from now that well-deserved pina colada, right? Hold on, you have some important tasks to do!

Relatively recently the major search-engines agreed to extend the robots.txt's functionality. Until recently this little txt file in your root has served mostly to deny the search-engine robots to index files which aren't their business. And the file is up there in any recent Joomla install, and not without purpose, just remember the story when lot of Joomla webmasters where discovered their site backups, complete with admin passwords and other sensitive data readily indexed and publicized by Google. (Just not blame Google that don't index your site fully Evil)

So, since mid May 2008 many SEO sites had reported, that all major webrobots are looking in the robots.txt file for the command:


And if you have such a command, (and a valid XML file at the given URL), this will be processed. The advantage is obvious: this will allow your sitemap to be picked up by several search engines automatically. Nobody is sure how long this feature has been around, but it's a welcome thing, and you should use it on your favor!

And measured in Internet's dog years it's an old trick enough to be in your default arsenal. If isn't... you can't consider yourself a SEO-aware webmaster.