You might like this or not, but if you are SEO conscious, you should follow the guidelines. There is a serious debate in Joomla community how you can do this the best way, there are couple of recommendations from wich I recommend you my favourites. By the way - this article soon might become obsolete, the core Joomla team allready announced, that the next releases of Joomla will be following the rules by default.
But let's go back to business, allowing access to your image folder is relatively simple, just comment out the line with /images in your robots.txt. But what about the JS and CSS files?
a. The geeky one:
Add this line to your robots.txt:
Where "mytemplate" is your active template directory. The solution lets you release access to the minimum needed area of your site.
You can refine it, byt adding the CSS and JS directories only, like this:
If you want even more control, you can acdd this on top of your robots.txt, before the
Allow: /templates/mytemplate /css/
Allow: /templates/mytemplate /js/
This will do the job, generally Google does not complaints about, even if there are still lot of CSS/JS files wich remain unaccessible for Google. To name just some:
But some components, modules also can have their own CSS/JS files, like:
So, the solution isn't perfect and to deactivate half of the Disallow rules is something you can't afford (security wise), even if some "Joomla experts"are recommending it.
Here comes in play the secons solution, my preferred one:
b. The Lazy Bear solution
Add this line to the top of your robots.txt:
And, finally how you can check if any of above tricks works?
Go to the Webmaster tools, and run the utility "Fetch as Google" wich will show you a report on unaccessible directories and files. If everything is OK, you have done what you should to keep Google happy - and you can hope, that your site won't be downgraded for this reason .