WordPress Sitemap Auto-discovery via Robot.txt File

Last November, I wrote that the Google Sitemap Protocol (open-source protocol based on XML) had become the de facto Standard in the search engine industry. Webmasters no longer had to submit sitemaps in different formats to different search engines. However, only Google and Yahoo! had sitemap submission sites.

In a recent post on Official Google Webmaster Central Blog, What’s new with Sitemaps.org?, Vanessa Fox announced that Google will ‘discover’ your sitemap from your robots.txt file. All you have to do is add a single line to your robot.txt file in the following formatYou must use the complete URL. Put this line at the very beginning of the Robot.txt file.:

Sitemap: http://www.mysite.com/sitemap.xml

This makes perfect sense because all search crawlers recognize the robots.txt file. OK, so most webmasters have already submitted their sitemaps via Google’s Webmaster’s Tools and Yahoo!’s Site Explorer. So what’s the big deal? The key change is that other search engines including Yahoo!, MSN Live Search, and Ask.com are supporting both the sitemap protocol and robots.txt inclusion.

This is especially good news for WordPress users because a sitemap can be automatically generated in the XML format using the Google Sitemap plugin.

Also, I highly recommend using the Smart Update Pinger plugin to insure that services are pinged only when you first publish a post and not each time you edit it.

Getting your WordPress blog properly indexed just got a lot easier!

See also:
Discovering Sitemaps
Webmasters Can Now Auto-Discover With Sitemaps