The End of Open Spider Standards?

Yahoo and Microsoft have signed on to Google’s sitemap.xml format and published it at Two weeks ago, Yahoo announced support for wildcards in robots.txt which seems to be something similar to Google’s (non-standard) robots.txt.

It is well past time for updates to these, but it is sad that there was no attempt to include anyone outside of the Big Guys and that there is no invitation for anyone else to contribute. is still there, as is the ROBOTS mailing list, but both were bypassed.

The sitemap protocol is published under a Creative Commons license, but there is no mailing list, no wiki, not even a feedback e-mail address on the website. Questions are referred back to each individual search engine.

This is both sad and foolish. The big three are not the only bots in the world, and more eyes make a better spec. Submit this to the IETF, OK? It isn’t that hard, and the spec will be much, much better afterwards. Look at the improvement from RSS 2.0 to Atom (RFC 4287). It is like night and day.

1 thought on “The End of Open Spider Standards?

  1. Oy vey, they really truly should have opened this up for discussion. There are so many fiddly little details of implementation that they are missing. It’s particularly sad when there is a structure in place to support an open standard.
    I’ll blog as well, maybe we can get some movement. You going to post to the Robots list?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.