Yahoo and Microsoft have signed on to Google’s sitemap.xml format and published it at sitemaps.org. Two weeks ago, Yahoo announced support for wildcards in robots.txt which seems to be something similar to Google’s (non-standard) robots.txt.
It is well past time for updates to these, but it is sad that there was no attempt to include anyone outside of the Big Guys and that there is no invitation for anyone else to contribute. Robotstxt.org is still there, as is the ROBOTS mailing list, but both were bypassed.
The sitemap protocol is published under a Creative Commons license, but there is no mailing list, no wiki, not even a feedback e-mail address on the website. Questions are referred back to each individual search engine.
This is both sad and foolish. The big three are not the only bots in the world, and more eyes make a better spec. Submit this to the IETF, OK? It isn’t that hard, and the spec will be much, much better afterwards. Look at the improvement from RSS 2.0 to Atom (RFC 4287). It is like night and day.
Most Casual Observer 
Oy vey, they really truly should have opened this up for discussion. There are so many fiddly little details of implementation that they are missing. It’s particularly sad when there is a structure in place to support an open standard.
I’ll blog as well, maybe we can get some movement. You going to post to the Robots list?
LikeLike