I write occasional articles on search and Ultraseek for ultraseek.com, so I’ve collected links to them all from here. I’ve clipped the first paragraph or two of each one so you can see whether you’d like to read the entire article.

Relevance and User Satisfaction

Search relevance is usually thought of as a statistic that measures whether the search results match the query. That is useful in the lab, but not as useful for a search installation.

When search is part of a site, we need to understand how it helps the users of that site. Can they find things quickly? Are they comfortable with the search?

My Favorite Customer Problem

We are concerned about all of the problems reported by our customers, but there is one problem I don’t mind hearing about.

Keeping Your Index Fresh with Add URL

Everyone wants their search index to be an accurate, timely reflection of their content. Ultraseek automatically revisits pages to find new URLs, and that is very effective, but some sites have even stronger reqirements for how quickly documents need to be available in search results. This is called “index freshness.” A stale index frequently misses new pages and has old information including pages that have already been deleted, and old copies of pages that have changed since they were indexed. For maximum index freshness, use Ultraseek’s Add URL feature for notifications of deleted, changed, or new URLs.

Don’t Reindex Every Week!

If you have used other search engines, you probably had to manually configure your indexing schedule to make sure new content was found and indexed. This is not necessary with Ultraseek.

Ultraseek has “continuous spidering with real-time indexing and adaptive revisit intervals.” It sounds complicated, but it means that Ultraseek will automatically spider most pages at the right times.

Why Not Use “All Terms” Queries?

Google, Yahoo!, and MSN all default to matching all of your search terms, but Ultraseek does not. Why? What do you say when your users want Ultraseek to “work like Google”?

In most cases, it is good for an enterprise engine to behave like the WWW engines because users can intuitively transfer their searching skills. But, this is a case where doing the right thing is more expensive for the WWW engines, and more reasonable for enterprises.

It looks like I’ve been careful to write useful leads, because selecting the first few sentences makes a pretty fair intro to each article.

Yahoo’s Open Shortcuts is a nice simple extension to search implemented entirely within the query box. We’ve been able to do queries like wikipedia emma bull or zip code palo alto or weather baton rouge for a while, using the first word as a context command. Some of those (“wikipedia”) bias the results, while others display custom information above the results.

Open Shortcuts adds the ability to punch through to a different search engine (like !ebay dagor) and also to define your own contexts that go to your favorite spots.

It is easy to define a shortcut for any Ultraseek search engine. Here’s how to define a shortcut for ultraseek.com.

1. Go to Create a Shortcut.
2. Scroll down to the Search a Site section.
3. Enter a shortcut name, like ultra.
4. Enter this for the URL: http://search.ultraseek.com/query.html?qt=%s
5. Click Set, then OK
6. Try a search, like !ultra all terms and enjoy results direct from the Ultraseek site.

You can use this pattern for most Ultraseek-powered sites. Do a search to find the host with the search engine, then use that host with /query.html?qt=%s. Some sites may have customized pages with a different path. Check your browser’s location bar to make sure.

Nice work, Yahoo, and much simpler than A9’s OpenSearch.

Reading The Spider of Doom on The Daily WTF reminded me of a similar story with Ultraseek from years ago, though ours had a happier ending.

Back in 1998 or 1999, we got a call from a customer asking if it was possible that the spider could delete pages from Lotus Domino. 10,000 pages had disappeared overnight, and the only thing they could think of was the evaluation copy of Ultraseek. After looking at the access logs, we figured out that they had a link to “delete this page” on every page. Also, they’d logged the spider in as Admin so that it could access everything. Oops!

I said there was a happy ending? They restored everything from backups, figured out that a link (GET) was a bad idea and changed it to a button (POST), and they bought Ultraseek because they knew that it could access every single page. On our end, we added a rule to never, ever follow that link on Lotus Domino. We all solved our problems and learned something, too.

I decided to do some quick data gathering with a Google account. For a couple of weeks, I did all my searches on Google (I ususally switch between several engines), then transcribed the results. These are posted here, but with the names of a couple of personal acquaintances redacted. They don’t need even the tiny bit of fame or spam they’d get from this blog.

With a (small) set of personal queries, it should be possible to think about approaches for search result personalization. But looking at these, I don’t see any way to improve search with one person’s query stream. I can figure out the context, but there just isn’t enough data for an algorithm to get its teeth into. Also, there are so many different information needs here that any personalization would always be about four steps behind. Several of these are “gold star queries” — one query, get the info.

Want to guess some of the information needs? For extra credit, can you spot the query tactics used?

• enterprise search
• ultraseek
• enterprise search blogs
• macintosh cad house
• coveo, sharepoint
• sketchup
• google terms of use
• google terms and conditions
• how long does cologne last
• perfume storage
• parfum storage
• jean shepherd
• proper name of friend from Rice
• 3.15mm
• 3.15mm lead
• tarptent
• yosemite webcap
• yosemite webcam
• yosemite conditiona
• yosemite conditions
• yahoo term extraction service
• proper name of friend from HP
• proper name of friend from HP (alternate spelling)
• hp narrative history
• bike helmet cover ladybug
• Louis Menand, Tetlock
• verio signature backroom
• palo alto aspca
• palo alto spca
• pets in need
• typekey
• lake huntington weather
• art in the age of mechanical reproduction
• buck sharpening guide
• buck honemaster
• david rains wallace
• adam gopnik
• frank westheimer
• eagle required
• english entropy, shannon
• glass cockpit
• ohlone wilderness map

Most web sites fail to use robots.txt or the robots meta tag, but this rarely causes problems beyond a few junk pages in the index. This is not true for wiki sites. Wikis have tons of extra pages for “edit this” or “show changes”. Those are not the primary content, but a web robot doesn’t know that. The page needs to tell the robot “don’t index me” or “don’t follow my links”. That is what the robots meta tag is for.

I ran into this last week on our intranet. Our spider (Ultraseek) was visiting over 1000 URLs for every page of real content on a wiki!

One of our internal groups put up a copy of MediaWiki. This can show every revision of a page and diffs between each of those revisions. All of these are links, so a spider will follow all the links and find itself with a combinatorial explosion of pages that don’t really belong in the index. This can get really bad, really fast. The Ultraseek spider sent me e-mail when it hit 300,000 URLs on the wiki. After investigating and putting in some spider URL filters, the site has about 300 URLs and about 150 pages of content (it is normal to have more URLs than docs up to about 2:1). There might have been more than a 1000X explosion of URLs — the spider was still finding new ones when I put in the filters.

To get a feeling for the number of URLs, look at the the Names page on MediaWiki then look at all the URLs on the history tab for that page. Yikes!

At a minimum, there should be a robots meta tag with "noindex, nofollow" on the history tab and on all of the old versions and diffs. That would result in the spider visting one extra page, the history tab, but the madness would stop right there. A spider can deal with one junk page for each content page, but the thousand-to-one ratio I saw on our internal wiki is bad for everybody. Imagine the wasted network traffic and server load to make all the diffs.

Some people would suggest using POST instead of GET for these pages. That would be wrong. A request to a history resource (URL) does not affect the resource (a POST), it get the information (a GET). The response should be cachable, for instance. But please mark it "noindex, nofollow".

I don’t mean to pick on MediaWiki, that is just a specific example of a widespread problem. MediaWiki actually gets extra points for sending a correct last-modified header on the content pages. It is sad that something as fundamental as a correct HTTP header gets extra points, but that is what the web looks like from a robot point of view.

WikiPeople everywhere, use the robots meta tag! Robots will thank you, and they will stop beating the living daylights out of your servers. While you are at it, make sure that last-modified and if-modified-since work, too.

John Battelle comments on gaming a recommendation system as a possible explanation for the embarrassing “similar items” result at walmart.com:

We all know about Google Bombing. But Recommendation System Bombing? That’s a new one to me!

Not new at all. We saw this at Infoseek in roughly 1998. As I remember, a vendor came to us with a system and we showed them how easy it was to spam. I’m sure that Firefly knew about it before then. Any kind of adaptive system (ranking, recommendation) is vulnerable. That is probably why personalization keeps reappearing on WWW engines, the login makes it much harder to spam.

Think like a spammer — if the system reacts to you, you can probably hack it.

David Heinemeier Hansson and John Battelle have predictions pointing in opposite directions for enterprise search.

David’s prediction covers all enterprise software:

Enterprise will follow legacy to become a common insult among software creators and users.

John’s prediction is specific to search:

Enterprise search will show us a few new approaches to consumer search, and vice versa. In fact, we may get to the point where the two are often indistinguishable.

I disagree with both of them. Enterprise search and WWW search are more different than they seem. Yes, there is a bit of cross-fertilization, but each one has critical problems that just don’t exist for the other. WWW search engines must fight spam, sell ads, and deal with insane scale. For enterprise search, access to repositories, security, and ease of admin are essential.

Will WWW search and enterprise search be indistinguishable? Yes, for end users. Each one already has a box, a button, and a result list. Underneath, they are quite different, like two antibiotics designed for different kinds of bacteria. They may look the same, but you’d better pick the one designed for your problem.