Click Residual: A Query Success Metric

How do you find out which queries need the most improvement? Look at the ones that are underperforming compared to their expected number of clicks. If you look for low click-through rate (CTR), you’ll find underperforming queries, but they’ll almost all be in the long tail. Improving those won’t make an overall improvement. Click residual is a metric that combines CTR with overall traffic to give a useful number.

To find the queries with the most impact, start with the click count. “Click residual” is the difference between the expected number of clicks and the actual number of clicks. When that is negative, you can see how many times a customer did a search, but wasn’t satisfied enough with the results to click, relative to the overall performance of the search system.

Continue reading

Measuring Search Relevance with MRR

At Chegg, we test the relevance of our search engine using customer data. We extract anonymous information about queries and clicks, then use that to automatically test improvements to search. When our search engine provides results that better match what our customers are choosing, we call that an improvement.

The most basic measurement of search quality is clickthrough rate. For each search page that is shown, how often is at least one search result clicked on? This fits well with overall website conversion, showing how search works in getting a visitor to a successful result.

Continue reading

Query Box as Confessional Box

We have a very small number of very long queries that are timing out in the search engine, so I was digging through the logs looking at long queries. I found this.

“something that i think will never happen just did and i dont like it one single bit, no not even a drop, and i wish it never did because it just ruined my life and i just want to watch indianna jones”

199 characters. I think I’ll set the limit somewhere over 200 characters. I’d hate to make their day worse.

Search Evaluation by Kitten War

On a search engine mailing list, the topic of simple A/B testing between search engines came up. This can be between different implementations, different tunings, or different UI presentations. The key thing is that users are offered two alternatives and asked which one they like better. One bit of information, this one or that one. If you’ve been to the Kitten War site, you’ll understand why I call it “kitten war testing”. Others may call it a “beauty contest”. They are wrong, of course.

Continue reading

Odd Cataloging Decisions at Palo Alto Library

I really wonder about the cataloging at my local library. I was looking for books by Jo Walton and I noticed that a series by her was spread across two areas, both arguably wrong. First, Ha’penny is a sequel to Farthing, so they really should be shelved in the same section. Second, they are both alternate history novels from a fantasy author, and I wouldn’t look in either Mystery or Fiction for them.

Check out this screenshot from their search on July 2nd.

PA Library webcat screenshot

Big hint, Tor has been a major SF&F imprint for over 25 years.

I’m looking forward to Palo Alto’s choice for Half a Crown, the next book in the series. Maybe DDC 737 (Numismatics)?

I reported this to the reference desk at Main. Let’s hope they fix it.

The fun doesn’t stop there. I’m currently reading The Fall of the Kings. That was shelved in YA Fiction, where it doesn’t even belong. I read a fair mix of books, from Westerfeld to Dostoevsky, with plenty of YA, and this just doesn’t fit in the Teen collection. It is long (476 pages of small print), there are no teenage characters, nearly every chapter has sex and/or violence, it is quite slow moving, and it helps if you care about university politics. I read Valiant immediately before, and that book has half the word count with double the action and four times the dialogue, plus teens, fairies, drugs, NYC, and a massive betrayal by mom. Valiant belongs in the Teen section. Dreamhunter belongs there. The Fall of the Kings does not.

I thought that maybe, just maybe, they put it in YA because the most recent book in the series, The Privilege of the Sword, has a 15 year old girl as the main character and can easily be considered YA, so they decided to keep them together. Sorry, they shelved that one in Science Fiction.

I know that strictly defining Science Fiction (or Fantasy) is nearly impossible, but they must be able to avoid howlers like this. Yes, Michael Chabon has written fantasy (Summerland) and SF (The Yiddish Policemen’s Union) but it might as well be shelved in the mainstream section because that is where people will look for him. On the other hand, Jo Walton has written a sword and sorcery trilogy and a book set in Victorian England where the nobility are dragons. Where would you look? Heck, ask Jo Walton. Her answer to the FAQ “What genre is Farthing?” reads “It’s an alternate history mystery. I think that makes it SF.”

Hmm, Palo Alto also shelves The Lord of the Rings and Jacqueline Carey’s Kushiel series in mainstream Fiction. Bizarre. The Kushiel books are also published by Tor. Can we just shelve all the Tor in SF, as a stopgap?

Searchers Punt Early

Amidst the usual creative spellings and phonetic thrashing around (“napolinian dynomite”) that I see in the search logs, I’ve noticed a small but distinctive subclass of searcher behavior. People type as much as they are sure of then, instead of making a mistake, they stop typing and submit the fragment to the search engine. Said that way, it kinda makes sense, but search algorithms are tuned for complete, if imperfect, attempts instead of exact prefixes.

Here are some selected examples from logs.

  • Frank Gehry
    • frank g
  • The Adventures of Baron Munchausen
    • the adventures of baron
    • baron munch
    • baron munc
    • baron mu
  • The Last Mimzy
  • Final Fantasy
    • final fan
  • Apocalypto (lots of misspellings)
    • apocalypse
    • apocalypso
    • apacal
    • apoca
    • apoc
    • apo
    • ap
    • rudy (yeah, that one is for real)
  • Ratatouille
    • ratatou
    • ratato
    • ratat
    • rata
    • rat
    • ra
  • Koyaanisqatsi
    • koyaanisq
    • koyaanis
    • koyaani
    • koya
    • coonskin

The “coonskin” query may seem bizarre, but that is exactly what phonetic search is tuned to solve. Unfortunately, people don’t seem to be that brave or that deluded.
Querying for “mimsy” instead of “mimzy” is a typical, supported phonetic match.

The Koyaanisqatsi example is the one that tipped me to this other behavior, with additional evidence from Frank Gehry and Baron Munchausen. Note how they get the double-“a” in Koyaanisqatsi, but freak out at the “q” not followed by “u”. They are almost there, then punt because they are not sure what to type next.

Is this behavior learned from auto-completion, from texting completion, or is it caused by our reluctance to make mistakes? Maybe it doesn’t matter, since I need to help these folks regardless.

This is probably best addressed with auto-completion, not matching in the engine.

Do all-stopword queries matter?

Many search engines don’t index “stopwords”, words that are very common and have little meaning by themselves. The stopword list is often just the most frequent words in the language: “the”, “be” (and its inflections), “a”, “of”, and so on.

Search engines that index all words like to show off searches for “to be or not to be”, because stopword elimination can remove every word in the phrase. Of course, no one really searches for “to be or not to be” because we all know where it came from.

Are there any real titles that are all stopwords? Does this matter? I’ve been indexing movie titles, and found a more than a few that are 100% stopwords.

The last one isn’t a traditional stopward, but think about the number of “click here” links on the web. It is a web stopword, for sure.