How do you find out which queries need the most improvement? Look at the ones that are underperforming compared to their expected number of clicks. If you look for low click-through rate (CTR), you’ll find underperforming queries, but they’ll almost all be in the long tail. Improving those won’t make an overall improvement. Click residual is a metric that combines CTR with overall traffic to give a useful number.
To find the queries with the most impact, start with the click count. “Click residual” is the difference between the expected number of clicks and the actual number of clicks. When that is negative, you can see how many times a customer did a search, but wasn’t satisfied enough with the results to click, relative to the overall performance of the search system.
To calculate this, start with the overall CTR across all queries. Then for each query, get the number of query attempts (att) and the number of clicks (clk). Multiply the number of attempts times the average CTR to get the expected number of clicks (exp). Subtract the expected number of clicks from the actual number to get the residual (res) against expected.
exp = att * ctr res = clk - exp
Sort the queries by click residual, descending, so the biggest negative counts are at the top. Now you know which underperformaing queries have the biggest impact on your customers. Sometimes it is a moderately-common query with very low CTR, sometimes it is a very popular query underperforming by just a few points.
At Netflix, where I developed this metric, it highlighted multiple different problems, all important. One week, the worst query was
batman. Why? Because the movie title The Dark Knight did not contain the word “batman”. We added an alias and it was fixed after the next indexing run. Another time, the query
cj7 was a problem. That Stephen Chow movie had not been released in the US yet, but his fans were already searching for it. We asked the movie metadata team to add an entry for the film so subscribers could track when it would be available.
A “query attempt” is usually a query submitted or a search results page displayed, the denominator for CTR. I’ve used a more general term because I’ve used this for character-at-a-time search, where one query interaction is a sequence of queries, like “b”, “ba”, “bat”, “batm”, …
You probably already have the data to calculate this. Give it a try and see what you think.
I think the second formula should be “res = clk – exp”?
Oops, you are right. Thanks for pointing that out! Fixed.