The Hiring Process, Bayes Theorem and American Idol
It seems there is no shortage of blog posts about how to hire great workers. Most focus on the interview technique. If you are looking for a job, you should read as many of these blog posts as you can.
Here are two of the best ones …
In this post, I suggest that if you are trying to hire great workers, the process may be more critical to your success then your interviewing technique.
Your success in hiring great workers depends on how many great workers you have the opportunity to interview and your ability to interview.
Suppose you can identify the top 10% of any group 90% of the time. You get it wrong only 10% of the time. After you interview ten candidates, you’ve narrowed the field to one from the top 10, and one from the bottom 9. If you pick one now, you’re looking at a 50% chance that you pick correctly. You don’t like symmetry? Ok, you only get it wrong 5% of the time. Your odds of picking correctly have only improved to 2/3. Clearly, its best not to choose after a single evaluation.
What can be done to improve the odds of choosing correctly? You can either increase the number of great hires within the population of candidates you interview, or use a process of sequential multiple evaluations. This post will focus on the latter approach, improving the process. Unfortunately, everything we do to make our company attractive to great workers will make it attractive to everyone.
One way to improve your results is to make a serial sequence of decisions that narrows the field such that the number of qualified candidates remaining after each decision increases as a percentage of the total remaining candidates. In other words, the process has the effect of increasing the probability that any of the remaining candidates meets the criteria. A single elimination process is the easiest to illustrate.
In this post, I won’t address the merits of any particular evaluation technique. Pick any you like that is more effective than a coin toss. That’s not a flippant comment; you have to do better than 50/50 for the following process to work. Lou Adler has some interesting things to say about how to improve your interview techniques.
What does this look like in practice? It looks like American Idol. The candidate pool is evaluated and only candidates that pass the first evaluation continue. In successive rounds the evaluation focuses on different criteria that are increasingly more challenging and more relevant.
To illustrate, I’ll use another numerical example. Suppose the goal is to hire someone who is in the top 20% of qualified candidates. For the sake of argument assume all interviewers can pick winners 80% of the time and they mistakenly pick unqualified candidates only 20% of the time.
Because I like easy numbers, suppose 100 candidates are to be evaluated, and we need to pick one from the top 20.
After the first interview, 16 from the top 20% make it to the next round and 16 from the bottom 80% go forward as well. Sixty-eight are sent home. If a choice is made after the first interview, again, we only have a 50/50 chance of getting it right. After the second interview, 13 from the top 20% survive and 3 from the bottom 80% survive. One more round, and we’re done with over 94% probability that we pick a top 20% candidate. Pick any numbers you’d like, apply Bayes theorem, the results will point to the same process. In each iteration we are increasing the prior probability that a surviving candidate meets our criteria.
Most interviewing processes don’t work this way. Usually, a group interviews every candidate and a group decision is made. Invariably, the group is dominated by one or more influential members. Effectively, there is a single decision maker. But we’ve seen that even if an interviewer has a very good technique for selecting qualified candidates, she doesn’t have a very good chance of getting it right when the prior probability of a candidate meeting the criteria is low.
In the single elimination process I’ve described, it’s important that the interviewers do not discuss candidates prior to interviewing the candidate, or prior to making their decision. It’s equally important that they understand the math and don’t rely on the fact that the candidates have survived previous interviews.
What are the downsides to a single elimination process? No matter how you go about it, qualified candidates will not be selected. Therefore, it’s important that candidates understand the process and are treated respectfully.
Thanks to Greg Yut of Supply Beyond for reading and commenting on a draft of this post.
