210 applications. First of all, for all three searches, we had 200+ applications. Initial evaluation and grading is done by each member of the committee independently. Then, we come together to make a "long list" and then after more consideration, a "short list". We can throw away at least 1/3 of them right away due to lack of qualifications (no PhD, no postdoc*, wrong field, wrong sub-field*, or application incomplete). I should note that we do keep applications in the no postdoc or wrong sub-field if they are SUPER AMAZING, but that is not most of them. When we write the ad, we have some type of expertise in mind that we are looking for to fill out the department, move into a new area, or cover a needed area of teaching. Sometimes we will hand off applications to another (more suitable) department that has an opening. As for postdocs, we generally require at least one. In the past (before my time), the department has hired someone with a deferred start so they can complete a postdoc first, so it isn't a total waste of time to apply without one, if you have something good lined up. This is very, very rare though. A surprising number of applications are not complete. The most common missing thing is one or more reference letters (we ask for three). We will often over look one missing letter (we assume the writer flaked out), and contact the listed writer if we need more information. If more than one letter is missing, we assume it is the candidate.
140 applications. We score all the remaining applications in A, B, or C. Applications scored C are removed from consideration. Applications scored B are (briefly) reviewed a second time in case we missed something, especially if the decision was not unanimous. Usually, the A's can pretty clearly be separated from the B's and C's. This cut is made mostly on CV and letters of reference. We look primarily at publication record--we like to see first author papers from both PhD and postdoc. We also like to see some signs of collaborations (like non-first author papers). We look at where the candidate is publishing (Nature/Science papers are not required, but 17 publications in Joe's Journal of Microscopy won't cut it). In the searches I saw, I'd say around half of the remainder were C's.
70 applications. Now it gets tougher. We need to separate the A's from the B's. At this point, we consider the proposed research plan, as well as previous track record. We are looking for a colleague who can 1) be successful in our department and 2) bring something to improve the department. If the research plan is completely unrealistic ("I'll cure cancer in 5 years!" or "I need the entire contents of my massively successful advisor's lab to get started") or incomprehensible, the applicant is a B, no matter how great their CV is. If the applicant can't write a decent research plan, we have grave doubts they will be able to write a decent grant proposal. When I am scoring the applicants, I look a little at citations, especially for publications more than a year old, but only with a super-quick citation search (definitely not comprehensive in any way). This is just to get an idea of how relevant the prior research is, particularly for sub-fields I don't know that well. We don't generally care too much about where the applicant did their PhD/postdoc as long as the productivity is good. It helps to have at least one recommender who is known someone on the committee by reputation, if nothing else. Admittedly this is harder if someone is coming from a less well-known university, but that is where conference networking comes in. We don't consider teaching experience at all at this point. If there are any weirdnesses (no letter from advisor, unusually long postdoc, gap in CV, etc) we expect them to be addressed in the cover letter or by one of the reference letters. The ratio varies, but let's say the applications are split 50-50 between A's and B's (not too unusual these days with so many people chasing so few jobs).
35 applications. These are the people we are considering for invitations to interview (the "long list"). If any are missing letters, we contact the listed recommenders. At this point, we are considering everything we asked for in the application, but most important is research experience and research fit with the department. On the committees I was on, at the first meeting(s), we went down the whole list, and graded every candidate as A,B,C, or F. At the next meeting, we considered if we should promote any of the B's and worked on a rank order for the A's. We now make the "short list" after much debate. At this point, most/all of the candidates are well qualified, and the distinctions are based on research fit, and how much people like the research proposals. This is where the luck part comes in. We'd probably have a great pool if we invited any of these 35 candidates, but we can only invite 5. So being in the right place at the right time is important. If we already have a great person who studies widgets, we are not going to interview another person who does really similar research, no matter how great they are when we have so many other great choices available.
At this discussion, we will also think about whether the number of women (as determined by name, generally--we don't see the EE cards) on the list is representative of the pool to make sure we didn't accidentally overlook anyone. If we could identify other underrepresented minorities, we would do the same, but that is pretty much impossible from the applications. This is NOT AA. We are doing this to make sure we didn't overlook a file due to unconscious bias (as has been shown by many, many studies and anecdotes). In both searches where I was on the committee, this was not a problem.
We rank all of the A's and decide who to invite to campus. This was not too contentious, which I found pretty surprising. In both cases, we converged on a list of 10 or so, and then spend the most time whittling that down to 5. We generally picked the "best" person (on paper) doing a particular technique or studying a particular subject, and in general, only one person from a given research group is invited out. This does help prune a little bit.
5 applications. These are the people we bring to campus. At least one of them will bomb. We hope that one or more will dazzle us and make the choice difficult. The ranks start out fresh, and we let the candidates place themselves. I'll talk about interviews in a future post or two. We do not notify anyone else of their status, in case we need to revisit the pool.