One of my most popular posts ever was from 2010 on how our search committees deal with applications for TT positions based on my early experience at ProdigalU. The process certainly is very opaque until you are on the other side, so I can understand this. It is now 7.5 years later, and I have some more thoughts on academic searches.
It is still the case that about 2/3 of the applicants are removed from consideration from the CV alone. Those who have the minimum qualifications (like a PhD in the right field) are usually eliminated by lack of productivity. Someone who can't publish reasonably regularly at the level of good society journals in an already established and funded lab is unlikely to be able to do so when starting from scratch no matter how wonderful their ideas are. Also, there MUST be non-review first author publications. There is no minimum number really, since productive means different things in different fields and sub-fields. In my area (a physical science), 3-5 publications from grad school is typical (usually 2-3 first author, the rest as a contributor). Postdocs typically are 1-2 years, so 1-2 first author publications is pretty normal, also usually with some mid-author contributions as well. In my field, one Science/Nature publication does not make up for having nothing else, so trainees who plan to stay in academia that are currently in glam hunting research groups should make sure to look out for other publishing opportunities (like find a collaborator).
For the remaining 1/3, the research plan/proposal is the main difference maker. For a TT hire, we are looking for someone who can both teach our classes and set up a successful research group, where success is ultimately determined by the ability to get funding. This means TT applicants need to demonstrate that they can write clearly and communicate the excitement and novelty of their ideas effectively. In addition, they must have SOME idea about research practicalities (like how much things cost, how much time things take, and what sorts of instrumentation they might have access to). This really comes out more in the interview, but I have seen some application research plans that require access to some unique instrument found at the PhD or postdoc institution, or that rely on unreasonably large amounts of synchrotron time/user facility time. Fit is also important--we already have people at ProdigalU, so we want to hire someone who brings something new, but not so new that they can't find students in our applicant pool.
Reference letters are also important, but usually not the clincher. A good letter will address the specific contributions the applicant made in research as well any other important relevant ways the applicant contributed (mentoring, writing proposals, teaching assistance, etc). This is so useful in evaluation! I find it is often helpful to letter writers to respectfully suggest specific things to address in a letter, because the letter writers are busy people who may not remember to put in something important to the applicant. Letter writers can also help address specific issues that are hard or inappropriate for an applicant to bring up, which can be important as well. Red flags in a letter are taken very seriously (also true for grad student applications), and can definitely sink an application.
The teaching statement matters. Honestly, we don't usually look at the teaching statement until the long list point, but at that point, it is part of the decision. We are a University looking for people to teach our students, so we want someone who takes teaching seriously.
The perceptions of job seekers and search committee members are far apart, but also strangely both correct. When I was looking for a job, all I heard were horror stories about 500 applicants for one opening, and about how people spend years searching and never get a position. Now that I am on search committees myself, I hear about searches where there are hundreds of applications, but practically none are good enough/have the right qualifications to be brought in (this is usually caused by a bad ad, in my opinion), and I hear horror stories of searches taking 3 or 4 years to make a hire.
The funny thing is that both things are true, in my experience. There are searches with 30-40 people under consideration for the interview list (usually 4 or 5 people at ProdigalU), all of whom look really good, but only one of which can be hired. There are also searches where the committee didn't end up finding anyone the department wanted to make an offer to.
The prior experience/training time is getting longer. When I first started, it was pretty typical that most of our pool consisted of applicants in their first postdoc. While postdoc length remains 1-2 years in my field, more and more of our applicants are applying from a second postdoc or from a non-permanent, semi-independent position (like group leader, other sorts of fellowships, or contract positions). So the time between finishing the PhD and starting on the TT is rising. Not as dramatically as in the life sciences, but the experience level of our new hires is also creeping up.
In talking to potential applicants at conferences and/or our own interviewees, some of this is voluntary (commonly this is from people who wanted experience abroad and then wanted to apply from a position in the country), but some is not (people who couldn't get a permanent job and/or perceive themselves as not competitive without a second postdoc). I am not sure what to do about this, but it came up for discussion in our last search as issue in evaluating productivity.
Now I am mid-career
2 days ago