Survivorship Bias and Opinions

Many of our opinions are formed based on our personal experiences. That’s natural, but we are subconsciously influenced by survivorship bias.

Sep 24, 2023

Thank you for reading our work! If you haven’t yet subscribed, please subscribe below:

If you would like to support us further, please consider sharing and liking this article!

As Nominal News grows larger, we will be able to make this a full-time project, and provide more content. If you would like to suggest a topic for us to cover, please leave it in the comment section.

If you have subscribed to Nominal News (thank you!), you may have seen in the welcome email or the about page that I have completed a PhD in Economics. Out of all the social science degrees, the field of Economics for PhDs has a unique, little known (unless you’re an economics PhD or know one) job process. Having experienced this process, I will share with you a bit of information on this process and how it is influenced by survivorship bias. Then I will link to how this bias can be seen everywhere – from opinions on immigration to job advice.

Brief Aside – What is Survivorship Bias

We briefly discussed survivorship bias in our post on layoffs in the tech sector. Survivorship bias occurs when an analysis focuses only on individuals that have passed a certain ‘selection’ process, and ignores the ones who did not. The headline image is a famous example of survivorship bias occurred during World War II with statistician Abraham Wald.

Bombers that returned from flight missions were analyzed for where they had bullet holes from anti-aircraft fire in order to establish where to put additional armor on the planes. One common suggestion was to strengthen the parts that had many bullet holes. Wald, however, recommended adding armor to the areas of the plane that did not have any bullet holes, because the planes that were hit in those areas simply did not return. The planes that returned with bullet holes ‘survived’, meaning the areas they were hit were already well protected.

Economics Degree – the “Job Market”

When starting a PhD degree in Economics at a US university, on day 1 you will be told about the Economics Job Market. The Job Market is basically the end of the PhD degree for any economics graduate student. If you are or were in a PhD Economics program, you know about the job market. If you know a PhD Economist, you’ve almost certainly heard about it, as it is a very stressful period for them. So what is the “Job Market”?

When starting an Economics PhD, just like with any degree, the aim is to land a job at the end of the degree. Most often, but not always, the job PhD students initially aim for is becoming a professor at a university (other typical roles are in government policy or central banks). Moreover, rankings, whether implicit or explicit, of university economics departments occasionally also matter to some PhD students – that is, they'd prefer to be a professor at a "top" university. Naturally, there is an issue with these preferences – there are more PhD students than available jobs in research universities, not to mention the dearth of jobs available at “top” research universities.

In order to match PhD candidates with universities, the economic profession created what is informally referred to as the “Job Market” – a mostly centralized job application process. By November of each year, every university that wants to hire a PhD Economist posts their job listing on the website of the American Economic Association (AEA). PhD candidates apply to these positions by submitting their main research paper called colloquially the "Job Market" paper (JMP). Then, by about mid-December, candidates are notified if any of the universities are interested in interviewing them. The interviews are conducted at the annual Allied Social Science Associations (ASSA) meeting in the first week of January. After the interview process, the next step is to receive a ‘fly-out’ – this is where the candidate goes to the university that is interested in them and presents their Job Market paper at a 90 minute seminar. This is also an opportunity for the PhD candidate to get to know their future colleagues, the Economics department and the University.

Summarizing, the “Job Market” has three key critical goals for a PhD student:

Receive interviews.
Receive flyouts, after successful interviews.
Receive the job offer, after a successful flyout.

Limited Attention

How does this process play out in practice actually? In the first stage of the application process, as mentioned above, each PhD candidate submits their research paper, which can easily be longer than 40 pages. Since there are no costs of applying for the listed professor positions, PhD candidates are usually encouraged to apply to all job postings. But if you are the university that received all these applications, how do you decide who to interview?

The decision to hire someone for a professor position is decided by the current professors at the economics department of the university that is hiring. This means that applications will be entirely reviewed by professors. Now, given how lengthy the applications and research papers are, it is impossible for these professors to read the research of all applicants. So naturally, there are other ways economic departments tackle this problem. One way is by the rank of the university of the applicant. For example, a top university will typically only hire PhD Candidates from other top universities. Additionally, if you are a lower ranked university, you will also ignore PhD Candidates from much higher ranked universities, believing that these PhD Candidates are unlikely to accept an offer from them since the PhD candidate will likely get an offer from a higher ranked university. This can create a situation where certain PhD Candidates might end up being "too good" and "not good enough" for any university. Regardless, even filtering by the PhD Candidate’s university rank doesn’t narrow down the application pool sufficiently.

Along with the application submitted by each PhD candidate, each candidate must also get three recommendation letters from their current professors – their “Advisors”. These letters are actually the critical piece of whether you are likely to get an interview. Professors at the hiring university mainly focus on these recommendations, and moreover, they look for certain keywords. Having seen some of these recommendation letters, they often explicitly list things such as – “this candidate is great for every university, but the top 50”. Thus, a top 50 university will simply ignore this candidate's application – they will not look at their research and will not give this candidate an interview.

Additionally, the Advisors of PhD candidates can also directly reach out to their colleagues at other universities, and let them know that they should consider giving their PhD candidate an interview. Advisors, however, also have a reputation to uphold, especially since they work and interact with the people who will be reading their letters. Since, naturally, not everyone can be the best or even great, Advisors prefer not to over-recommend students. This is why certain recommendations, like the example mentioned above, explicitly state that this student is not good enough for the top 50 universities, because in a future situation, if this Advisor now says this candidate should be in a top 50 university, the professors reading this letter of recommendation will take this recommendation more seriously.

Mechanism and Incentives

The Job Market system, therefore, influences the behaviors of Advisors and the professors reading the recommendations, who are the ultimate decision makers. This leads to a lot of ‘unspoken rules’, which can be seen to be a consequence of how the Job Market is constructed. An Economics department, via a professor-run “Placement Committee”, has to coordinate on strategies on how to undertake the recommendation process of their PhD Candidates. For example, it might not be optimal for two Advisors of two different PhD Candidates to describe their students as the ‘best in the department’, because that could be interpreted by the hiring university as a confusing signal. Additionally, recommending multiple PhD Candidates that have similar research topics (for example, two PhD Candidates that do monetary policy) to the same university might also be seen as confusing by the hiring university, as it is difficult for the hiring university to determine who to hire. Since the overall aim of the Placement Committee is to place their students, and placing in ‘higher’ ranked departments is seen as better, the Placement Committee has to heavily coordinate this process, which influences outcomes for PhD Candidates.

The Job Market system also has several incentive issues. The hiring process is very time-consuming for professors at hiring universities – reading papers, conducting interviews and then meeting candidates consumes a lot of time.There are no, at least to my knowledge, immediate benefits for the professors participating in the hiring process. Many professors would prefer to spend the time on research, especially for professors that are on tenure track, as publications are extremely important in the tenure decision. Thus, professors do not want to spend much time going through applications, and prefer to listen to the opinions of Advisors.

Advisors, on the other hand, are also not significantly rewarded for having their PhD Candidates get into universities (one benefit is having your PhD student as a future colleague at that department), and they receive even fewer benefits if the PhD Candidates enters the private sector. These issues create significant problems and put a lot of pressure on PhD Candidates in terms of managing their career prospects. Some of this pressure manifests itself in significant levels of depression among economics PhD Candidates, with 36% of students experiencing moderate or severe symptoms and only 26% of students finding their work meaningful (Bolotnyy, Basilico, Barreira, 2022)

The Job Market, on paper, sounds like it may be a good method of matching PhD Candidates with university jobs. However, the Job Market is heavily reliant on the opinion of your Advisors and your department. This means that for many PhD Candidates, their outcomes really depend on the beliefs and actions of a few professors (their Advisors), about whom the PhD Candidate does not really know much when they start in their PhD studies.1

Survivorship Bias

So why is there so little interest in improving or changing the Job Market process? This is where survivorship bias comes in. Many of the professors at universities who are either Advisors or on hiring committees went through the Job Market process when they themselves were PhD Candidates. Since they are now university professors, it means they succeeded – or survived – the Job Market process. To them, the Job Market process is an effective process and works well enough that it does not need to be changed – at most, a few slight changes. This is a natural response.

As a PhD Candidate, you mainly interact with these professors – the survivors. On the other hand, the PhD Candidates that ended up with different results when completing their PhD degrees are not easily observed as they are no longer at universities. Their opinions and thoughts of the Job Market process are not incorporated into the discussions on whether any changes are needed.

Why am I writing about this

This topic originally came to me around the July 4th holiday – US Independence Day. There were a few tweets and comments that stated that the US is great for immigrants and much better than other countries – France being named as one worse example due to the protests that occurred in the summer. These comments were made by people that immigrated to the US and used their example as a success story.

Now I do not know whether the US is better for immigrants than other countries (it may very well be). But one thing I know is that there are many cases and situations where immigrants or refugees did not have good experiences with the immigration system, with many cases of deportations, family separations and other issues. The many visible public comments, on the other hand, clearly had very positive experiences. They survived the immigration system and are now enjoying the benefits of the US. Their views and perspectives are not representative of the wider experiences, and thus making such sweeping statements seems unfounded.

Practical Examples

We observe survivorship bias in many other contexts too. One example to keep survivorship bias in mind is when seeking a new job. When applying for a new job, you are likely to speak with employees who are already at the firm. Employees, especially employees that have been at the firm for a long time, will have a skewed view of their firm, since not only did they survive in the firm, but also chose to be there. This will not give you a representative view of the firm. It would be best to also speak with people that left the firm (or at least get a sense for how many have left). Since they didn’t survive at the firm, you will get a better picture of what it is actually like at the firm.

Conclusion

Survivorship Bias can occur everywhere. People that are extremely successful are often quick to forget that just because they survived, it does not mean that the process that led to their success is correct. Even here, as a Susbstack writer, if one day I am able to do this full time, my perspective of the Substack platform will be influenced by the fact that I survived. Writers that stopped posting articles and quit will not have their voices heard in terms of improving the Substack platform.

There’s not much we can do about Survivorship Bias beyond being aware of it. It is important to take it into account when receiving advice or an opinion, knowing that the advice might not be reflective of the whole reality. Similarly, when giving advice, our views and perspectives might differ from what many experience, because of the limited amount of data we observe.

Article:
Alice Evans
covers a recent paper that studied people’s perspective on whether the world is ‘zero-sum’ – i.e. your success is my loss. Findings suggest people have become more ‘zero-sum’ recently.
Article/Video: More coverage of child poverty in the US by
Economics with Dr. A
who discusess it in both written and video form.
News: The Federal Reserve kept interest rates fixed. Why we don’t expect inflation to rebound based on most recent economic models.

If you enjoyed this article, you may also enjoy the following ones from Nominal News:

News: The US Credit Downgrade (August 10, 2023) – On August 2nd, 2023, Fitch downgraded the credit rating of the US. What does it mean and what does economics tells us about the US debt burden.

The Work from Home Illusion: Is It? (July 15, 2023) – The Economist covered several work from home studies. The conclusions The Economist drew from them, however, were quite wrong. What did these studies actually say, and why work from home might be the optimal form of work.

The usefulness of the GDP measure (February 7, 2023) – GDP is very often the focus of countries and businesses. What are its limitations and why institutions and economists are considering designing a new measure called “Beyond” GDP.

Although a PhD Candidate can choose Advisors, often the set of potential Advisors is limited due to the research agenda the PhD Candidate is interested in, as the Advisor should have a similar research profile. Moreover, PhD Candidates have limited knowledge on how their potential Advisors place students, and finding out is not easy.

Nominal News