Hiring is Broken: What Do Developers Say About Technical Interviews?

11 min readJun 19, 2019

We read tens of thousands of Hacker News (HN) comments, so you don’t have to. Read on for a summary of our latest research results [pre-print].

Prelude

Let’s begin with a technical interview problem. Consider the following coding question from LeetCode, an online platform for preparing software development candidates for interviews:

You can solve the problem interactively. The provably optimal solution to this question — called Kadane’s algorithm — is described in Bentley’s 1984 column, Programming Pearls. The column presents various solutions to this question with cubic, quadratic, and linear time complexities. Bentley notes that the problem “is really a ‘toy’ — it was never incorporated into a system.’’

Before going further — and regardless of your coding proficiency — we’d like you to spend a few minutes and take a stab at this question.

Well, how did it go? Did you find an O(n) solution?

Developers within the LeetCode community report that — within the past six months — this coding question has been used in technical interviews at well-known software companies such as Apple, Amazon, Microsoft, Google, Facebook, and Uber.

Technical Interviews

A technical interview for software development consists of one or more stages within the interview life cycle. It begins with an initial screening of the candidate, usually conducted over the phone or through an online coding site, such as CoderPad. Depending on their performance, the candidate may be invited for an on-site visit. This on-site visit consists of a series of one-on-one interviews (each 45 minutes to an hour, over a period of half a day to several days) with engineers, and sometimes managers. The interviews primarily focus on technical coding or algorithms, either on the whiteboard or using a simple text editor on a computer. In other words, technical interviews are primarily a test of the candidates’ problem-solving or “analytical ability’’. And if all goes well, the candidate can expect to receive an offer.

For hiring managers, technical interview have considerable appeal. First, hiring managers are able to ask questions directly from the companies’ question bank, instead of having to design their own interview questions. Second, the format purports to reduce variation between interviewers and teams, since candidates can be evaluated through objective scoring criteria. Finally, the interview process becomes scalable: new interviewers are straight-forward to train, and these interviewers can interchangeably ask coding questions to any software engineering candidate. All of these, in theory, result in a more-or-less standardized and meritocratic technical interview pipeline.

The collective experiences of interview candidates, however, appear to tell a very different story — even a cursory glance reveals that “technical interviews are broken’’, that they are an “antagonistic’’ form of high-pressure “whiteboard algorithm hazing’’ that have “nothing to do with real day-to-day developer work’’, are “humiliat[ing] professionally’’, assess candidates through an “algorithm question lottery [of] luck and chance’’, and require substantial “upfront investment’’ from the candidate to learn the “cultural norms necessary to get themselves into a desk at a technology firm’’. Still others argue that technical interviews may even “promote exclusion and discrimination, serving only as a barrier to entry for qualified underrepresented candidates’’.

The goal of this work is to take meaningful, personal, and yet disjoint anecdotes such as these — and amplify them into a principled theoretical foundation to support research towards improving technical interviews in software development. To that end, we conducted a qualitative study in which we obtained over forty six thousand authored comments from Hacker News — a social website for software practitioners focusing on computer science, software development, and entrepreneurship — pertaining to the topic “interviews.’’ We framed these comments through the analytical lens of small stories— stories of their personal experiences and their past events — and through thematic analysis identified concerns software developers have about technical interviews. The contribution of this work is a state-of-the-practice synthesis of concerns from the Hacker News community about technical interviews for software developers, reflected through their own words.

Our analysis of Hacker News identifies several concerns in current software engineering practices with regards to technical interviews. Though hiring managers justify these practices as being meritocratic, our findings suggest that candidates perceive these practices as subjective, arbitrary, unnecessarily stressful, non-inclusive — and at times — demeaning to their sense of self-worth and self-efficacy. We propose guidelines to make hiring more inclusive and equitable without sacrificing interviewing effectiveness, for example, providing candidates with explicit evaluation criteria in advance.

Concerns from Candidates

Relevance

“Building a great and useful app rarely requires Herculean feats of logic and puzzle solving,’’ says HN18944553. Indeed, adds HN18945198, “the number of times I’ve seen things like dynamic programming come up in a real world application are vanishingly small.’’ HN18944499 describes an interview experience: “I’m a data scientist, and Google asked me to sum all values of nodes at each height of a tree. I had to implement the tree, bfs, and the algo (which was easy once you have bfs) in a 25 minutes, minus any talky time. BFS is not something I thought about much in the last 5 years, and quite frankly could care less about. I got stuck when I knew I needed ‘something’ to finish implementing BFS, but couldn’t remember and the google interviewer offered no help.’’

These expressions capture the discrepancy between the skills needed for performing a software development job and the problems they are asked to be solved to get that job. HN18946686 elaborates, “This then becomes representative of your experience in spite of that fact that you are never likely to be confronted with that kind of problem with that kind of time-frame. Whatever is on your CV, and whatever you can say about what you’ve learned over the years, becomes totally irrelevant in the face of that.’’

A noteworthy undercurrent we found within the discussion of relevance is view that developers should be able to solve technical interview problems “from scratch’’ or view them as enjoyable: “If you’re a programmer, you like to solve puzzles,’’ says HN11248511. HN18944989 explains their belief, “I wrote a custom HashMap for Java, used BFS and DBS on graphs, wrote a top down custom parser, custom string searching algorithms to solve real business problems. I would rather work with someone who knows how neural networks really work rather than who knows pytorch or keras, because they can be learned rather easily. In some industries knowledge of algorithms can be more valuable than knowing a myriad of frameworks that change every few years any way.’’

Finally, HN18944164 offers that relevance could vary based on experience: “There are career stages. For candidates just out of school or with little experience asking algorithmic questions totally makes sense. For more senior candidates (who actually progressed to the next stage, not just spent a lot of years), the questions become more real-life, more open ended and with more than one (or sometimes none) correct answer.’’

Anxiety

“The problem is that interviews are high stress affairs,’’ says HN6251756. They continue, “this is great when you’ve got to climb a tree to get away from a tiger. This is horrible if you are trying to demonstrate your ability to function mentally. It does not matter how reasonable your questions are. If this is what someone faces, you do not get an accurate picture of how good anyone with interview anxiety is. And a lot of people suffer from this.’’ HN6253596 also describes anxiety that arose during an interview with their own team, “I once interviewed for a job within the company I already worked for [with] my existing boss, and a colleague.’’ Then, “I was asked a question about ASP.NET ViewState which I drew a complete blank on. I just couldn’t wrap my ahead around the question because of nerves. It was only when my boss reminded me that this was something I had actually taught him and my colleague only 6 months ago that my nerves cleared, I relaxed, and then I could suddenly think clearly again.’’

Having stress can cause disruptive effects on memory which can negatively affect candidates’ performance during interviews, even with people they already know: “The interview process creates a dynamic relationship between interviewer and interviewee that never exists between colleagues and bosses,’’ continues HN6253596. One possible cause of anxiety during the interview is being watched and judged by the interviewer. HN6252459 says, “I also get very anxious during exams and always make silly mistakes. It saddens me to see some interviewers on this thread that naively believe that the kind of stress a person feels during an exam (especially one where the tester is sitting in front of you and watching your every move) is the same kind of stress you might experience while trying to fix some problem in front of your computer.’’ Some developers were not sympathetic with the concern of anxiety and HN6253513 believes the problem is fixable, “I think the real problem is in your head: your anxiety about job interviews is sabotaging something you’re otherwise perfectly good at,’’ and continues by adding “You need to be more zen about this. More laid back, relaxed, confident, or something like that.’’

The full set of concerns, methodology, and limitations can be found in the research paper pre-print.

Hiring is Broken: What Do Developers Say About Technical Interviews? Mahnaz Behroozi, Chris Parnin and Titus Barik, to appear at VL/HCC 2019.

Guidelines

We can offer a few guidelines that may alleviate some of these concerns.

Guideline I — Use rudimentary questions for screening.

When applying for software development positions, candidates may have limited notice before having to participate in a phone screening. Consequently, they may not have the time available to adequately prepare for this interview (Practice). The goal of the phone screen shouldn’t be to fully assess the candidates’ capabilities. Instead, the interview should be a rudimentary filter to assess whether the candidate can program at all, through what Atwood describes as “blindingly, intentionally easy’’ questions. A second goal of the phone screen is informational: to share what the team does and to identify whether the candidates’ advertised skill set fits these needs.

Our suggestion is that algorithms at this stage of the interview be no more complicated than the Rainfall Problem — a programming task that has been used in a number of studies of programming ability. The original wording of this problem is simple enough, though variations exist: “Write a program that will read in integers and output their average. Stop reading when the value 99999 is input’’

When complex algorithms are requested in interviews, hiring managers may be unnecessarily excluding candidates simply because of their time commitments, and not because of their lack of technical qualifications (Missing Out). If hiring managers need further evaluation, they can propose a small take-home project, where the candidate has more flexibility and autonomy in how they conduct this work.

Guideline II — Share the interview description in advance.

To make technical interviews more equitable for all candidates, the hiring manager should share the details of the interview procedure with them. This includes not only the length of the interview, but also the types of questions that they will be asked. If certain resources are useful for being successful in the interview, these should be recommended to the candidate. Importantly, the scoring rubric for the technical interview should not be opaque to the candidate. Opaque hiring criteria gives an unfair advantage to those with prior interviewing experience, and can be frustrating to candidates who are unaware of the otherwise unwritten rules particular to the organization (Affect).

Guideline III — Offer alternative interview formats.

Allow candidates to opt-out of certain interview formats or make minor adjustments to the format, without reducing their ability to assess problem-solving skills. For example, a simple change to existing interview format could reduce the anxiety associated with public performance in front of an interviewer, for example, by offering the candidate the opportunity to initially think about the problem in private (Anxiety). Moreover, having to think-aloud while performing a cognitively demanding task has been shown to inhibit task performance.

Similarly, while some candidates may prefer conducting the technical interview on a whiteboard, others may feel more comfortable working within an integrated development environment on the computer, or find it more natural to explain and sketch a problem on pencil-and-paper(Affordances).

Guideline IV — Use a real problem.

Several comments on Hacker News were critical of the use of artificial, puzzle-like problems that did not reflect the types of engineering tasks that candidates would do in their day-to-day software development activities (Relevance). Such problems were also perceived as biased towards junior candidates just out of school, as these junior candidates were more likely to have recently solved these kinds of homework-style problems. For senior developers, our suggestion is to use technical interview problems that have real-world scenarios, and resemble programs that the candidate would actually write as a software developer within the team. For example, consider the purely academic problem of performing a depth-first traversal of a graph structure versus framing this problem as determining the order in which to install packages from a package management system when provided with an input of a hierarchy of dependencies. In the latter scenario, it may even be sufficient that the candidate can simply identify that this problem is an instance of depth-first search, without needing them to actually write code.

Guideline V — Solve problems as colleagues, not as examiners.

Not all candidates can effectively solve technical algorithms in a fast-paced, high-pressure whiteboard setting — nor do these candidate often work in such stressful situations in their day-to-day software development jobs (Anxiety). Instead, the conversation between the interviewer and candidate should be less of an antagonistic interrogation, and more of a conversation in which both the candidate and the interviewer work together to solve the problem. Preferably, the candidate should be interviewed by the team that they intend to join, so that they can see experience first-hand how they would work with one or more of their potential teammates.

The Future of Interviews

Mike Hearn writes:

Despite all that I must finish by observing that regardless of our best efforts in the industry, hiring is still largely random. Well designed interview processes make it slightly better than random, which is why we do them.

How we interview people will continue to be a messy problem. But by at least documenting and framing these problems, we are beginning to investigate and build a more scientific view into the process — making the experiences better for hiring managers and candidates alike. This work is only a small step in this direction.

We have some interesting results in development. Here’s a sneak peak.