How We Should Interview
The Evolution of the Technical Interview
Ignorance played a crucial part in me getting an internship offer at Microsoft 36 years ago. I had no idea what Microsoft's interview process was like. There was no internet full of cautionary tales or sample interview questions. I prepared the same way I had prepared for every other interview up until that point. I read all the information they mailed me ahead of the interview and took notes of the relevant questions I had from that reading. I was ready for all the standard questions: "Why did you choose Computer Science?" "Why do you like programming?" "Tell me about a challenge you faced and how you overcame it." "Where do you see yourself in five years?" But within fifteen minutes of being on Microsoft campus, I became keenly aware that this was going to be a very different experience.
My day began with my Julie, my recruiter. She told me I was going to be meeting with several engineers from different product teams throughout the day. Then Julie closed her introduction with a key line, "Today you're going to be presented with all sorts of different technical problems. Just stay relaxed and remember to think out loud." She wished me luck and then sent me on my way. I got in my rental car to drive across campus to my first interview in building 15. In that short drive I was thinking, "What kinds of technical problems? Why do I need to relax?"
My first interviewer greeted me in the lobby and we took a one minute walk down the hall, during which all standard pleasantries were exchanged. We reached a conference room and walked inside. He asked me to have a seat. Then he handed me a pad of paper and a pencil and said, "You're standing on the face of the earth. You walk a mile due south, then a mile due east, and then a mile due north and you end up in exactly the same place. Where are you?"
I said, "I've heard this before. It's the polar bear / penguin question. You're a polar bear at the north pole. A penguin at the south pole can't walk any further south."
He responded. "Right. Where else?" I grinned, realizing that the fun was now beginning.
"North pole is one. Can't be the south pole. How about the equator?" I drew an equator line on the paper and then drew a slightly slanted line down, then a flat line across, and then a slightly slanted line up. I looked at the resulting trapezoid and say, "Nope, equator doesn't work."
The interviewer encouraged me, "Great, you've knocked out the obvious cases. Keep going."
I drew a circle to represent the globe, then drew the polar bear's path at the top of the circle: a curved line down the side of the circle, a curved line across the circle, and a curved line up the other side of the circle. I drew a curved line across the center of the circle and put an X across it. I added an X at the bottom of the circle. Now I had a visual to work with. "Where else?" I thought out loud. In the middle of the northern hemisphere, I drew another triad of curved lines, down, across, up, and reasoned about the resulting shape. "Once you're south of the north pole, anywhere in the northern hemisphere is going to have you ending on the same latitude that you started, but in a different spot on that latitude."
Then I turned my attention to the southern hemisphere, where I drew yet another triad. "The same looks true for the southern hemisphere." I stared at the page for a minute, then said, "Well, the south pole was never an option because you can't walk a mile due south from the south pole. What if we start a mile north of the south pole?" I drew a line down the bottom part of one side of the circle. "Hmmm, no, because you can't walk east from the south pole." I studied my drawing a bit more, then said, "aha." I drew a curved line across the bottom of the circle, saying, "The latitude that is a mile north of the latitude where the circumference of the globe is one mile. You walk a mile south from anywhere on that upper latitude, and then you'll walk east right back to where you just were. Then your walk a mile north will land you back in the same spot on that upper latitude." I drew what essentially looked like a lasso on the paper. One line down and a circle around the globe. "So an infinite number of spots, from anywhere on that latitude." I looked at my paper again and added my conclusion, "Very neat. Had never thought about that before."
Without a pause, the interviewer said, "Very good. Where else?" I wasn't exactly shocked, but his comment had me thinking (this time just to myself), "No way … there's more? More than infinity?" I then said out loud, "The only two shapes that work are the triangle at the north pole and this lasso in the southern hemisphere." I took another short look at my paper and said, "Aha, it's an infinite number of latitudes in the southern hemisphere, all below that first latitude ring. A mile above 1/2 mile circumference. A mile above 1/3 mile circumference. 1/4 mile. etc." I then thought of the more generalizing statement, "A mile above any latitude where walking a mile east will result in you returning to the same place."
The interviewer said, "Nice work."
I smiled and said, "That's interesting."
The smile was short lived as the interviewer immediately asked, "Pick any procedure you want and explain it to me."
Me: "Okay; driving a car from point A to point B."
Interviewer: "Okay. Go."
Me: "First, you get in the driver's seat."
Interviewer: "Where is that?"
Ah ha, this is going to be fun.
Me: "A car has four wheels and one or more doors. You can identify the front of the car by two or more clear glass lamps, which are called headlights. If you see red glass lamps, you're at the back of the car. Standing in front of the car and looking at these headlights, walk around the car to your right and open the first door. This is the driver's door."
Interviewer: "How do I open it?"
And so it continued.
My brain was now awake. And my technical communication was now warmed up. That was the first 30 minutes of my day of technical interviewing. And I now understood what Julie had been telling me.
The Leetcode Interview
The day proceeded through a gauntlet of five interviewers, with the last technical challenge being, "Write a function that counts all the set bits in a byte." After going through several iterations with this last interviewer, he seemed satisfied that we had exhausted all the possibilities for this challenge. Then he sat back in his chair and asked, "What questions do you have about Microsoft?" Our last 20 minutes of cordial conversation was like all the other interviews I had had before. But the five hours that preceded that conversation were a blur.
I left campus, with a very tired mind. I felt exhilarated and exhausted. A week later Julie called me with my internship offer.
This was a success story of surviving the classic Microsoft "leetcode interview."1 And, as I said, my ignorance of what was coming played a big part in this success. I was sure that if I knew in advance what was coming, I would have done a bunch of studying trying to sharpen my coding skills and problem solving skills ahead of the interview. And I would have worked myself into a very stressed state before ever making it to campus.
This approach to interviewing certainly over-filters, declining candidates that aren't ready while also declining some candidates that are capable, but simply aren't able to perform in this environment. It is a stressful environment. And, when I later became an interviewer in this process, I was always repeating that line Julie said to me, "Today you're going to be presented with all sorts of different technical problems. Just stay relaxed and remember to think out loud." I was always on the lookout for signs of stress during the interview, and I would try to talk the candidate down.
Of all the interviews I ever did, there is one case that stood out as the most stressed candidate I had worked with. He appeared very relaxed when I met him, and he warmed up nicely over a conversation about the self-initiated projects he had taken on2. When I presented the technical challenge to him, he was quick to the whiteboard, and was thinking out loud as he walked me through the approach he was taking. Fifteen minutes later, he had successfully worked through the solution and was sitting back down. I said, "Nice work." He smiled. And then I looked down and watched his nametag sticker fall off of his shirt. The poor guy had sweated so much that his sweat unstuck his nametag. I added more encouragement. "That thinking out loud was right on. We do these problems so that we can get a glimpse into how you problem solve. Just keep doing that the rest of the day and you're gonna do great."
The over-filtering and the stress-inducing were understood undesirable side effects of this interview process, but this was how we were doing interviews across Microsoft. And my own interview experience formed the basis of how I approached all of the interviews I conducted at Microsoft for years. Until I found a much better way to interview.
My First Job Opening
I followed the typical progression for the hiring process. First, I was a lunch interviewer, which was 3/4 social and 1/4 technical. Next I became an interviewer, where I had one hour with the candidate to present challenges and assess. Then I became the hiring manager that in addition to owning one of the candidate's interview slots, also reviewed all of the interviewer feedback to make a hiring decision. And then I got to the point of creating a position that I was also hiring manager for.
The first position I ever created was for an additional software engineer on my newly formed UI Controls team. I had two developers from my previous org, and then I made a pitch to management that I needed another engineer for the growing scope of our work. Amongst the pool of candidates that applied, one of them was Mark.
Mark worked in our developer support organization, and his area of focus had been Windows User Interface, with a further specialization in UI Controls. His portfolio of work included a Controls Test application that he had taken the initiative2 to write. It was an app that allowed him to play with any control, bringing it into the app's play space and then sending any series of commands to that control. This tool allowed him to quickly reproduce any problem with controls that his support customers were having. And, then, he could provide the minimal repro steps to the Windows Controls development team. I had spent my first years on that Windows Controls team, and could appreciate the time savings of Mark's work, as the repro scenarios we would typically get were very broad and the first 10-20% of our investigation would always be narrowing down the repro scenario for quicker debugging and resolution.
Before I ever met Mark, I saw his work. I installed his Controls Test app and spent an hour playing with it. I looked at the app's interface to see how intuitive it was. I went through all of the controls to see how well the app functioned. And then, knowing Windows Controls pretty intimately, I tested out various "dark corners" of the controls space. His app held up beautifully through all of my analysis.
Mark had scheduled an informational with me, so during that informational I was able to ask him some follow-up questions about his app, and at the end I asked him to send me the source code so I could take a peek. I spent another hour going through the code to see how he had designed his app. The code was very easy to follow, thanks to good documentation and great code organization.
By the time interview day came for Mark, I had already seen his work. In my one hour interview slot with Mark, I wasn't evaluating 20 lines of code written by Mark on a whiteboard in 15 minutes. I had already seen thousands of lines of his code, coherently arranged and fully functional. We spent most of our interview talking about what led him to building this application, and what follow-up ideas he had based on this app to further accelerate his developer support work. And then I dove into his collaboration experience on his current team.
Mark was relaxed and articulate. All of the other interviewers, following the standard interview format, gave him hire recommendations. And I also saw him as a strong hire. I extended an offer to Mark and he accepted. And Mark and I worked together, directly and indirectly, over the next decade. Mark became one of "my 15", the group of 15 employees that I collected over the course of my tenure, telling them "I will work with/for you anytime."
Contrast Mark's interview experience to my interview years earlier. In a standard interview, we have five hours of time with the candidate to evaluate the candidate's abilities and assess their readiness for the role. But with Mark, I had three hours of additional context. I got such a complete picture of his skills and his strengths, that it made the hiring decision obvious. And the success of that hire in terms of strength of output and length of stay proved that we had made a great hiring decision. The hiring process for Mark became my gold standard for how to approach recruiting. I wanted more context on all of the candidates I interviewed, looking through their resume for links to projects they contributed to.3
The Success of Interns
If three additional hours with Mark created so much better of a picture of the candidate, what would three months of exposure give you? This is why Microsoft kept growing their internship program. They found huge success in this program, as measured by (a) the number of intern candidates that were given full time offers, (b) the percentage of those offers that were accepted, and (c) the long-term tenure of these full-time hires.
An internship is basically a three month interview. There is a lot time for us to see the candidate in the real-world scenario of collaborative software engineering. We see how they work with the team. We see how they approach their work, how they design their algorithms, and how they produce their code. We see them testing and debugging their work to get it ship-ready. And, all the while, we see how well they are working with the people and the processes of the team. This provides us with a very convincing picture of this candidate's readiness to be a full-time employee.
In my coaching of interview candidates, I always remind them that interviews are two-way. Don't go into the interview in a reactive state ready for whatever is presented to you. Build your own list of questions and areas to explore, and make sure that you get the answers you need to form an opinion about this role. This two-way nature of interviews is also incredibly enhanced by a three month internship. Not only is the company getting a very clear picture of how well you work with the company, you are finding out just how well this company's work and culture align with your passions. This is the reason that interns coming back full-time have such long-term tenures with the company.
Open-Source Hiring
I managed seven teams whose products were developed in open-source repositories. This was a wonderful exploration for us, with tons of learnings. We brought improvements to the GitHub tools we were using, and the open-source conventions helped us tune our own internal processes. As we got better at being good stewards of our repos, we were rewarded with increased community engagement in our repos.
It was completely magical to have additional engineers working with our own engineering teams to accelerate the progress of our products. Each repo provided the rules of engagement for that repo, detailing all conventions and processes that we followed. So the "engineers in the wild" that contributed to our repos were able to provide aligned and lasting additions to our codebase.3
One of the happiest accidents that came from this environment was when it came time to hire for any one of these open-source-facing teams. Whenever we had a job opening, we would post that opening in the repo itself. "Do you enjoy working in this repo enough to make it your full-time job? If so, please apply."
This was the Mark Experience and the Intern Experience rolled into one. For any candidate that we hired from the open-source community, we weren't assessing them just based on what they demonstrated on whiteboards and pads of paper over a five hour day. We saw tangible contributions to a large codebase. We saw them following the processes of the team. We saw them collaborating with the team via design reviews and code reviews.
What Comes Next?
Based on this open-source experience, I had an ongoing conversation with several of the leads of these projects about how we could take our learnings and improve the standard interview process. I never got to the point of pitching that to Microsoft recruiting, so instead I'll share it here for broader consideration.
Ahead of the interview, we should ask the candidate which project of theirs best highlights their abilities. Share that project's repo in their application, along with a short explanation of why they selected that project and where they contributed to it.
Then our interviewers, each of whom is assigned to probe different skills (technical expertise, collaboration, problem solving, technical communication, maintainability, etc.), would dive into the repo ahead of the interview to assess. They would build questions for the candidate based on their exploration, and that would be what was covered on the day of the interview.
Some challenges with this approach is that the project the candidate selects may be too small to demonstrate much. Or perhaps it's too complex of a codebase for the interviewers to reason about in the limited amount of preparation time before the interview. For such cases, the interview can just fallback to the older approach of presenting new problems to the candidate in the interview.
The more that we can make the interview an extended dialogue about a larger body of work, the more complete picture we'll have about how the candidate works in the target environment.
Footnotes
How Do I Stand Out? | "Your Portfolio" section | "Initiative" attribute
How Do I Stand Out? | "Your Portfolio" section