Making Interview Questions

Why Your Interview Questions Keep Failing

I spent about three years building out technical interview processes for a mid-size SaaS company before I realized most of our questions were measuring the wrong thing. We were testing whether candidates could solve puzzles under pressure, not whether they could actually do the job we hired them for. That realization came after I watched a candidate who clearly knew their stuff choke on a question that had nothing to do with the role they were applying for. It was a waste of everyone's time. The process of Making Interview Questions is deceptively simple on paper. You have a role. You need to assess people for it. You write questions and see how they answer. But the gap between that simple summary and actually producing questions that separate competent engineers from anyone who memorized LeetCode solutions is where most teams fail.

Starting With the Job, Not the Question

The first step nobody follows correctly is identifying what you actually need the person to do day-to-day. Not the aspirational version. The actual version. I keep a spreadsheet of every task someone in the role touches in a typical week, weighted by how often it comes up and how much it matters if done poorly. Then I look for the gap between what candidates claim they can do and what they can actually demonstrate under realistic conditions. One specific problem I ran into was with senior backend positions. We had candidates who could explain distributed systems theory flawlessly but froze when asked to debug a real production issue. I rewrote our debugging section to include an actual sanitized production log with a subtle performance regression baked in. The candidates who could spot the issue in under five minutes were the ones who stayed. The ones who spent twenty minutes building a whiteboard architecture diagram without reading the logs first didn't make it past round two, and that was exactly the filter we needed. It took me about two weeks to build out a realistic scenario that felt authentic without exposing any actual customer data. The workaround was dumping real error patterns into a synthetic dataset generator and letting it run overnight to create plausible but fake incidents.

The Structure That Actually Works

A good interview question has three components that most people conflate into one vague prompt. There is the surface problem you present, the skills it actually tests, and the failure modes you are looking for in the answer. When I write a question, I fill out a simple card that lists those three things before I even draft the actual wording. Without that, you end up with questions that sound impressive but measure nothing useful. The card format looks like this on my end. The prompt itself is usually 2-3 sentences describing a realistic work scenario. The skill being tested is written as a specific competency, not a vague trait like "problem solving." The failure mode is what a bad answer looks like. This last part is the one most teams skip, and it is also the most important part. If you do not know what a weak answer looks like, you cannot tell the difference between a candidate who understands the material and one who is bluffing convincingly.

The Math Behind Question Selection

There is a reason why teams that run four well-designed questions per interview round consistently produce better hiring decisions than teams that run twelve shallow ones. Each question costs time. The interviewer's time, the candidate's time, and the time spent calibrating scores across panel members afterward. A four-question interview usually takes about forty-five minutes total. A twelve-question version runs close to two hours and the quality of each individual assessment degrades significantly by question seven because everyone involved is mentally fatigued. I track question reliability using a simple metric. After each hiring cycle, I note whether the candidate's performance on each question predicted their actual job performance at the ninety-day mark. Questions that consistently correlate with on-the-job outcomes get kept. Questions that do not get discarded regardless of how clever or interesting they seem. Over two years, this cut our question bank from roughly eighty questions down to about thirty-two, and our offer acceptance rate on the hires that came through that process went up because we were evaluating the right things instead of just more things.

The Hidden Cost of Behavioral Questions

Here is a counter-intuitive point that most hiring guides ignore. Behavioral questions like "tell me about a time you failed" have extremely low predictive validity for actual job performance unless they are anchored to a specific competency from the role. A generic behavioral question is basically a personality quiz in disguise. Candidates who are good at self-reflection and framing their stories well will score high regardless of whether they can do the work. This is not a minor issue. It is a systematic bias that favors certain communication styles over actual capability. The fix is straightforward but requires discipline. You tie every behavioral question to a concrete work scenario that mirrors actual job demands. Instead of asking someone to describe a time they handled conflict, you give them a written scenario involving a product decision with two stakeholders who disagree, and you ask them to walk through their reasoning. The scenario replaces the memory exercise. The reasoning replaces the rehearsed story. Another pitfall is the difficulty calibration trap. A question that seems appropriately challenging to the person who wrote it is almost always too easy for someone who has been doing the work for five years or too hard for someone who is competent but early in their career. I use a scoring rubric with five tiers for every question, and each tier describes what the answer should look like, not just a point value. The tiers are calibrated against actual work samples from top performers in the role, not against other people's guesses about what good looks like.

When This Approach Breaks Down

There are scenarios where investing time in custom interview questions is not worth it. If you are hiring for an entry-level role with a high volume of applicants, the return on building custom questions is negative. The time you spend writing and validating a question will not be recovered through better hiring decisions at that level. For those situations, a structured skills assessment platform or a standardized coding test gives you acceptable predictive validity at a fraction of the effort. Custom questions also break down when you do not have enough historical data to calibrate them. If you have only hired two or three people for a role, you do not have enough signal to know whether your questions actually predict success. Running a small question bank through multiple hiring cycles before relying on it for important decisions is the only way to avoid building something that feels rigorous but is actually random in practice. I learned this the hard way when a question we thought was a great filter turned out to have zero correlation with performance because we had not yet gathered enough data to validate it properly. There is also a scaling limit. Building and maintaining a quality question bank requires someone to treat it as an ongoing responsibility, not a one-time project. Every role change, technology shift, or team restructuring means old questions become less relevant. I have seen teams let their banks rot for two years and then wonder why their interview process felt stale and inconsistent. A quarterly review cycle where you retire questions with low predictive value and add new ones tied to current work realities keeps the whole thing from becoming decorative.