An interview scorecard describes what evidence looks like at each level.
For each thing you test.
Write it before the job ad.
Decide what the person must do in year one. Then what evidence proves they can.
Then design the question.
If you cannot write the strong answer in advance, the question is not ready.
Most hiring problems are decided the moment a team writes a job ad.
Before anyone agreed what a strong candidate looks like.
The fix is boring and it works.
What does a scorecard actually contain?
Three written levels for every line you score, describing evidence rather than a rating.
Strong describes what the candidate did. Weak describes what they left out.
Now 2 interviewers have something to disagree about that is not taste.
Take “system design”. Useless as a scorecard line on its own.
Useful version:
Strong. Clarified requirements before designing. Estimated scale with numbers. Named a trade-off nobody prompted them about.
Chose a simpler option and said why.
Mixed. Produced a workable design. Needed prompting to consider failure modes. Explained the choice when asked but did not volunteer it.
Weak. Started drawing components without establishing requirements. Could not say what breaks first under load.
Why does writing it first change the questions?
Because bad questions expose themselves.
You find questions whose strong answer is “has seen this before”. That is a recall test you did not mean to run.
You find questions where you cannot write a strong answer at all. Those get cut.
At Zalando this removed about a third of our questions before anyone used them.
The order that works
1. What does this person need to do in the first year? 3 to 5 concrete things. Not “be a great engineer”.
Things like “own the ingestion pipeline”. Or “be the person the team asks about Kafka”.
2. What evidence would convince you they can? For each item, what would you need to see. This is the scorecard.
3. Which question produces that evidence? Now design the interview. This is the last step, not the first.
4. Now write the job ad. It practically writes itself, and it is accurate, which means fewer mismatched applications.
How do you know your scorecard is real?
Give it to 2 engineers who did not write it. Have them score the same recording independently.
If the scores land more than one level apart, the scorecard is not specific enough. Fix it and repeat.
This is the calibration exercise you should run anyway. It takes an afternoon.
What does this fix downstream?
Four things. Debriefs stop being arguments about taste and become arguments about evidence, which converges.
Feedback to candidates becomes possible, because you wrote down what you were looking for.
New interviewers ramp faster instead of guessing at an unwritten bar.
And bias narrows, because a written standard is harder to bend unconsciously.
Bias narrows rather than disappears. A written standard is still a human one.
The number that justifies the afternoon
Unstructured interviews explain about 4% of on-the-job performance.
Structured technical ones explain 26%.
An afternoon on the scorecard is the difference between those 2 figures.
On every hire you make afterwards.
Where do teams give up?
They write the scorecard for 1 role and never for the next one. Keep them, because most roles are variations.
The second scorecard takes an hour and the fifth takes 20 minutes.
Treat the first one as a project, not a template, and you pay every time.
Then you stop doing it.
Want this built into the team, calibration included? That is what SmartHiring does.
Audit at 500 euro, workshop sprint at 2,500.
Common questions
How long does writing an interview scorecard take?
An afternoon for the first role. An hour for the second, 20 minutes by the fifth. Most roles are variations on one you have written. The cost people resist is the meeting where the team agrees what the job is. That meeting is the point.
How many lines should a scorecard have?
Four to 6, one per thing the person must do in year one. A 14-field rubric does not get filled in, and engineers are right to refuse it. If a line does not map to something the role requires, cut it. Do not score it out of habit.
Can you reuse a scorecard across levels?
Reuse the lines, rewrite the evidence. Senior and staff are tested on the same dimensions with a different bar under each. Naming an unprompted trade-off is strong at senior and expected at staff. Writing both columns side by side also settles levelling disputes fast.
What if the team will not agree on the scorecard?
That disagreement was already in your loop, hidden inside 3 different bars. Surfacing it in a meeting is cheaper than in a debrief after 5 candidates. Write the version you can defend. Run 2 loops with it. Then revise from what the scores showed.
The short version
Decide what the person must do. Decide what evidence proves they can. Then design the question.
If you cannot write the strong answer in advance, the question is not ready.