13 item-writing flaws that let people pass a quiz without knowing the answer
Updated
A multiple-choice item is flawed when someone can get it right without the knowledge it is meant to test, or get it wrong despite having it. Most flaws fall into a short list, and every one of them can be spotted by reading the item carefully. Here they are, with examples and fixes.
The list below follows the item-writing guidelines researchers such as Thomas Haladyna have refined over decades of assessment research. You do not need a psychometrician to apply them. Read each item once as a learner who skipped the course, and ask: could I get this right from the wording alone?
Flaws that give the answer away
These are the most damaging, because they reward test-wiseness instead of knowledge. A quiz full of them measures who has taken a lot of quizzes.
What should you do if a customer reports a data breach? A) Ignore it B) Email IT C) Log the report in the incident tool within one hour, tell your manager, and do not contact the customer again until Security replies D) Call them back
A customer says they received another customer's invoice. What do you do first? A) Log it in the incident tool B) Apologize and resend the right invoice C) Ask the customer to delete the email D) Tell your manager and wait
The key was the only option with detail. Make every option roughly the same length and specificity, and write distractors that are things people really do.
Which form is used to report a safety incident? A) The expense form B) The safety incident report C) The timesheet D) The leave request
Which form do you complete after a near miss on the warehouse floor? A) Hazard observation card B) Incident report C) Risk assessment D) Maintenance request
"Safety incident" in both stem and key is a free point. Remove the echo, and use options from the same family.
- 3. Grammatical cue. The stem ends in "an" and only one option starts with a vowel, or the stem is plural and only one option is. Write options that all fit the stem grammatically, or end the stem with a question mark instead of a sentence fragment.
- 4. Implausible distractor. One option is a joke ("Throw the laptop out of the window"). Each implausible option turns a four-option item into a three-option one. Use real mistakes from real people instead.
- 5. Absolute terms. Options with "always" or "never" are usually wrong and test-wise learners know it. Avoid them, or make sure they are sometimes the key.
Flaws that make an item hard for the wrong reason
Which of the following is NOT a step in the returns process?
A customer returns an item after 45 days with no receipt. What is the next step?
Negative stems test reading attention more than knowledge, and they are rarely how a question comes up on the job. If you must use one, bold the NOT and make sure every option is a plausible step.
- 7. Double-barreled stem. The item asks two things at once ("When and how should you escalate?"), so a learner who knows one half can still miss it. Split it into two items.
- 8. Ambiguous item. Two options are defensible, or the stem is vague enough that the answer depends on an assumption. Ask a colleague to answer it cold; if they argue for another option and have a point, rewrite.
- 9. Overlapping options. Ranges such as "1 to 5 days" and "5 to 10 days" share a value, so two options can both be right. Make options mutually exclusive.
- 10. Unbalanced options. Options in mixed formats, tenses or lengths draw the eye to the odd one out. Keep them parallel.
Options that do not measure anything
- 11. All of the above. A learner who spots two correct options can choose it without reading the third. It also makes the other options partly true, which muddies feedback. Replace it with a single best answer.
- 12. None of the above. When it is the key, you only learn that the learner can recognize wrong answers, not that they know the right one. Use it rarely, and never in a "best answer" item.
- 13. Trivial recall. The item asks for a definition, a date or a policy number when the objective is to make a decision. It can be technically flawless and still measure the wrong thing.
Check the level, not just the wording
An item can avoid all thirteen flaws and still test the wrong thing. If the objective says learners will apply a policy, an item that asks them to recall the policy name sits two levels too low on Bloom's taxonomy. The quickest fix is to put a realistic situation in the stem and ask what the person should do. That one change usually removes the trivial-recall flaw and raises the level at the same time.
A five-minute review routine
- Cover the stem and read only the options. If one stands out by length, detail or format, fix it.
- Read the stem and the key together. If they share a distinctive word, rephrase.
- Read each distractor and name the real mistake it represents. If you cannot, replace it.
- Ask whether the item puts the learner in a situation or asks them to remember a fact. Compare that with the objective.
- Have someone answer it without the course. If they score well above chance across the quiz, the quiz is measuring test-wiseness.
If you have more than a handful of items, Assessment Doctor runs this review for you: paste the items, mark the keys, choose the level the quiz should test, and it names each flaw with the exact words at fault, rates the level and rewrites the item.