Using AI to Write Quiz Questions Without Sacrificing Quality

AI can write good quiz questions when you give it a measurable objective, source material and clear rules for plausible distractors and feedback, and when a human reviews every item before it's used. Without those, AI tends to produce recall questions, giveaway answers and occasional factual errors.

Used well, it removes the biggest bottleneck in assessment design: producing enough varied, realistic items for practice, spaced review and question banks.

Why AI-generated questions often disappoint

Ask a model to "write 10 quiz questions about our expense policy" and you'll typically get:

• Recall questions ("What is the maximum meal allowance?") instead of judgment questions.
• Distractors that are obviously wrong or silly.
• "All of the above" options and clues hidden in answer length.
• Occasional facts that aren't in your policy at all.

These aren't AI problems so much as prompting and review problems. The same issues show up in human-written quizzes.

Step 1: Start from the objective

Give the model a measurable objective. "Given a submitted expense claim, decide whether it complies with the travel policy and identify the reason if it doesn't" leads to much better questions than "expense policy."

Step 2: Ask for scenario-based items

Scenario questions put the learner in a realistic situation and ask what they'd do. A prompt template I use:

Using only the policy below, write 5 scenario-based multiple-choice questions for [audience]. Each question: a 2–3 sentence realistic workplace situation, a question asking what the learner should do, one correct answer and three plausible distractors based on common mistakes. Distractors must be similar in length and style to the correct answer. No "all of the above" or "none of the above." For every option, write one sentence of feedback explaining why it's right or wrong, citing the policy section. [Paste policy]

Step 3: Make distractors earn their place

The best distractors reflect real misconceptions. Ask SMEs, "What do people usually get wrong here?" and give those answers to the model as distractor sources. A distractor nobody would choose doesn't measure anything.

Step 4: Review every item

My review checklist:

1. Accuracy: is the correct answer actually correct according to the source?
2. Single best answer: could an expert argue for a distractor?
3. Alignment: does the question measure the objective, or just a detail?
4. Clarity: is the stem readable without the options? Any double negatives?
5. Clues: is the correct answer longer, more specific or grammatically different?
6. Bias and inclusivity: names, cultural references and assumptions in the scenario.
7. Feedback: does it teach, or just say "Incorrect"?

Expect to rewrite or discard some items. Even so, reviewing is far faster than writing from scratch.

Step 5: Pilot and analyze

For anything high-stakes, such as certification or compliance, run questions with a small group first. Look at:

• Difficulty: if nearly everyone gets it right, it may be too easy or have a giveaway.
• Discrimination: do stronger performers get it right more often than weaker ones?
• Distractor use: if no one picks a distractor, replace it.

Build banks, not single quizzes

Because generating variants is cheap, create several versions of each question with different scenarios. That supports retakes, spaced practice and reduces answer-sharing. In my AI-assisted design workflow, question banks are one of the biggest time savers.

Sources

• Thomas M. Haladyna, Steven M. Downing and Michael C. Rodriguez, A review of multiple-choice item-writing guidelines for classroom assessment, Applied Measurement in Education (2002)
• Henry L. Roediger III and Jeffrey D. Karpicke, Test-enhanced learning, Psychological Science (2006)