Beyond the Basics: Best Practices for Writing Effective Questions

This article is a summary of a presentation by Emily Belcher and Morgan Leafe, MD, MHA, at the 2024 American Medical Writers Association Conference.

For everyone who loves a good “dos and don’ts” list, Emily Belcher and Morgan Leafe presented Beyond the Basics: Best Practices for Writing Effective Questions. This lively and conversational session covered the art of writing assessment questions for educational activities. The session was part of the Continuing Education for Health Professionals track offered for attendees of AMWA’s 2024 annual meeting. It was intended for writers in the continuing medical education/continuing professional development (CME/CPD) field.

Morgan and Emily are both employed by CE Outcomes, an educational research company that assesses outcomes for CME/CPD activities. Emily is the VP of Research and Analytics and Morgan is the Medical Director. As a physician, Morgan brought the additional perspective of having been on both sides of the assessment process: as question writer and as test taker. The most important takeaway from this session was that writers of questions should be guided by an understanding of how the answers will be used later. Answers = data to analyze.

Appropriately enough for a presentation about writing questions, Morgan and Emily began with some questions. Why do medical writers write questions? What types of questions do they write?

Of course, medical writers in the CME/CPD field write questions for learners to encounter in education activities. (Did the learners arrive with a strong grasp on the subject matter? Can the learners apply new information to critical decision-making points in example patient cases? Etc.) But CME/CPD writers also write questions for analysts to use later when assessing how effective the educational activities were. (What trends in clinical practice did the learners reveal? Might those trends change? Were the learners a diverse group? Were they engaged and satisfied with the activity?)

Multiple choice questions tend to be used most often, but there are roles for open-ended and ranking/Likert scale questions in CME/CPD content. Morgan and Emily discussed the importance of knowing when to use each type of question, and best practices for writing them. They even offered best practices for the demographics questions that used to understand the test takers.

And now, let’s get into those dos and don’ts!

Likert scale questions ask respondents to rank their agreement or disagreement on a numeric scale (of 5 or 7, for example). These questions are a good way to measure nuance in opinion and attitudes. They can also be used to track changing trends.
Do: use these questions to measure a learner’s confidence in their answer to a preceding question.
Do: consider two-dimensional ranking to measure attitudes about items (eg, treatments) on a list.
Don’t: offer “neutral” as the middle choice; “moderate” is often more meaningful to analysts.

Open-ended questions are good to use when possible choices are too numerous to list or when a deeper reflection is called for.
Do: use these questions as a way to gather quotes to use in a report.
Don’t: make the question too vague (eg, “What would you do next?”); Do: focus the answer on a certain path (eg, “What treatment would you choose?”)
Don’t: write leading questions (eg, “Given the success of Treatment X…”)

Multiple choice questions (MCQs) are good for simple recall and in cases where there are objective data. Morgan and Emily provided many dos and don’ts for this type of question because careless writing of MCQs can end up testing learner on their test-taking ability rather than their grasp of the material.
Do: alphabetize the list of answers.
Do: limit the choices to 4.
Don’t: mix “choose all that apply”-type MCQs with MCQs that have a single correct answer.
Don’t: write a correct answer that is longer than all the incorrect ones; equal lengths are best.
Do: craft incorrect answers as plausible-sounding distractors.
Do: avoid negative wording (eg, “Which of the following is NOT…”).
Don’t: make MCQs double as true/false (eg, “Which of the following is true/correct…”).

Demographics questions should be used intentionally, with inclusive language.
Do: allow multiple selections for race, ethnicity, and languages.
Do: provide options for self-identification and “prefer not to say.”

All of these recommendations provided the audience with plenty of “aha” moments and opportunities to jump in with comments. Morgan and Emily ended the session by recommending the following post from the AMWA blog as further reading on best practices for writing questions:

AMWA Blog: Best Practices in Writing Test Items

Jennifer Epler is a freelance biomedical writer and founder of Concise Biomedical Writing, LLC. She earned her MA in science writing from Johns Hopkins University, MS in immunology/cell biology from the University of Minnesota, and BS in biochemistry from the University of Washington.