Survey & Instrument Design

Moderate · Methods & Study Design track · ~35 min hands-on + readings and quiz

← Back to the Methods & Study Design track

Use AI to draft and critique survey items — while guarding against bias and bad measurement.

What you’ll be able to do

  • Draft survey items, scales, and skip logic with AI
  • Critique items for bias, ambiguity, and double-barrels
  • Plan cognitive testing and a codebook

Overview

AI can draft candidate items, suggest response scales, and write cognitive-interview probes. It can also produce leading or double-barreled items, so you critique every one against measurement principles.

Good instruments are validated, not just written. AI accelerates drafting; piloting and cognitive testing with real respondents still decide what works.

Practice activities

Activity 1 · Moderate — Draft and critique items

Time ~18 min · Tools ChatGPT Edu

Goal. Generate candidate items, then pressure-test them.

Setup. A construct to measure — e.g., perceived pedestrian safety in your neighborhood.

Steps.

  1. Draft items:

    Draft 6 candidate survey items measuring [construct], with a response scale. Keep them concrete and answerable. [construct]

  2. Critique them:

    Critique each item for bias, ambiguity, double-barreling, and reading level.

  3. Cut or fix based on the critique — your judgment, not the model’s.

Expected result. A vetted short item set with a response scale.

Check your work. Does each item measure the construct, or something adjacent?

Common pitfalls. AI produces leading and double-barreled items confidently. You own measurement quality.

Stretch (optional). Ask for two items at different reading levels and keep the clearer.

Activity 2 · Moderate — Write cognitive-interview probes

Time ~12 min · Tools ChatGPT Edu

Goal. Plan to test whether respondents understand your items.

Setup. Two of your items.

Steps.

  1. Draft probes:

    Write cognitive-interview probes for these two items (comprehension, retrieval, judgment). [items]

  2. Decide which probes you’d actually use.

  3. Predict failure:

    What’s the most likely way a respondent could misunderstand item 1?

Expected result. Usable probes and a predicted misunderstanding.

Check your work. Real piloting validates items — drafting doesn’t.

Common pitfalls. Skipping cognitive testing ships ambiguous items. AI helps draft, not validate.

Stretch (optional). Plan a 3-person cognitive pretest.

Check your readiness

Answer these, then check — your score suggests whether to dive in or skim the readings first.

Useful resources