Background This study uses an online survey panel to compare two approaches for assessing ad awareness. The first uses a screenshot of a television ad and the second shows participants a full-length video of the ad.

Methods We randomly assigned 1034 Minnesota respondents to view a screenshot or a streaming video from two antitobacco ads. The study used one ad from ClearWay Minnesota's We All Pay the Price campaign, and one from the Centers for Disease Control Tips campaign. The key measure used to assess ad awareness was aided ad recall. Multivariate analyses of recall with cessation behaviour and attitudinal beliefs assessed the validity of these approaches.

Results The respondents who saw the video reported significantly higher recall than those who saw the screenshot. Associations of recall with cessation behaviour and attitudinal beliefs were stronger and in the anticipated direction using the screenshot method. Over 20% of the respondents assigned to the video group could not see the ad. People who were under 45 years old, had incomes greater than $35 000 and women were reportedly less able to access the video.

Conclusions The methodology used to assess recall matters. Campaigns may exaggerate the successes or failures of their media campaigns, depending on the approach they employ and how they compare it to other media campaign evaluations. When incorporating streaming video, researchers should consider accessibility and report possible response bias. Researchers should fully define the measures they use, specify any viewing accessibility issues, and make ad comparisons only when using comparable methods.