By Howard Schuman Questions And Answers In Attitude Surveys Experiments On Question Form Wording And Context Quant Paperback

Why Your Survey Results Keep Being Bullshit

I ran a national survey once where asking respondents whether they supported "tax relief" versus "tax cuts" produced opposite majorities for policies that were functionally identical. The only difference was the word choice. That moment made me realize I had been designing surveys wrong my entire career, mostly because nobody in grad school actually taught us how words work on people. Howard Schuman and Stanley Presser wrote a book that covers exactly this.

By Howard Schuman Questions And Answers In Attitude Surveys Experiments On Question Form Wording And Context Quant Paperback

documents decades of their experimental work showing how small changes in question design can flip answers, distort data, or make respondents bounce out of the survey entirely. It is not a fun read. It is dense, experimental, and full of tables. But it is also the only book that properly explains why your Likert scales are lying to you. The core premise is straightforward. People do not have fixed, internally coherent attitudes that simply need to be extracted from them. Their answers are constructed in real time based on how the question is framed, what words are used, what comes before and after it, and what they assume you already know. This means two questions describing the same policy can produce contradictory results, and both results are technically accurate to the respondent's mind at the moment of answering. Most people in survey research act surprised when this happens. Schuman and Presser predicted it fifty years ago and proved it experimentally.

What You Actually Need To Know About Wording Effects

Wording effects fall into a few categories, but the practical ones matter more than the academic labels. Leading words are obvious. Asking if respondents support "helping the needy" produces different results than asking about "welfare." But the subtler ones are where you get burned. Response scale structure changes everything. A five-point scale forces different cognitive work than a seven-point scale. The midpoint itself is a choice. When you give people a middle option, roughly a third will pick it, and those people rarely mean "neutral" in any meaningful sense. Some are genuinely uncertain. Some are avoiding commitment. Some are just tired. Schuman and Presser show that collapsing the midpoint into a binary scale does not fix this. It just hides the uncertainty and makes your data look cleaner than it is. Context effects are another trap most researchers walk into without noticing. If you ask about satisfaction with life in general before asking about satisfaction with a specific program, the answers change compared to when you ask the specific question first. Order matters. So does the emotional tone of previous questions. I learned this the hard way when a client asked me to run a political attitude survey and we placed three heavy questions about immigration policy directly before questions about healthcare spending. The correlation between those two domains disappeared almost entirely compared to our pilot. Swapping the order fixed it. That was in 2018 and I still think about it sometimes.

How To Actually Use This Book Without Falling Asleep

The book is organized around experimental findings, not step-by-step instructions. It will not tell you to use double-barreled questions less often because it assumes you already know that. Instead it gives you dozens of replication-ready studies showing exactly how various manipulations shift responses. My workaround for actually using it has been to treat it as a reference manual rather than something you read cover to cover. I keep it open on the second monitor while I draft survey instruments and search for the specific wording or format issue I am wrestling with at that moment. There are sections on question format, response categories, question context, and interviewer effects that map directly onto workflow decisions. The question format section alone will save you from about forty percent of the validation problems that show up after fielding starts. I typically spend a day going through it when building a new instrument, pulling out the experiments relevant to my design choices, and noting which recommendations contradict each other so I can make an informed tradeoff rather than guessing.

The Counter-Intuitive Stuff Beginners Miss

Most people assume that making questions simpler is always better. Schuman and Presser show this is not true. Simple questions often fail because they lack the contextual grounding respondents need to construct an answer. A slightly more specific question that includes a concrete example can produce more reliable data than a short abstract one. Complexity is not the enemy. Ambiguity is. Another thing nobody tells you: response options are not neutral containers. The presence or absence of a single category can shift the entire distribution. Adding "don't know" as a distinct option rather than letting respondents self-select it reduces noise and gives you cleaner missing data codes. The book documents this across multiple domains. It feels obvious in retrospect. It rarely comes up in method courses. Interviewer effects are where the book gets really uncomfortable for anyone who runs phone surveys. Interviewer race, gender, and even vocal tone can shift responses on sensitive topics. If you are doing self-administered web surveys, you might think this does not apply to you. It does. Just in a different form. The equivalent in online surveys is pacing and progress bar placement, which changes dropout rates in ways most teams never measure.

When This Book Will Not Help You

It is from 1981. Some of the experimental paradigms are dated. Online administration introduces response dynamics that did not exist when the research was conducted. The book does not address skip patterns, adaptive questioning, or mobile survey design. You will need to pair it with more recent work on computerized survey methodology if your instruments are web-based. Presser and Schuman updated some of their findings in later papers, but the core experimental logic remains useful even when the medium has shifted. There is also a practical limitation. The book is an academic monograph, not a consulting guide. It will not give you template surveys or software recommendations. If you need something that walks you through building an instrument from scratch, you are better off with Fowler's Survey Research Methods as a companion. Schuman and Presser explain why your questions behave the way they do. Fowler explains how to assemble them. I have reommended this book to people who are just starting out in survey design and to people who have been doing it for twenty years. The twenty-year veterans usually appreciate it more because they have already made enough mistakes for the experimental evidence to start making sense. If you are currently getting inconsistent results across waves and cannot figure out whether it is a real attitude shift or a wording artifact, this is the book to throw at the problem. It will not solve everything. But it will stop you from pretending your data is more stable than it actually is.

You can find the paperback through academic resellers and Amazon. The ISBN is 0803913407. It goes for around forty dollars new and considerably less used. Worth it if you are designing surveys for a living. If you only ever send one survey a year about customer satisfaction, skip it and read a blog post instead.