UNIVERSITY PARK, Pa. — In the final stretch to the midterm elections, polls have come out tracking candidates' likelihood of victory at the local, state and federal levels. According to Tim Servinsky, project manager at Penn State Harrisburg’s Center for Survey Research, polls built with the right questions and asked to the right population can be incredibly accurate and a critical tool for policymakers. However, particularly in the context of political polling, there are important details to keep in mind when reading and interpreting the most recent polling projections.
In the following Q&A, Servinsky, a polling data expert who has been in the field for 20 years, explained what goes into building a credible poll, why polling matters, and what to look for to confirm a poll's accuracy or reliability.
Q: What factors go into building a poll? How do pollsters determine what questions they will ask?
Servinsky: Polls focus on a sample of people from a population of interest and ask them preselected questions. By asking the respondents the same set of survey questions, pollsters can analyze the results to learn more about the views or beliefs of that population. Different factors go into building a reliable poll. Let's say you want to make an inference about all of Pennsylvania — this means your population of interest is Pennsylvanians. You might select 20,000 people from all of Pennsylvania, and that's your target sample. The final group that you get results from — let's say it's a thousand people that complete the survey — would be your sample size.
Deciding a final sample size is usually a balancing act between the goals and budget of the poll, alongside the level of risk you're willing to take with the size of your sample. For example, if I survey 400 people, that results in what's called a margin of sampling error of about 4.9%, with 95% confidence. That means that if I did this survey 100 times, I would expect the results for each question to vary by only about 5% in either direction in 95 of those 100 times. Those other five times would be outlying responses. So, when I conduct my survey, I'm thinking about what this margin of error will mean when I go to interpret my results.
Crafting good questions is also important. The perfect poll questions should be unbiased, succinct, short, and easily understood. Pollsters consider many variables when drafting questions, including the vocabulary and reading level of the sample and whether the questions asked are neutral. To test questions, pollsters use construct validity to help ensure there’s no bias in the questions we're asking, and other measures to make sure the questions are consistent and understandable. We also want to be sure the questions only measure one construct at a time, so we avoid something called double-barreled questions, which measure multiple ideas simultaneously.
Q: Aren’t polls just predictions? Why do they matter?
Servinsky: At the Center for Survey Research, we don’t do political polling, but about 75% of our work is done with state agencies looking to inform policy decisions and make legislative decisions based on public opinion, especially using our omnibus Lion Poll. If we don't ask the public what they think about these decisions, we're going to make assumptions, and that can be dangerous. Decisions must account for the thoughts of people we might not interact with on a day-to-day basis, reaching everyone regardless of their perspectives or opinions — that's what polling is designed to do.
Q: Are there unique challenges to running a political poll?
Servinsky: There's science showing that, when done correctly, polls are remarkably accurate. However, we’re most familiar with political polls that look to predict behavior, which can be difficult due to people’s minds changing at the last minute or over something as unpredictable as the weather keeping them from going to a polling location. It's really hard for someone to predict what they're going to do later today, much less tomorrow or in a few weeks — but if you poll a sample on what they did yesterday, they can report with reasonable accuracy, making these polls extremely responsive and accurate.
Another challenge with polling is non-response bias, which is where people who participate in a survey have different opinions than those who choose not to participate, and we have to make assumptions about those who did not participate. It’s nuances like these that make the process of developing or evaluating political polls, which are focused on people’s behavior, a little bit more difficult.
Q: How can we assess a poll or a pollster’s credibility?
Servinsky: Evaluating polls can be tricky, mainly because not every pollster releases everything about the poll in terms of the methodology. When evaluating a poll, I start by looking at who conducted the poll and how the poll was paid for. If the poll was funded by an organization, I ask if they are reputable, is there publicly available information about them, and did they disclose who the poll was done for or by? These questions can help reveal the motivations for the poll. If the pollster is not willing to release that information, I might want to question the results. Any sort of information that can be evaluated is extremely helpful. But some polls aren't going to report anything, and that's a red flag that should cause skepticism.
American polling organizations rely heavily on guidance from the American Association for Public Opinion Research (AAPOR), which provides professional standards for methodological disclosure through its Transparency Initiative. These standards ensure that anyone can accurately assess the quality of the data collected. AAPOR requires that organizations disclose extensive methodological information about the polls, including how the data were collected, who sponsored the research, the questionnaire used to collect the data, how the sample was constructed and recruited and any use of artificial intelligence.
There are also organizations that assess pollsters, which is helpful when evaluating an unfamiliar group. For example, FiveThirtyEight does great work assessing pollsters' success over time and grading them. It’s a resource I turn to regularly, because they've done the work of looking at how pollsters have performed in the past and whether they've adjusted when they've made mistakes.
Q: How should voters use and interpret polls?
Servinsky: Voters should first look at the face value of what they say — does it make sense? If not, why? In political polling, for example, someone may report that candidate A is ahead of candidate B, with candidate A at 50% support and candidate B at 48%. That seems straightforward, until you look at the survey and see that it has a margin of sampling error of 5% with only 400 people interviewed or surveyed. So, that means the true population results for candidate A could actually be anywhere from 45% to 55%, and candidate B's results would be between 43% and 53% — and those numbers overlap. That is important because it tells me that the survey results don’t necessarily support the conclusion that candidate A is actually ahead of candidate B; if I were to do this survey over, the results might be flipped. Of course, it’s possible that candidate A is ahead of candidate B, but the data we collected aren’t strong enough to support that conclusion. So, we really have to consider those results with caution. There's a lot to consider — you really have to be cautious and read beyond the headline to understand what's happening with the data that were collected.
The bottom line is that, while polls are helpful for voters to keep informed about possible election results, they shouldn’t change your voting behavior or decisions. Polls are designed to track sentiment, experiences, and intention – not tell you what you should believe or do. Every eligible American still has a right and a responsibility to make their own decisions, regardless of what the polls say.