How do Americans reconcile their optimistic beliefs about chances for social mobility and economic equality with the day-to-day reality of remarkably unequal distributions of economic rewards and mobility opportunities? This question is the driving motivation behind new research by Assistant Professor of Sociology, Dr. Siwei Cheng and PhD student Fangqi Wen. Their project, “Survey Experiments on Americans’ Perceptions and Attitudes about Economic Inequality and Social Mobility” received IHDSC Seed Award funding in Fall 2016.
Leveraging the power of experimental design and large online surveys, the researchers probed Americans’ often-times paradoxical and puzzling beliefs, attitudes, and perceptions about inequality and mobility. Cheng and Wen discussed their interest in studying this topic and the utility of survey-based experiments. This Q&A was edited for clarity and length.
What led you to begin studying how Americans feel about economic inequality and social mobility?
Cheng: Our interest in studying how Americans feel about inequality and mobility stems from a puzzle of two seemingly contradictory observations. On one hand, there has been a sizable and growing empirical literature suggesting that America’s economic inequality has been rising in recent decades, and that social mobility has remained low and remarkably stable in the United States. On the other hand, however, Americans seem quite optimistic about mobility chances in the country.
We suspect that how Americans feel about these social issues may be a complex matter that involves different and sometimes contradicting beliefs about different dimensions of inequality and mobility. Hence, we propose that one approach to better understanding this puzzle is to conduct a set of survey experiments that help distinguish the public perceptions about these various dimensions. For example, in one of our survey experiments, we proposed a new questionnaire design to distinguish between the public beliefs about the overall chances of economic mobility and those about the relative chances of mobility between children from rich and poor families.
Wen: Social scientists have conducted a large number of empirical studies to explore the causes and consequences of economic inequality and social mobility. We believe that understanding how individuals perceive inequality and social mobility is also very important, because some research has documented that ordinary people generally have a vague sense about the objective level of inequality and social mobility. In other words, people can misperceive the social realities. It’s incredibly interesting to study the gap between reality and perception and how such discrepancies might affect individuals’ other opinions.
Your project initially used MTurk, a popular online service that allows individuals to complete simple tasks — such as taking surveys — and receive a small incentive for their time. What tips would you give to researchers who may be considering MTurk in their study?
Cheng: I have two tips to share. First, MTurk can be especially useful in the preliminary stages of the project. For example, we can use pilot studies to test the effect of different randomization strategies, alternative wordings of questions, or whether it makes sense to include certain sample restrictions. Second, the MTurk sample is not a randomly selected sample, and it thus may not be representative of the general population. To alleviate this limitation, when designing an MTurk survey, it’s useful to collect information on key demographics and political orientation, which will allow the researcher to construct weights for the analyses so that the results better reflect the general population distribution.
Wen: Generally speaking, American participants registered on MTurk are younger, more educated, more liberal, and more likely to be white relative to the general population in the United States. When recruiting participants via MTurk, researchers should be aware of these unique demographic characteristics. If necessary, weights should be constructed, as Siwei suggests.
Based on your experience, what should researchers consider before designing an online survey-based experiment?
Chen: I think online survey-based experiments provide a very useful tool for understanding public opinions about social, political, and economic issues. They offer an important alternative to traditional household-based surveys because the online questionnaire design platform, such as Qualtrics, allows us to design flexible, interactive, and innovative survey instruments. However, because the questions used in these survey experiments can be quite unconventional, the potential response bias associated with these question designs may be less understood as compared to conventional survey questions. Therefore, I recommend that the researcher conduct in-depth interviews with a small convenience sample, or field a number of pilot surveys with different types of question wording, before finalizing the design of the main survey.
Wen: Online survey experiments allow researchers to randomly assign subjects to treatment groups, which facilitates causal inference. However, many researchers are concerned that the treatment effects detected in survey experiments might be only short-term. In response to this critique, we recommend that experimenters should consider re-surveying their respondents after a certain period (e.g., one week) following the original contact. The longitudinal design enables researchers to test if the treatment effects in the first wave of the survey persist over a longer period of time. Currently many online survey platforms such as the Qualtrics Online Panels support this kind of research design.