A PRIISM Seminar by Cornell University's AJ Alvero
Join PRIISM and Dr. AJ Alvero to examine two timely questions in academia: (1) how a student’s sociodemographic profile can change the way their AI-assisted writing is perceived, and (2) how knowing if a student uses AI alters an evaluator’s judgment of a student’s intelligence.
Abstract
The release of ChatGPT has shifted the ways humans communicate and restructured the economy of writing style in education. With ChatGPT, anyone can generate text that is highly polished and well-regarded. In this presentation, I present descriptive and experimental studies which begin from this perspective and start to unpack the following: whether or not there is variation in how AI generated text is evaluated with respect to student sociodemographic profiles; and whether or not knowing a student used AI alters perceptions of student authors.
In the first half of the presentation, I present analyses of admissions essays submitted to a selective university and then show that applicants using essays which were likely written or co-written with AI were evaluated differently based on their socioeconomic status. I will also discuss how essays that were prompted to write like students from specific demographic backgrounds failed to do so. In the second half, I will go over an experiment that focuses on how written language is used to assess intelligence and how AI systems perpetuate a particular version of this stance. We also find that authors are rated as less intelligent if they knew the student used AI to co-write. Combined, the results map a changing economy of style: as “good writing” becomes cheaper to produce, evaluators shift what they reward and what they distrust, with unequal consequences across students.
Bio
Dr. AJ Alvero is a computational social scientist at the Cornell University Center for Data Science for Enterprise and Society with departmental affiliations in Sociology, Information Science, and Computer Science. Most of his research examines moments of high stakes evaluation, specifically college admissions and parole hearings. He addresses questions and topics related to the sociological inquiry of artificial intelligence, culture, language, education, race and ethnicity, and organizational decision making. This work has appeared or is forthcoming in journals such as Science Advances, Poetics, The Oxford Handbook of the Sociology of Machine Learning, the American Journal of Sociology, Sociological Methods & Research, Big Data & Society, and other venues. AJ earned his PhD at Stanford University along with an MS in statistics. Prior to entering academia, AJ was a high school English teacher in Miami, FL.