Predictive Modelling

Predictive modelling is a core learning analytics technique that uses supervised machine learning to identify patterns of relationships in past data that can be used to make predictions about the future. If you are new to the field or want a quick refresher, we recommend the following resources. 

Thank you to Craig Thompson, Dr. Chris Brooks and Dr. Quan Nguyen for providing some of the resources in this section.

1. Get Oriented

Predictive Modelling in Teaching and Learning

This 2017 Handbook of Learning Analytics chapter by Christopher Brooks and Craig Thompson provides an overview of predictive modeling through the lens of an educational data scientist. The authors also delve into some of the most prominent predictive modelling techniques.  

Read the Predictive Modelling in Teaching and Learning chapter [8 pages]

A Visual Introduction to Machine Learning

A short two-part, high-level introduction to machine learning by R2D3 using dynamic visualization of decision trees to explain the logic underlying classification models.

View Part One: Building a Model (a visual introduction to machine learning) on the R2D3 website 

View Part Two: Model Tuning & Tradeoffs (model tuning and the bias-variance tradeoff) on the R2D3 website

2. Dive Deeper 

A Large-Scale Implementation of Predictive Learning Analytics in Higher Education

This 2019 paper by Christothea Herodotou, Bart Rienties, and colleagues examined the use and impact of predictive learning analytics by teachers in a distance learning institution with over a thousand students. Findings show both how the data qualitatively impacted teaching practices and quantitative effects on students’ performance.

Read the A Large-Scale Implementation of Predictive LA in Higher Education paper [34 pages]

Student Success Prediction in MOOCs

This comprehensive 2018 paper by Josh Gardner and Christopher Brooks reviews the literature on predicting student success in MOOCs (Massive Open Online Courses). Looking at different kinds of outcomes, predictors and underlying theoretical models, the article overviews the main data sources, feature engineering methods, modeling algorithms, and evaluation methods used. The paper also points out key methodological gaps in the literature and highlights opportunities for future research.

Read the Student Success Prediction in MOOCs paper [76 pages]

Practical Machine Learning Tools and Techniques

This 2016 book by Ian Witten, Eibe Frank and colleagues is an accessible introduction to machine learning that discusses inputs, outputs, common algorithms and evaluation methods used in this data mining approach. It also includes coverage of more advanced topics including data transformations, bayesian networks, deep learning and semi-supervised approaches 

Read the Practical Machine Learning Tools and Techniques book [freely available online]

3. Tools & Tutorials

Data Science in Education Using R

This 2020 book by Ryan Estrellado, Emily Bovee and colleagues offers several great walkthroughs on how to use data commonly available in online courses to predict student performance through both linear regression (Chapters 7 & 8) and random forest models (Chapter 14). It also offers an introduction to R and R Studio (Chapters 5 & 6) for those just getting started.

Get Started with R Linear [Chapters 5 & 6] 

Regression Walkthroughs [Chapters 7 & 8]

Random Forest Walkthrough [Chapter 14]

Hands-On Machine Learning (in Python)

This 2019 book by Aurélien Géron explains and demonstrates the use of a variety of predictive modeling techniques in Python, ranging from simple linear regression to deep neural networks. It also offers exercises to practice each technique. While the book requires a purchase, a free 10-day trial is available.

Try out the Hands-On Machine Learning book

Network Analysis

Network analytics are useful for identifying patterns of relationships or interactions between people, learning materials or other entities. This page offers resources for those starting out in network analytics and links to introductory courses on conducting network analytics in both R and Python.

Thank you to Drs. Bodong Chen, Sasha Poquet and Tobias Hecking for suggesting some of the resources in this section.

1. Get Oriented

What is Social Network Analysis? 

This 2015 video tutorial by ModU at Duke University introduces the unique affordance of social network analysis for analyzing, visualizing and understanding the connections and relationships between data points. Multiple examples of practical questions are used to illustrate the potentials of social network analysis.

Watch the What is Social Network Analysis video [4 mins]

The Social Network Perspective and Educational Research 

This 2014 introductory chapter by Brian Carolan explains the distinct theoretical and methodological approach of network analysis as a focus on ties between actors and the properties of groups that emerge from these relationships, examining recent advances in the field and applications for education. Chapter one is freely available in the preview of Google Books. The remainder of the Social Network Analysis and Education: Theory, Methods & Applications book details methods, measures, applications and examples for those who wish to go further. 

Read the Social Network Perspective and Educational Research chapter [19 pages]

2. Dive Deeper

Social Network Analysis as a Learning Analytics Tool

This 2019 paper by Mohammed Saqr and Ahmad Alamro demonstrates several typical uses of network learning analytics to describe the level and structure of interactions among group participants, identify central and isolated students, and predict academic performance.

Read the Social Network Analyss as a LA Tool paper [11 pages]

Analyzing Learning and Teaching Through the Lens of Networks

This 2020 webinar by Sasha Poquet and Bodong Chen introduces the notion of learning as a networked phenomenon and explores the use of network analysis for understanding learning. It focuses on three specific ways of using network analysis in learning analytics: one, for visualizing connections of all sorts; two, in offering metrics of various dimensions of learning; and three, to develop explanatory theories about learning processes. For each area, case studies of real-world learning and teaching scenarios were presented.

Watch the Analyzing Learning & Teaching through the Lens of Networks video [56 mins]                View the Analyzing Learning & Teaching through the Lens of Networks presentation [67 slides]

SNA Key Concepts

This 2017 video by Bodong Chen offers useful clarification of many of the specific terminology used in social network analysis. It introduces key concepts of social network analysis, including actors, ties, networks, and matrices as well as the alternative language used to refer to them across different research areas.

Watch the SNA Key Concepts video [15.5 mins]

3. Tools & Tutorials

Applied Social Network Analysis in Python

This four week course offered by the University of Michigan through Coursera provides a basic introduction to network analysis and the NetworkX Python library. The course covers the motivations for and key concepts of network analysis, ways to measure centrality of a node in a network, models of network generation and the link prediction problem. Basic familiarity with Python is required.

Enroll in the Applied Social Network Analysis in Python course 

Social Network Analysis Tutorial with R

This 2016 tutorial by Katya Ognyanova introduces the igraph package for network analysis and visualization in R. It provides step-by-step guidance on how to create networks (including reading network data from files), turn networks into igraph objects and plot networks with igraph. For learners who are not familiar with R, a basic introduction to R is also provided at the beginning of the tutorial.

Access the Social Network Analysis with R tutorial

Text Mining

Text mining and natural language processing (NLP) methods in LA use computational linguistic techniques to generate insight into student writing, learning dialogues, teacher talk and other forms of educational text. Start off with some overviews of the uses of text mining in learning analytics and then try out one of several tutorials for conducting text mining in either R or Python.

Thank you to Dr. Jelena Jovanovic for providing some of the resources in this section.

1. Get Oriented

Handbook of Learning Analytics Chapters 

Three chapters in the 2017 Handbook of Learning Analytics provide complementary perspectives on the use of texting mining in learning analytics.

The Content Analytics (CA) chapter by Vitomar Kovanovic and colleagues describes a set of methods to evaluate, index, filter, recommend, and visualize digital learning content. Considering both educator-created learning materials and student-created learning products, they provide an overview of key text mining methods useful for analysis.

Read the HLA Content Analytics chapter [16 pages]

The Natural Language Processing (NLP) chapter by Danielle Macnamara and colleagues introduces a set of NLP tools commonly used in educational contexts and the linguistic features they index with specific examples of their application. 

Read the HLA NLP chapter [12 pages]

The Discourse Analytics (DA) chapter by Carolyn Rosé and colleagues summarizes different representations of text and outlines popular unsupervised and supervised analytic methods for DA as well as their strengths and limitations.

Read the HLA Discourse Analytics chapter [10 pages]

2. Dive Deeper

Text Mining for Learning Content Analysis

This 2019 Learning Analytics Summer Institute (LASI) workshop by Jelena Jovanovic serves as an introduction to text mining through the lens of LA. It covers the main steps of the text mining flow, including text preprocessing, text transformation, feature selection, data mining, and interpretation/evaluation and provides concrete examples. 

View the Text Mining for Learning Content Analysis presentation [68 slides]           Access the Text Mining for Learning Content Analysis workshop materials

AcaWriter: A Learning Analytics Tool 

This 2020 paper by Simon Knight, Antonette Shibani and colleagues introduces the open-source AcaWriter tool, which uses text mining techniques to provide feedback to students on their use of rhetorical moves in their academic writing. The paper describes the tool’s theoretical background and technical implementation as well as discusses three examples of its use.

Read the AcaWriter: A Learning Analytics Tool paper [46 pages]

3. Tools & Tutorials

Text Mining in R

Introduction to Text Analytics with R

This 2019 tutorial by Data Science Dojo provides introductory coverage of analyzing text using R. It focuses on the creation of predictive classification models based on textual features and uses practical examples to walk through all phases of the process. This includes pre-processing textual data, extracting textual features, developing and training classification models using these features, and evaluating accuracy of the trained classification models. All the code and the datasets used in the examples are made available in the description of each video. 

Access the Introdiction to Text Analytics with R tutorial [12 parts]

Text as Data 

This free course by Chris Bail covers a wide range of ways to obtain and analyze textual data. It offers a detailed introduction and tutorials with sample data and R codes for commonly used techniques in collecting, processing, and analyzing text-based data. Topics covered include mining data from application programming interfaces (APIs), screen-scraping, basic text analysis, dictionary-based text analysis, topic modeling, text networks, and word embeddings. Basic familiarity with R is required.

Access the Text as Data course [8 parts]

Text Mining with R – A Tidy Approach

This 2020 book by Julia Silge and David Robinson is a great introduction for those who are familiar with R but new to text mining. It introduces the tidy text format as a way to structure text data and offers methods to work with text in tidy format using the tidytext package and other tidy tools in R. Compelling examples of real text mining problems are provided to demonstrate the application of these methods. 

Access the Text mining with R book [freely available online]     Access the Text Mining with R GitHub repository

Text Mining in Python

Natural Language Processing in Python Tutorial

This 2018 introductory level tutorial by Alice Zhao covers text preprocessing techniques, machine learning techniques and Python libraries for Natural Language Processing. It walks you through all of the steps of a text analysis project using an example in Jupyter Notebook. Some familiarity with programming in Python is required.

Watch the NLP in Python tutorial [2 hours]       Access the NLP in Python tutorial materials

Modern NLP in Python Tutorial

This 2016 tutorial by Patrick Harrinson offers an introduction of preparing, modeling, visualizing, and analyzing textual data in Python through examples run on a dataset about user reviews published by the business review service Yelp. It assumes a working knowledge of Python, but does not require any pre-knowledge of text analysis. 

Watch the Modern NLP in Python video tutorial [1.5 hours]       Access the Modern NLP in Python tutorial materials 

Introducing Text Analytics and Natural Language Processing with Python

This free EdX course is a practical and scientific introduction to NLP with Python. It takes you through the overall text analytics process (from data collection and preprocessing to evaluation of the results) while also explaining the underlying science of NLP and how artificial intelligence views language differently from humans. It also discusses the limitations of and provides ethical guidelines for applying NLP to real-world problems. The course materials are useful to both those who are new to Python and those who have Python programming experience.

Enroll in the EdX Introduction Text Analytics and NLP with Python course 

Visual LA & Dashboards

Visual LA relates to the graphical presentation of data, either to aid in the analysis process or as a tool to communicate the results of analyses to various stakeholders. LA Dashboards are one common format for sharing data visualizations with stakeholders. This section compiles some high quality resources that can guide you in learning about and improving data visualizations and dashboard designs.

Thank you to Dr. Sharon Hsiao for providing some of the resources in this section.

1. Get Oriented

Learning Analytics Dashboards

This 2020 paper by Katrien Verbert, Xavier Ochoa and colleagues describes both early research on visualization and prediction that demonstrated the potential of learning analytics dashboards and current work to integrate participatory design methods, tailor dashboards for particular users and expand the theoretical underpinning for the work. Drawing on an International Conference on Learning Analytics & Knowledge (LAK) 2019 workshop on Visual Approaches to Learning Analytics, they articulate common practices and challenges in the domain and offer an agenda to shape future research.

Read the Learnging Analytics Dashboards paper [6 pages]  

Visualizing Data

An introduction to data visualization by Arizona State University that explains what stories can most effectively be told by different types of data visualization, how important the central questions about the data are in determining what type of visualization to use, and how design choices can influence the stories told by the visualization. 

Watch the Visualizing Data video [10 mins]

2. Dive Deeper

Driving Teachers’ Attention through Educational Data Storytelling

This 2018 paper by Vanessa Echeverria, Roberto Martinez-Maldonado and colleagues presents a human-centered approach to designing learning analytics dashboards focused around the narratives educators construct about student progress using data. In it, they describe how Data Storytelling can help align educational visualizations with intended learning designs and present a case study to explore its effectiveness.

Read the Drving Teachers' Attenttion through Educational Data Storytelling paper [25 pages]

Pitfalls of Learning Analytics Dashboards 

This 2017 paper by Ioana Jivet, Maren Scheffel and colleagues offers a systematic review of the literature on learning analytics dashboards, with a particular focus on theoretical foundations and the application of educational concepts. Their critical examination suggests cause for concern as current designs provide problematic frames of reference that lead to a focus on learner competition rather than knowledge mastery. 

Read the Pitfalls of Learning Analytics Dashboards paper [14 pages] 

Towards User-Centered Analytics: User-Adaptive Visualizations

This keynote by Cristina Conati at Learning Analytics and Knowledge Conference 2018 problematizes a one-size-fits-all approach to data visualization and argues for the needs of user-adaptive visualizations. It discusses what user characteristics may impact their experiences with data visualization, how to model these characteristics in real-time using data including eye-tracking measures, and how to use these models to provide adaptive support to improve user experiences with data visualizations. 

Watch the Towards User-Centered Analytics video [1 hour 6 mins]

Visual Analytics: Definition, Process and Challenges 

This 2008 paper by Daniel Keim, Gennady Andrienko and colleagues goes beyond the simple question of visualization to highlight the larger purpose of visual analytics and other important elements of it such as data management and human-computer interaction. Add one more sentence

Read the Visual Analytics: Definition, Process and Challenges paper [23 pages]

Guiding and Motivating Students Through Open Social Student Modeling

This 2017 paper by I-Han Hsaio and Peter Brusilovsky describes the development of a visual analytics system to present students with a model of their learning progress on specific programming language topics. Following the tradition of Open Student Modelling, the system allows students to navigate through their own model and identify the most relevant learning resources. In addition, the system introduces a social component into the visualization to allow students to compare their own progress with those of the class and individual peers. 

Read the Guiding and Motivating Students through Open Social Student Modeling paper [33 pages]

3. Tools & Tutorials

Data to Viz

A tool by Yan Holtz and Conor Healy that helps you choose the most appropriate chart type for your data and context. The tool works as a decision tree that is both guided and allows for exploration. Each visualization type is presented with a description of what it is useful to show and common mistakes to be avoided in using it. Links to examples with code in R, Python and D3.js are provided.

Explore the Data to Viz tool 

Design in LA

The process of designing LA has a critical influence on the tools that result and has recently received a great deal of attention in the field. Recent approaches emphasize the inclusion of stakeholders in the design process, including intended end-users, with specific attention to including members of often overlooked or underrepresented groups.

The resources in this section offer an introduction to human-centered and participatory design of LA.

1. Get Oriented

Human-Centered Learning Analytics

This 2019 paper by Simon Buckingham Shum, Rebecca Ferguson and Roberto Martinez-Maldonado lays out the importance of human-centered design in the field of learning analytics and explains the benefits and challenges of a human-centered approach for addressing issues associated with learning analytics implementations. The paper also provides an overview of five studies about theoretical foundations, techniques, and application of human-centered learning analytics.

Read the Human-Centered Learning Analytics paper [9 pages]

Designing LA for Humans with Humans

This 2019 webinar by LEARN Director Alyssa Wise presents a diverse set of examples of the ways that NYU-LEARN is including educators and students in the process of building and implementing learning analytics. Dr. Wise provides examples of how to: involve students in the creation and revision of learning analytics solutions for their own use; work with instructors to align analytically available metrics with valued course pedagogy; and partner with an educational team to design and implement interventions based on at-risk students predictions. Webinar attendees gain a sense of both the conceptual issues and practical concerns involved in designing learning analytics for humans with humans.

Watch the Designing LA for Humans with Humans webinar [1 hour]      View the Designing LA for Humans with Humans presentation [58 slides]

2. Dive Deeper

Engaging Students as Co-Designers

This 2020 paper by Juan Pablo Sarmiento, Fabio Campos and Alyssa Wise provides a concrete example of how to involve university students in the design of a LA tool. It describes a series of co-design workshops conducted with students and shares three strategies that were found useful for engaging students in the design process.

Read the Engaging Students as Co-Designers paper [5 pages]

Working Together in LA Towards the Co-Creation of Value 

This 2019 paper by Mollie Dollinger, Danny Liu and colleagues offers a good overview of the current literature on Human-Centered Design in LA. It clarifies important differences between similar-sounding terms (e.g. user-centred vs.human-centred vs. participatory design) and summarizes the major challenges in adopting such approaches in LA. The paper concludes with a case study of the redesign of an educator-facing analytics system to demonstrate the use and benefits of human-centered approaches to design.

Read the Working Together in LA Towards the Co-Creation of Value paper [17 pages] 

Towards Value-Sensitive Learning Analytics Design

This 2019 paper by Bodong Chen and Haiyi Zhu introduces the theory, methodology, and methods of Value Sensitive Design, with the aim of addressing the ethical and privacy concerns and human values considerations in learning analytics systems. It reports on two case studies where Value Sensitive Design methods are used to 1) investigate an existing learning analytics tool for future tool refinements and 2) design a recommendation system holistically. 

Read the Towards Value-Sensitive LA Design paper [10 pages]

3. Tools & Tutorials

6 Techniques for Participatory LA Design

This 2021 workshop led by Juan Pablo Sarmiento offered through the Learning Analytics Learning Network (LALN) offers concrete techniques and methods you can leverage to create participatory workshops with educators and students. It serves as a good introduction to some of the participatory tools that are often used in PD for LA and can help you think through different ways to involve stakeholders in a project.

Watch the 6 Techniques for Partcipatory LA Design video [1 hour]

Co-Designing LA Tools with Learners

This 2018 chapter by Carlos Prieto-Alvarez, Roberto Martinez-Maldonado, and Theresa Anderson describes and provides illustrative examples of a set of tools and techniques that can be used to include learners in the loop of learning analytics innovation. It is particularly aimed at LA practitioners and researchers who are looking for ways to involve students in the design process of LA and enhance the adoption and success of the resulting applications

Explore the examples in the Co-Designing LA ToolS with Learners chapter [20 pages]

Ethics & Equity

Learning analytics operate as part of socio-technical systems in which issues of privacy, agency, power, equity, consent and accountability must be considered. Yet these are new areas of thinking for the field.

This section includes a selection of resources to introduce and raise awareness about these issues and a number of tools for taking them into account when designing and implementing LA.

1. Get Oriented

Learning Analytics: Ethical Issues and Dilemmas

This 2013 paper by Sharon Slade and Paul Prinsloo examines early research on learning analytics from a sociocritical perspective to systematically document ethical issues that need to be considered in higher education. Looking at questions surrounding the role of power, the impact of surveillance, the need for transparency, and an acknowledgment that student identity is a transient, temporal, and context-bound construct, the paper proposes six principles as an initial framework to help guide higher education institutions in addressing these ethical issues.

Read the LA: Ethical Issues and Dilemmas paper [27 pages]

Subversive Learning Analytics

This 2021 paper by LEARN team members Alyssa Wise, Juan Pablo Sarmiento, and Maurice Boothe Jr. introduces the notion of Subversive Learning Analytics as arising from the premise that data should be used as a transformative force for change rather than to perpetuate an often-problematic status quo. It describes how learning analytics researchers and practitioners can take a critical stance to reveal and challenge the power structures and inequities in education.

Read the Subersive Learning Analytics paper [7 pages]

Free access via SoLAR

Algorithmic Bias in Education

This 2021 video by Ryan Baker and Aaron Hawn offers an overview of what we currently know (and don't know) about different sources of algorithmic bias in education and the kinds of harm to particular subgroups of students that can result. The latter half of the talk offers suggestions to identify and reduce bias by improving data collection practices, developing targeted tools and broadening the community of people who are involved in algorithmic development.

Watch the Algorithmic Bias in Education video [55 min]

2. Dive Deeper

Empirical Evidence for Algorithmic Bias in Education Wiki

This wiki, maintained by the Penn Center for Learning Analytics, provides a place to find and share peer-reviewed publications on algorithmic bias in educational settings. The research is organized both by the groups studied (e.g. race, ethnicity, gender and other categories) and by the kind of situation investigated (e.g. grade prediction, essay scoring, affect detection, etc.).

Explore the Penn Center for Learning Analytics Wiki

(Re)centring Students in Learning Analytics

This 2019 paper by Christine Broughan and Paul Prinsloo discusses the issue of exclusion of students from current learning analytics practices and explores the importance of student-centered learning analytics in response. Using the teachings of Paulo Freire as a framework, they examine ways to include students as partners rather than mere subjects in the collection, analysis and use of their data, thereby recentering students as data owners, rather than data objects.

Read the (Re)centring Students in Learning Analytics paper [22 pages]

How Colonial Continuities Underlie Algorithmic Injustices in Education

This 2021 paper by Shamya Karumbaiah and Jamiella Brooks argues that many of the inequities and structures  created by colonialism are pervasive in educational and technological systems, and thus should be taken into account when building learning analytics tools. Using the lens of sociotechnical systems and critical race theory, the authors identify a number of different, often surprising, ways in which algorithmic systems replicate and can even deepen the ongoing inequities that arise from coloniality.

Read the How Colonial Continuities Underlie Algorithmic Injustices in Education paper [6 pages] 

3. Tools & Tutorials

Learning Analytics Strategy Toolkit

This 2020 toolkit created by Tyton Partners in partnership with Every Learner Everywhere provides guidance and resources for higher education institutions to implement equity-focused Learning Analytics at scale. Specifically, it characterizes the current state of higher education in using learning analytics to address educational equity, offers evidence-based principles to guide institutional learning analytics implementation, and shares strategies, tools and case studies to support the application of the principles and strategies.

Explore the Learning Analytics Strategy toolkit [38 pages]

What-if Tool

This tool by People + AI Research (PAIR) works as an extension in Jupyter, Colaboratory, and Cloud AI Platform notebooks for conducting model understanding and evaluation tasks. It can be used to analyze and visualize regression and classification model behaviors in hypothetical situations, evaluate the importance of data features, and understand model performance and fairness. Easy-to-follow tutorials and examples are provided to walk you through how to set up and use the tool.

Explore the What-if tool

Privacy by Design

This 2011 framework by Ann Cavoukian offers a “design-thinking” perspective to approach the issue of privacy in our present-day information society. Building on the universal principles of Fair Information Practices, Ann Cavoukian establishes seven foundations for Privacy by Design as a framework to guide the protection of privacy in the modern era.

Explore the Privacy by Design framework [9 pages]