Research Alliance Shows $80 Million Data System Offers Limited Help with Day-to-Day Teaching

study released by the Research Alliance for New York City Schools finds that The City’s Achievement Reporting and Innovation System (ARIS), an $80-million data system built to make schools more data-driven and to focus attention on the achievement outcomes for which they are being held accountable, is being used largely as a school-wide planning tool and much less as a direct aid to classroom instruction.

“ARIS does not include much ‘real-time’ data, like students’ daily assignments, quizzes, and tests,” said James Kemple, executive director of the Research Alliance. “As a result, it is not surprising that its use appears to be confined largely to school-wide analysis and planning, rather than daily classroom instruction.”

Teachers with access only to information about the students in their classes, each spent an average of less than 25 minutes on ARIS over the course of the year. On average, educators logged on to ARIS 21 times throughout the year, for about five minutes per session. Educators also made limited use of the system’s more complex analytic functions and virtual collaboration tools.

While many teachers reported that ARIS is their primary source for information about students’ backgrounds and achievement, they identified a number of areas where the system could be improved. These include the need for more regular student assessment data; better training and professional development; and dedicated time to work with and learn about the system.

The heaviest users tended to be school administrators and teachers with school-wide roles—for example, as data specialists. On average, these heavy users logged on to ARIS more than 55 times during the year and accumulated more than four and a half hours of usage. The study found that about three quarters of the City’s nearly 95,000 administrators, teachers, and support staff used the system at least once during the year, but generally for only brief periods. A subset of administrators and teachers used the system much more heavily; this group accounted for just 28 percent of users, but racked up more than 80 percent of all the time spent on the system.

Authors of the report suggest that developers do more to consult educators when designing data systems, and that advanced features like complex analytical tools and “virtual communities” be tested under real-world conditions before going to scale. For example, it has piloted the integration of diagnostic assessments in reading, allowing teachers to evaluate and monitor students’ progress on a more regular basis.

“ARIS is an evolving system,” said Thomas Gold, lead author of the report. “We look forward to continuing our study to learn more about new features that have been introduced in the last year or so and about conditions that seem to support or inhibit wide usage.”

The new study represents the first independent examination of actual usage of the system. The Research Alliance is planning a second report from its study in the summer of 2013.