Involving Teachers in the Data-driven Improvement of Intelligent Tutors

Aug 2022 - Sep 2022

An early iteration of intelligent tutors’ design is sometimes suboptimal due to the expert blind spot. Students’ log data, and interpretation of the data with domain knowledge can be helpful.

Examples of the interactions with the intelligent tutor x+3=5.

However, teachers, who have pedagogical content knowledge and practical experience, have only been involved in initial design stages and classroom use scenarios, but not the process of data-driven redesign of intelligent tutors.

This paper investigates what data representations might be intuitive for teachers and lead to good tutor redesign ideas.

We first developed an interactive visualization interface, EqLens, to present students’ log data. We then conducted a within-subjects user study with eight middle-school math teachers to derive insights for tutor improvement with two baselines.

The layout of EqLens includes multiple stages: invalid, add/subtract items, combine like terms, and transform coefficient of x to 1. It also shows an enlarged example to show the shift in the y-axis from stage combine like terms to a previous stage.

Results showed that teachers derived new insights with data and EqLens helped them propose suggestions addressing students’ common problems.

Participants’ rating results for usability and usefulness on a 5-point Likert scale under three conditions: Original Tutor, List Interface, and EqLens.

We discussed potential ways to further engage teachers in the data-driven redesign process.