Underperforming students get more out of dashboards

Stephanie Teasley

Educational dashboards that report course progress to students are likely more motivating for the underperforming set than for those who are doing well, according to a new study conducted by researchers at the University of Michigan and Blackboard.

At the same time, students at all performance levels said they found the dashboard feedback useful, and that they’d appreciate continuous access to it. Data dashboards turn numerical information into graphical representations to provide rapid insights not easily found through numerical analysis. Typical of business intelligence applications, this technology is being increasingly incorporated into educational technology. Many of the educational dashboards available today are aimed at advisors and instructors, but the researchers say they have great potential for the benefiting students themselves. But scientists and educators have little empirical research about the effectiveness of student-facing dashboards.

“This study is the first to specifically investigate how feedback and students’ academic standing affects their experience with dashboards and points to the importance of considering how to design these systems to provide more personalized information that motivates students,” said Stephanie Teasley, a research professor in the U-M School of Information (UMSI).

Teasley led a team of researchers from UMSI in collaboration with researchers and staff from Blackboard, a leading provider of education technology. The company’s flagship learning management system, Blackboard Learn, is used around the globe. The company has recently developed student-facing dashboards for the system. 

“By asking students questions about their impressions of dashboards, we are able to gain critical insights to help us improve these tools,” said John Whitmer, Blackboard’s director for analytics and research. “As we broaden our knowledge about dashboards and their impact on student motivation, we can better ensure they are directly benefitting the students who need them the most.”

The researchers set out to answer questions like: How do students interpret the information these dashboards provide, and do they find it useful? Which students find the information motivating and when? Answers to these questions aren’t readily found in the literature, the researchers say.

For the study, the researchers conducted face-to-face interviews of 47 U-M undergraduate students and had the students go through several course-feedback simulations. For half the students, dashboard feedback showed they were performing in the top 10 percent of the class. The other half were shown they were in the bottom 5 percent. Roughly half of students in each group had what the researchers considered a high GPA in real life, and the rest, a relatively low one. Anything below a B was considered “low” due to the high-skewing academic performance of the students who volunteered for the study.

At the start of each student interview, the researchers asked the participant to think back to a course that was vital to their major, and to describe it. They instructed the students to keep this course in mind throughout the session. Then they gave the students three different scenarios of hypothetical course feedback.

The first feedback scenario told the students only how their dashboard login activity stacked up to their peers. The second was designed to mimic feedback from just after a mid-term exam, and it told students how they ranked in terms of both their login activity and academic performance. Similar info was given for the third scenario, which was styled to take place just before the final exam.

Based on the dashboard feedback, in all scenarios, the students with lower GPAs reported that they’d be more likely than students with higher GPAs to: turn on the summary feedback feature and check it regularly, and take immediate action. And regardless of their GPA in real life, students who were told in the simulation that they were doing poorly were significantly more likely to find the feedback useful.

The researchers found these results striking.

“Underperforming students are the ones who have the most to gain from dashboards,” Teasley said. “And a concern going into the study was that the dashboards would decrease their motivation. However, many of these students’ comments reflected insights about what they might do to improve, which is one of the intended benefits of dashboards.”

She cautions that because the students in the “low GPA” group were still doing okay in their coursework, further research should test whether the findings hold true for students at risk of failing a course.

The study was funded by Blackboard and collaboratively designed and interpreted. All direct observation and findings were conducted by the independent research team at U-M.

Read the study.

Posted February 2, 2017