Technology improving learning – an edtech meta-analysis

I’m re-tuning my network antenna for a new open education tech project I am going to be working on.

via GIPHY

I’ll have more details to share on the project in the coming weeks, but suffice to say that it has put a spring in my step this week, thanks in no small part to providing me with the opportunity to deeply re-engage with the open education community.

I am starting to look into online homework systems, and there were two items that popped onto my newly tuned radar this week.

The first was David Wiley’s post on the critical role of practice in learning. In the post, David points to a comparative study between a group of math students who used Khan Academy resources, and a group of students who used Khan Academy resources PLUS the built in Khan Academy online homework system. The results of the study showed the later group did better in the course. As David notes (emphasis mine);

This is not the first example, nor will it be the last example, of peer-reviewed research demonstrating that OER plus a system that provides opportunities to engage in online interactive practice results in better student learning than OER alone.

He then goes on to list supporting research from Carnegie-Mellon that shows a significant positive learning effect up to six times greater from doing online interactive practice assignments than just doing the reading. Which, as David notes, should not be a surprise to educators. Systems & processes that encourage repeated practice is a well established theory in our field. We have built entire models of education and training around this “doer” model (hello apprentice programs!) that have learners practice, reflect, and then practice again.

But it is interesting to see just how powerful the learning effects of online interactive practicing can be.

To add more evidence that online interactive practice can be effective comes the second item on my list, a newly released edtech meta-analysis  from J-PAL at MIT which examined 126 education technology studies, 30 of which focused on computer-assisted learning programs designed to help students practice particular skills. In their meta-analysis the researchers discovered that technology programs that emphasize practice can have a significant effect on learning, especially in math.

Computer-assisted learning programs have shown enormous promise in improving academic achievement, especially in math. Of all 30 studies of computer-assisted learning programs, 20 reported statistically significant positive effects, 15 of which were focused on improving math outcomes.

It is not surprising to me that math practice programs are so widely in use. Math is a discipline where practice is important. It is also an area that has a long, established history of practice tools from the pre-digital world (think worksheets and flashcards). There are precedents from the pre-digital math world that are not difficult to replicate in the digital realm. On the SAMR model of edtech, I suspect there are more than a few math practice programs that fit into the S quadrant.

It also strikes me (and I am generalizing a bit here based on my own extremely limited knowledge of teaching in math) that math, by its very nature and structure, lends itself really well to the creation of online interactive practice systems. Create a rule, and change the values to generate a new set of problems.

Not to say that ALL math practice programs are like this. I am sure when I start to take a deeper dive I’ll find there are more sophisticated applications in use. But I would suspect that, in the digital math world, there are a lot of digital equivalents of flashcards out there. So, in some sense, math may be the low hanging fruit of online interactive practice systems.

But there has to be more than just practice. You have to not only practice, but receive feedback on that practice, and be able to reflect on both the practice and feedback and alter your path accordingly. As I dig into how these types of systems are structured, I am going to be paying particular attention to ways in which these systems promote reflection and feedback, and prompt for next steps. Which makes me think my work in online homework systems (still figuring out what this particular phrase actually means) is going to lead me into the world of adaptive learning systems.

But that is a post for another day. For now, the early takeaway here is that there is some empirical evidence that education technologies that help students practice things can benefit learning, which I am grateful to see in the context of the project I am about to begin work on.

I contrast this to the early work I did in OER and the BCcampus open textbook project where there was a definite gap in empirical research supporting the use of OER and open textbooks. It is a gap that has shrunk considerably over the past decade, and a huge nod of thanks must be given to David’s early work in the OER space as he has always been a tireless advocate for not only the use of OER, but also the development of an empirical base of evidence for the community to work from. But from my early view, it looks like I won’t have that gap to deal with in my new project, and I have a lot of meaty research to dig deeper in to about online practice/homework systems.

(by the way, have I mentioned that I am pumped about this new project!)