Every year schools spend many billions on education technology - globally some $18bn in K-12 alone in 2022/23. This money buys school districts in the US an average of 2,591 edtech tools. In the past few years, school districts in the US and elsewhere have started to demand proof that the money spent on all those tools is actually improving learning. School districts want to know if edtech tooling is “evidence based”. In this blog post we will explore what schools mean by evidence based, how this relates to certification standards that are emerging and how WhatWorked fits into this picture.
The Every Student Succeeds Act (ESSA) from 2015 introduced four tiers of evidence for educational interventions. The lowest tier, “demonstrates a rationale”, requires a well-defined logic model: what is the desired outcome of the intervention and how is it going to achieve that. This logic model must be supported by robust research. The next tiers involve quantitative studies of increasing sophistication: a correlational study, a quasi-experimental study and a randomised controlled trial, respectively.
In their annual overview, LearnPlatform reports that 36 of the 100 most popular edtech tools in K-12 in the US publish ESSA-aligned research on their website. This amounts to roughly half of the student-facing tools on the list. Obviously the quality of the research that is presented will vary significantly from product to product. This raises the question how schools are to judge the quality and relevance of these studies. This is where edtech quality frameworks come in. Both private companies and national governments have introduced schemes that assign edtech products different quality levels depending on the evidence they present. Well-known examples are the schemes of LearnPlatform, Digital Promise, What Works Clearinghouse (all US), Tulna (India), AERO (Australia) and Edtech Impact (UK).
Over the past few years many such quality frameworks have emerged. In fact, a recent review identified 74 different edtech quality frameworks worldwide. To facilitate edtech companies operating internationally the International Certification of Evidence of Impact in Education (ICEIE) has set up a framework to align various edtech quality standards globally. To be evaluated by the ICEIE companies can submit their national certification or research study. When the certification or study meets the ICEIE standards the product is awarded a gold, silver or bronze badge in one of five categories: efficacy, effectiveness, equity, ethics and environment.
As of September 2024 the ICEIE website lists 500 edtech products with one or more certifications, of which 32 have been evaluated by the ICEIE. The overwhelming majority have applied for a bronze certification for efficacy (proven in laboratory conditions) or effectiveness (proven in the classroom). In the ICEIE scheme bronze is equivalent to the lowest, “demonstrates a rationale” ESSA level. Only two gold badges for efficacy have been awarded so far: My Math Academy by Age of Learning and Kahoot’s Quiz and Poll games.
The ICEIE database also reveals that none of the 500 listed products have applied for a gold effectiveness badge. This is a testament to the difficulty of performing randomised controlled trials of edtech products in schools. With WhatWorked we address exactly this problem. WhatWorked allows microRCT’s to be run by individual teachers that use edtech products in the classroom. The results of these individual microRCT’s are collected and combined into powerful effectiveness studies. In this way, WhatWorked will enable schools to select edtech tools based on the strongest possible evidence.
The Every Student Succeeds Act (ESSA) from 2015 introduced four tiers of evidence for educational interventions. The lowest tier, “demonstrates a rationale”, requires a well-defined logic model: what is the desired outcome of the intervention and how is it going to achieve that. This logic model must be supported by robust research. The next tiers involve quantitative studies of increasing sophistication: a correlational study, a quasi-experimental study and a randomised controlled trial, respectively.
In their annual overview, LearnPlatform reports that 36 of the 100 most popular edtech tools in K-12 in the US publish ESSA-aligned research on their website. This amounts to roughly half of the student-facing tools on the list. Obviously the quality of the research that is presented will vary significantly from product to product. This raises the question how schools are to judge the quality and relevance of these studies. This is where edtech quality frameworks come in. Both private companies and national governments have introduced schemes that assign edtech products different quality levels depending on the evidence they present. Well-known examples are the schemes of LearnPlatform, Digital Promise, What Works Clearinghouse (all US), Tulna (India), AERO (Australia) and Edtech Impact (UK).
Over the past few years many such quality frameworks have emerged. In fact, a recent review identified 74 different edtech quality frameworks worldwide. To facilitate edtech companies operating internationally the International Certification of Evidence of Impact in Education (ICEIE) has set up a framework to align various edtech quality standards globally. To be evaluated by the ICEIE companies can submit their national certification or research study. When the certification or study meets the ICEIE standards the product is awarded a gold, silver or bronze badge in one of five categories: efficacy, effectiveness, equity, ethics and environment.
As of September 2024 the ICEIE website lists 500 edtech products with one or more certifications, of which 32 have been evaluated by the ICEIE. The overwhelming majority have applied for a bronze certification for efficacy (proven in laboratory conditions) or effectiveness (proven in the classroom). In the ICEIE scheme bronze is equivalent to the lowest, “demonstrates a rationale” ESSA level. Only two gold badges for efficacy have been awarded so far: My Math Academy by Age of Learning and Kahoot’s Quiz and Poll games.
The ICEIE database also reveals that none of the 500 listed products have applied for a gold effectiveness badge. This is a testament to the difficulty of performing randomised controlled trials of edtech products in schools. With WhatWorked we address exactly this problem. WhatWorked allows microRCT’s to be run by individual teachers that use edtech products in the classroom. The results of these individual microRCT’s are collected and combined into powerful effectiveness studies. In this way, WhatWorked will enable schools to select edtech tools based on the strongest possible evidence.
Company Number 12789441
Copyright © 2024
Copyright © 2024
Created with
