Protocol for a Meta-Review on Education Meta-Analyses: Exploring Methodological Quality and Potential Significance for Research Use in Practice

Abstract

This protocol presents the planned methods and procedures for a meta-review that explores the methodological quality of education meta-analyses and their potential significance for research use. An ever-increasing number of meta-analyses are published every year while at the same time new approaches to enhance the validity and reproducibility of systematic reviews are in constant development. We aim to conduct a meta-review to examine the current practices of education meta-analyses in terms of review procedures, meta-analysis methods, and strategies for making meta-analyses useful for non-research audiences. We will focus on meta-analyses including randomized controlled trials and quasi-experimental designs on the effects of K-12 school-based academic interventions on student academic achievement. A comprehensive search will be conducted to retrieve all studies that potentially meet our criteria. Study features will be coded to evaluate three main dimensions: the quality of the review process, which includes the practices used at each systematic review stage (e.g., search procedure, selection, and critical appraisal); the quality of the meta-analysis methods, including the methods used for synthesizing the results, exploring heterogeneity and additional analyses, such as publication bias assessment; the significance for research use, which includes evaluating the ways authors present findings to increase accessibility and relevance for practitioners (e.g., stakeholder engagement in the process, reporting of results). We will identify strengths and weaknesses of current meta-analysis practices and compare them across time, publication outlet, and funding status.