A new GLO Discussion Paper replicates two published articles by seven independent research teams each to document the huge variety of different research strategies chosen and their large differences in outcomes.
The Global Labor Organization (GLO) is an independent, non-partisan and non-governmental organization that functions as an international network and virtual platform to stimulate global research, debate and collaboration.
GLO Discussion Paper No. 537, 2020
The Influence of Hidden Researcher Decisions in Applied Microeconomics – Download PDF
by Huntington-Klein, Nick & Arenas, Andreu & Beam, Emily & Bertoni, Marco & Bloem, Jeffrey R. & Burli, Pralhad & Chen, Naibin & Greico, Paul & Ekpe, Godwin & Pugatch, Todd & Saavedra, Martin & Stopnitzky, Yaniv
GLO Fellow Todd Pugatch
Author Abstract: Researchers make hundreds of decisions about data collection, preparation, and analysis in their research. We use a many-analysts approach to measure the extent and impact of these decisions. Two published causal empirical results are replicated by seven replicators each. We find large differences in data preparation and analysis decisions, many of which would not likely be reported in a publication. No two replicators reported the same sample size. Statistical significance varied across replications, and for one of the studies the effect’s sign varied as well. The standard deviation of estimates across replications was 3-4 times the typical reported standard error.
GLO Discussion Papers are research and policy papers of the GLO Network which are widely circulated to encourage discussion. Provided in cooperation with EconStor, a service of the ZBW – Leibniz Information Centre for Economics, GLO Discussion Papers are among others listed in RePEc (see IDEAS, EconPapers). Complete list of all GLO DPs – downloadable for free.
Ends;
