What Is Meta-Research?

John Dewey, an American philosopher, psychologist, and educational reformer, emphasized the necessity of “reflective thinking,” suggesting that we learn not merely from experience itself, but from deeply reflecting upon it.

While countless research papers are continuously being published, there is a need for ongoing evaluation to determine whether research is being conducted, analyzed, and interpreted appropriately. The study of the research process itself is known as “meta-research.” Taking a slightly broader perspective, the term “meta-science” is also used when this inquiry extends beyond individual research practices to the broader scientific ecosystem and science policy.

In the early 2010s, assessments of the reproducibility of key studies in oncology and biology raised serious concerns when many published findings could not be successfully replicated. These findings contributed to growing recognition of what became known as the “replication crisis.” Potential contributing factors included the “publish or perish” culture, statistical errors, intentional data manipulation, and publication bias. Such low reproducibility can result in substantial waste of resources and, in biomedical research, may ultimately threaten patient safety. Meta-research developed in part as a systematic response to these concerns.

While many people confuse meta-research with meta-analysis, which statistically aggregates data from individual studies, meta-research is not merely about combining study results. Rather, it is a discipline that examines and reflects on the entire research process, including study design, conduct, analysis, peer review, publication, and implementation.

The modern development of this field was strongly influenced by the 2005 publication of “Why Most Published Research Findings Are False” by Professor John Ioannidis, followed by initiatives such as the establishment of the Meta-Research Innovation Center at Stanford (METRICS). Although meta-research is a crucial discipline applicable to academic fields well beyond medicine, relatively few universities or scholarly societies currently have formal programs devoted to it, due to various structural barriers and a general lack of awareness. Nevertheless, the UK government recently established a government-level Metascience Unit to examine and improve the research ecosystem, reflecting the growing importance of this field.

The Five Domains of Meta-Research

While meta-research can evaluate many aspects of research, five major domains provide a useful framework.

First, the Methods domain of meta-research identifies structural flaws, biases, and statistical errors in the research process and seeks to develop rigorous, standardized methodologies for future scientific inquiry.

Second, the Reporting domain focuses on maximizing research transparency and reproducibility by creating standard guidelines for sharing study protocols, raw data, and complete methodologies. Although specific reporting guidelines for various study designs are available through the Enhancing the Quality and Transparency of Health Research (EQUATOR) Network, and many of these standards are endorsed or recommended by journals and editorial organizations, adherence remains inconsistent.

Third, the Reproducibility domain systematically repeats or reanalyzes existing studies to determine whether the original scientific findings can be reproduced or replicated. When research cannot be replicated, it may lead not only to substantial waste of funding and resources but, in biomedical research, also to potential harm to patients. Therefore, continuous efforts are needed to investigate reproducibility and improve scientific reliability through systemic reforms, such as facilitating the publication of negative findings and reconsidering academic incentive and reward systems. As part of these efforts, the UK Reproducibility Network (UKRN) was established, inspiring similar national networks worldwide, while Stanford University established the Stanford Program on Research Rigor and Reproducibility (SPORR) within its School of Medicine to strengthen meta-research education and scientific training.

Fourth, the Evaluation domain encompasses not only the assessment of research through peer review, but also the important question of how researchers themselves should be evaluated using fair and robust criteria.
Fifth, the Incentive domain addresses how academic rewards and incentives can be structured to foster a healthy scientific ecosystem. Without careful consideration of funding allocation and academic incentives, research can become heavily concentrated in fashionable or high-profile areas, while rarer but critically important diseases and research areas may receive insufficient attention.

These domains do not exist in isolation. Rather, they are closely interconnected. Through continuous self-reflection driven by meta-research, we can establish a rational scientific ecosystem that minimizes bias, promotes rigorous methodology, supports transparent and open science, and encourages adherence to reporting guidelines. This holistic approach can improve the reproducibility and reliability of research while ensuring that negative findings are also valued and evaluated fairly. Ultimately, cultivating such a robust and self-correcting scientific ecosystem can better equip us to address complex medical and scientific challenges in the future.

Applying Meta-Research Beyond Research Itself

Once we recognize the concept and importance of meta-research, we can apply it to virtually any academic field. Meta-research allows us to examine past practices, identify important gaps, and develop directions for improvement.

Regarding Evidence-Based Medicine (EBM), although it was introduced into medicine more than 30 years ago, many ambiguities remain regarding what to teach, how to teach it, and how teaching methods should be differentiated between undergraduate and postgraduate medical education. Furthermore, the definition of “evidence” itself remains subject to debate. When interpreting evidence, we should look beyond the results of randomized controlled trials (RCTs); we need to build the capacity to evaluate whether benefits and harms have been appropriately analyzed and whether the evidence is being correctly interpreted.

Medical students who learn to critically examine research through meta-research will not blindly trust an author’s conclusions when reading papers. Instead, they can develop the critical eye needed to detect flaws in study design, identify biases hidden within the data, and distinguish evidence that is both credible and applicable to real-world patients.

In addition, university presidents and deans can use the principles of meta-research to examine how institutional policies are developed, evaluated, and implemented. For example, students often experience significant mental health problems, such as depression, anxiety, and suicidal ideation, related to academic stress and other challenges. Consequently, it is important to critically evaluate mental health programs across universities, including how effectively they are implemented and the extent to which they are based on evidence.

From Meta-Research to a Scientific Community

Over the past decade, I have conducted meta-research alongside internationally recognized experts in the field, including Professor John Ioannidis. Returning to my primary discipline of pediatric nephrology, I came to recognize how much work remained to be done. I subsequently gave an invited lecture to the European Society for Paediatric Nephrology (ESPN), which ultimately helped pave the way for the establishment of the “Meta-Research and Quality of Evidence” Working Group, which I now lead within the society.

Furthermore, I am planning to establish a Meta-Research Community at Yale University in collaboration with Professor Chul Hyun, Director of the Gastric Cancer Prevention and Screening Program at Yale School of Medicine. Our goal is to create a collaborative forum that brings together investigators interested in improving how research is designed, conducted, reported, evaluated, and translated into practice. While meta-research remains an emerging institutional field, I envision our efforts growing into a multidisciplinary think tank that can contribute to research across academic departments and scholarly societies and potentially inform broader university research policies.

In an increasingly vast and complex scientific landscape, meta-research can serve as a compass, helping us continually examine where we have been, determine whether we are moving in the right direction, and improve the path ahead.