Systematic review vs meta-analysis

At Lexunit, we had the opportunity to work with one of the biggest non-profit organizations in the world and learned about a topic that interests many researchers we like to work with. Researchers are people who build fundamentals that shape our understanding of the world around us. Protecting us, by giving visibility into certain topics that would require better regulation to protect the future of our next generations.

In other cases, research gives a competitive edge. R+D ( Research and Development ) investments always make the entry barrier higher for others in the same market. This is how big tech giants keep their monopoly, by investing in R+D, thus making the life of the competition harder. Research could be very expensive, and hard to understand the ROI ( Return on Investment ), it is a complex problem and it is a privilege when a company can allow itself to invest in pure R+D.

Anyway.

For us at Lexunit, this interest is relevant – it connects to our expertise through natural language processing and AI. We see the potential for AI and machine learning to streamline and simplify many parts of the systematic review process.

Working with non-profit organizations gave us firsthand insights into the challenges of managing massive volume of information systematically and fairly, a task that becomes central in fields like policy and health where objective insights can be crucial.

I want to simplify things and give you a way to think about systematic reviews. I asked myself the following questions to kick off the thinking process; I hope it will kick yours as well. When researchers need an objective reference point to build upon, they do systematic reviews to understand the state-of-the-art knowledge of a certain topic to use as a baseline.

• What is the purpose of doing a systematic review?
  - To create the best objective, unbiased summary of a topic of interest by analyzing related public and private scientific literature.
• What is the difference between a systematic review and a meta-analysis?
  - Meta-analysis is part of systematic review. You can imagine a systematic review as a study of studies on a certain topic of interest. Meta-analysis is a sub-activity within systematic review, used when quantitative analysis can be executed on different studies.
• Why and how much money do companies spend on systematic reviews?
  - It depends on the industry, of course, but as a reference point, read this article: The significant cost of systematic reviews and meta-analyses. I don't want to take it as fact, because I know it depends on the size of the project, operation. Anyway the goal to reduce the cost associated with this activity.
• How to automate the process of systematic review?
  - Centralized searching. Look for tools.
  - Online collaboration and management tools can reduce the cost of doing a systematic review and could increase the experience of researchers.
  - Machine learning can be a game changer in this topic with the appearance of large language models.

This is where Lexunit sees the future – these activities we're familiar with, such as automated text analysis and document processing in many ways. These can reduce the manual work and human error that often come with large-scale reviews.

Of course there are ethical considerations of relying on AI: while it reduces manual workload, it also introduces new risks:

 If algorythim is used in biased data then it's output will be way off. For example wrong selection of studies could badly represent a given problem. With the data we always try to best represent the problem in order to achieve the best result. Any automation we explore or implement in this space needs careful oversight to avoid losing the objective integrity of systematic reviews.

Key Differences Between Systematic Review and Meta-Analysis

• Systematic Review: A comprehensive synthesis of all empirical evidence related to a specific research question, using a systematic method to minimize bias.
• Meta-Analysis: A statistical technique that combines results from multiple studies to derive a pooled estimate of effect.

I researched the web for available tools to make systematic review easier:

Tools for Collaboration around Systematic Review

• Covidence: A tool that supports the entire systematic review process, including screening, data extraction, and collaboration.
• Rayyan: A free tool for systematic reviews, especially useful for collaborative screening and managing multiple reviewers.
• EPPI-Reviewer: Comprehensive software for managing systematic reviews, designed for advanced collaborative research.

Literature Indexing Portals and Centralized Search

1. PubMed: A free search engine for accessing the MEDLINE database of biomedical and life sciences literature.
2. Google Scholar: A widely used academic search engine that indexes scholarly articles across various disciplines.
3. Web of Science: A research platform that offers access to multiple databases, supporting literature reviews across fields.

Automated Systematic Review and/or Meta-Analysis

1. RobotReviewer: An AI-based tool that assists with screening and assessing studies for systematic reviews, focusing on reducing manual effort.
2. ASReview: Software that uses active learning to expedite the screening process by helping prioritize relevant studies.
3. RevMan Web: A tool by Cochrane for creating systematic reviews and meta-analyses, providing automation for data extraction and analysis.

Factual Comparison of systematic review vs meta-analysis

To simply understand the process of a systematic review, I would like to give you the high-level steps to do that:

1. Preparation: design the workflow, ask the right questions, transparent reasoning, define methodologies, inclusion/exclusion criteria, etc.
2. Collect all relevant studies, online sources, different databases, and other offline sources.
3. Filter studies to a shortlist of documents for further analysis.
4. Analyze the shortlist of documents from a qualitative and quantitative perspective.
5. Draw conclusions
6. Publish findings

The problem with these reviews is that simply leaving out certain studies from analysis could hijack the outcome. This can be used to craft business advantage or can hide important details. We concluded that digitalization and automated workflows could eliminate corruption/misuse initiated from the top and forced down to the bottom of the organization.

Gathering studies is cumbersome when a researcher needs to collect documents from several different literature indexing portals. Why? Each portal requires the researcher to come up with a slightly modified version of the query that gives the best result. We found that researchers tend to query and collect documents with a broad approach, meaning writing queries to the portals that yield a more complete results listing with many irrelevant documents.

After getting the results from each portal, the researcher needs to eliminate duplicates from the documents and filter out irrelevant ones, as most studies are listed in multiple indexing portals. This requires much manual work, sorting through irrelevant documents for hours.

Let’s say at this point, the researcher has a shortlist of documents representing the research topic objectively and without bias, and now the real work can begin: analyzing the papers one by one, answering questions, collecting data, and organizing it for easier processing.

Why is it getting harder to do?

The number of studies coming to light year by year is increasing exponentially in certain topics. Of course, some topics lack enough attention and would benefit from more focus. I checked openalex.org to get base numbers and found 260,900,000 results without any search criteria. The number of published studies peaked at around 10 million in 2018, from that time the world produced 10 million studies annually. This is just one platform but gives a good baseline estimate.

When we start querying indexing sites to download studies with a simple query, we easily retrieve 30-40 thousand studies. After filtering and evaluation, this might be narrowed down to 500 relevant studies. How much time to manually read through 40,000 abstracts or even half of that? A lot. Here’s where automation and online collaboration make this process simpler and more enjoyable, just to get to the state-of-the-art knowledge.

With AI and large language models, this field sees great improvement. Lexunit, with a background in natural language processing, sees this as a tremendous opportunity to enable researchers to sift through millions of documents efficiently. Although reaching the right set of documents requires tweaking and analysis, this work is ultimately worth the effort for organizations working globally.

In summary, systematic reviews and meta-analyses each play crucial roles in research. Systematic reviews provide a broad, unbiased view of available literature, while meta-analyses add statistical depth to assess trends across studies. Knowing when and how to use each is key for critical evaluation and practical application of research findings.

Both AI advancements and Lexunit’s expertise in NLP allow us to imagine a future where access to unbiased, objective knowledge is simplified and improved.

In case of questions or feedback please send us a message here.

References

1. Introduction | Systematic Reviews and Meta-Analysis - Oxford Academic
2. Meta-analysis and systematic review - Deranged Physiology
3. The Differences Between a Systematic Review vs Meta Analysis - Distillers Research
4. The difference between a systematic review and a meta-analysis - Covidence
5. Systematic Reviews and Meta-Analyses - PMC