Meta-analysis has become a popular tool for summarizing data from a body of work that investigates a common research question. Because this provides an objectively objective means of summarizing a set of data, it has become the gold standard for testing when health guidelines are developed.
However, there are several problems in the meta-analysis that can contribute to inaccuracies. Because of these limitations, the meta-analysis – or "mega-silliness", to borrow a term from an early detractor – has been criticized for almost the time that the method has been available.
There are several tools to ensure the quality of meta-analyzes, which will be explained later in this post. But first, let's consider this considering this (fictitious) extract from the meta-analysis result section I will use to briefly illustrate three common meta-analysis problems:
As hypothesized, a meta-analysis that summarizes eight dimensions of effect from seven studies suggests that the use of social media is positively associated with self-reported anxiety levels [r = .073, 95% CI ( .004, 0.142); p = .038; Fig. 1A]. Egger's regression test did not suggest any evidence of publication bias (p = .19; Fig. 1B). "
When this imaginary author – who is also releasing a book on the dangers of social media – initially looked for studies, he actually found nine studies that met their inclusion criteria. However, the size of the summary effect of this meta-analysis of nine studies[[[[r = .062, 95% CI (- .004, 0.127)]he was associated with a p-value of .061 (Figure 2).
Because this result would hinder the possibilities of publication (and book sales), they gave a closer look at the study six, which reported a negative association between the use of social media and anxiety levels. . This study was the first study included in the meta-analysis.
Study 6 did not include the use of Instagram (which became popular in 2012), then adjusted the eligibility criteria to include only studies published from 2013 onwards, justified by an (post hoc) hypothesis that # 39; Use of Instagram was more associated with anxiety.
With the removal of the study, the meta-analysis became statistically significant. While this invented scenario might seem extreme at first glance, nearly 40% of over 2,000 psychologists interviewed admitted that they had excluded the data after examining the impact of that.
Pre-registration of the analysis plans
The pre-registration of the meta-analysis provides a remedy for analytical flexibility as it provides a first recording of the researcher's intentions of analysis. In the previous example, the exclusion criteria would have been specified prior to the execution of the analysis.
A common criticism of pre-recorded analyzes is that they offer no flexibility for analysis. However, this is not correct because the pre-registration does not block the researcher in a specific plan, but instead helps to clearly outline the pre-programmed and exploratory analysis.
In the case above, the researcher would be free to report his analysis with the removal of the study, but should declare his reasoning, which is rather thin. As an indication, the document Articles preferred for systematic reviews and meta-analysis protocols (PRISMA-P) provides a list of elements that should be treated in a meta-analysis protocol.
There are several options available for pre-registration of the meta-analysis. First of all, a meta-analysis protocol with timestamp following the PRISMA-P guidelines can be loaded on the Open Science Framework.
Secondly, if the meta-analysis investigates a result related to health, it can be pre-registered on PROSPERO. The PROSPERO website guides the researcher through a step-by-step registration form, which is used to generate a pre-registration record. These records can be edited, but any changes are timed.
Third, a meta-analysis protocol can be presented for peer review at a number of journals, such as systematic reviews. While the peer review process obviously requires more time to load a protocol, it can provide important feedback that can improve meta-analysis.
Fourthly, a meta-analysis can be presented using the recorded report format, in which the meta-analysis is examined in two phases. In the first phase, the introduction sections and meta-analysis methods are peer-reviewed before data collection. If the motivation of the study and the methodological approach are deemed appropriate, the journal will provide the provisional acceptance of the final document.
Once the data are collected and analyzed, the document is sent again for the revision of the second phase, which includes the same introduction and the methods of the first phase, but with the addition of results and conclusions. In the second phase of the review, the focus is on the fact that the results and the interpretation are in line with the pre-approved method.
It is important to stress that the acceptance of the card does not depend on the statistical significance of the results. As of August 2018, there are 42 journals that offer a metanalysis of Registered Report, with most of these journals publishing research on the bio-behavioral sciences.
Publication bias is a well-recognized problem in meta-analysis. Since statistically significant studies are more likely to be published, it is likely that there are several non-significant studies left to languish in the research files of researchers for each research area, which can not contribute to meta-analyzes.
Therefore, a dimension of the meta-analysis summarizing effect may not best represent the "true" effect dimension. A common approach to assess publication risk is to construct a funnel chart, which traces the size of the effect relative to a variance measure (eg, standard errors). A symmetrical funnel chart is indicative of a low risk of publication bias. Despite the inherent subjectivity of visually inspecting a funnel diagram to evaluate asymmetry tests, many researchers still rely on this tool.
Now let's go back to our imaginary studio. Since our fictitious researcher was aware of problems with subjective evaluations of the funnel plot, they objectively evaluated the asymmetry by constructing a funnel plot. Because the researchers are not able to identify the asymmetry from the funnel charts, they followed the recommendations and performed the Egger regression test to integrate the funnel diagram (Figure 1A).
A statistically significant test is indicative of the asymmetry of the funnel weft. In this case, the test was not statistically significant (p = .19). Therefore, our researcher concluded that there was no risk of publication bias. A closer look at the data, however, reveals that this conclusion was probably wrong.
A common misconception of funnel charts and associated tests is that they measure the distortion of the publication when they actually measure small study bias. This type of injury could include publication bias, but it also includes other types of injury commonly associated with small studies, such as the quality of the study.
A modification of the relatively simple funnel diagram, called a funnel-shaped frame optimized for the contour, can help to better understand the risk of publication errors compared to small study errors. For a given range of size and variance of the effect, it is possible to calculate the levels of statistical significance at each point. As a result, statistical significance intervals (e.g. p = .05 to .01) on a funnel chart.
Comparing our original funnel graph (Figure 3A) with a funnel-optimized graph for the contour (Fig. 3B) shows that, despite no apparent evidence of asymmetry of the funnel weft, half the size of the effect included had pthe values were very close to 0.05. It is highly unlikely that half of the available pthe values would be so close to 0.05. Both of these significant studies were also the result of selective analyzes or there are several unpublished studies that exist.
Several measures have been developed to correct the biases in the meta-analysis, such as the p curve, the p uniformity, and the three-parameter selection model, with the latter showing superior performance. But without access to unpublished research these methods will always be estimated, at best. A plausible solution to this scenario is to conduct a meta-analysis that includes only pre-recorded empirical studies, such as Gronau's recent meta-analysis on "posing power" and colleagues.
Now to our last common problem in meta-analysis: dependence on the size of the effect. Note that in the previous example of meta-analysis there were eight dimensions of effect but only seven studies? This was due to the fact that two dimensions of the effect were extracted from study four (Figure 1A). This is a common event because newspapers sometimes report different outcome measures.
However, because these dimensions of effect derive from the same study population, they are statistically dependent. Because typical meta-analysis procedures assume that the dimensions of the effect are statistically independent, the combination of dependent meta-analyzes can lead to distorted results.
There are different approaches to tackling statistical dependence in the meta-analysis. First of all, you can simply choose a size of the studio effect. While it's simple, you need a pre-specified system to help select the size of the chosen effect, otherwise it might be tempting to simply choose the larger effect. Furthermore, this approach excludes the size of the effect that is potentially important from the analysis. To include more than one effect from a study, it is possible to aggregate the effects statistically. But to do this, it is necessary to include a correlation measure within the study, which is hardly ever reported in the newspapers.
If the paper contains raw data, an event that is almost as rare, the correlations within the study can be calculated. Therefore, researchers almost always need to estimate the correlations within the study if this approach is to be taken. Alternatively, a robust variance estimate can be used to take into account the size dependency without the need to know the correlations within the study.
Meta-analysis is an important tool for a better understanding of the often disparate areas of research. This method can provide a more objective synthesis of a research area compared to the simple "counting of votes" of the number of significant and non-significant results.
But like almost all statistical tools, meta-analysis can facilitate inaccurate inferences if used inappropriately. With this post, I hope I have provided a brief overview highlighting some common problems in meta-analysis and practical solutions to these problems.