In the first part of this blog series, we looked at some general criteria to judge the quality of a scientific publication and why it is important to know about this in the first place. Going more into detail, the second part discussed the importance of using a suitable study design.
This last part of the blog series will now concentrate on the results. How we can know if they are reliable and how we can decide whether they allow the drawn conclusions.
Are the results supporting the hypothesis?
The results should be presented objectively, and the discussion should explain why these results are able to support the afore mentioned hypothesis.
- Do the results allow the conclusions that are made?
Sometimes conclusions are drawn from ambiguous data. The difference between causality and correlation needs to be clear.
- Are all results included in the analysis and the discussion?
If some results do not appear in the discussion, this is a hint, that the outcomes were not as expected. Moreover, if only a fraction of the participants enrolled in the study are analyzed, it is likely that some participants were censored as they did not meet the envisioned results or that many patients dropped out of the study (e.g., due to toxicities or lack of efficacy).
- Can the results be generalized to the clinical practice?
This depends a lot on the study population. If the participants had to fulfill a lot of very specific criteria to meet the presented result, the treatment might not be applicable to clinical practice.
- Are the results objectively presented?
The results should be stated without any rating. If results are not presented objectively, there is a great potential for bias of the authors.
- Does the discussion state whether prespecified endpoints have been met?
At the beginning of a trial the hypothesis and expected outcome should be defined. Conclusively, the discussion of a scientific publication should include a statement on whether the expected outcomes were met, if the hypothesis is correct and conclusively, if the study is positive or not.
- Do the researchers disclose possible limitations of the study in the discussion?
If the discussion includes a section on the limitations of a study this is a sign that the researchers are aware of potential flaws and are objectively presenting the results, inviting the reader to evaluate the outcome.
Are the study results statistically significant?
As clinical trials usually deal with a great amount of individual data, a statistical analysis of the obtained results is indispensable. Statistics can be difficult to understand, however, there is a few simple things to check for:
- Which statistical methods were used?
For people who are not familiar with statistical methods it is still recommendable to check whether a statistical analysis was performed. Health Care Professionals should invest the time to understand standard statistical methods and the meaning of statistical significance.
- What do odds ratios and confidence intervals mean?
Odds ratios are often used for safety outcomes and show how different treatments compare to each other regarding side effects. An odds ratio of 1 means that the occurrence of a certain side effect was the same in both groups. An odds ratio below 1 means the side effect occurred less frequent and an odds ratio above 1 means the side effect occurred more frequently in the assessed treatment group compared to the comparator (e.g., placebo). A confidence interval shows the interval that contains 95% of the results. If the confidence interval is very broad, the results varied a lot from participant to participant. Regarding efficacy this could mean, that in some participants the respective drug was very effective while in others no benefit was seen.
- Was statistical significance demonstrated?
It should be stated in the discussion if statistical significance was reached. Treatments with a very low risk of side effects, could still be considered in practice if statistical significance is not met but the researchers could show that the benefits most likely outweigh the risks and that no harm is done if the treatment would not be effective.
Herea Menopause Network is committed to make ongoing research available for everyone to understand. We hope that this guide will help you to evaluate the content of scientific studies more easily.
