Nowadays it is very easy to access a wide spectrum of information on various diseases via online media. While most people are aware that they shouldn’t believe everything they read online and have learned to look for sources like scientific papers, rating the quality of these papers can be somewhat difficult.
Clinical evidence proven by randomized, controlled trials as well as real-world studies is the gold standard for treatment decisions. Especially for conditions that are not well studied yet or which are not a central focus in medicine and society, as it is still true for menopausal symptoms, it can be helpful to stay up to date with current research. However, not all scientific studies hold up to the same standard and publishing processes can be confusing for people who are not familiar with the quality control mechanisms for scientific publications. Moreover, a scientific layout can be used to make non-validated information appear scientifically accurate. Such claims can thereby be quickly taken up by the media and broadcasted to a wide audience with no quality control in place.
Therefore, it is important for everybody to have a basic understanding of how scientific publications should be handled, how you can find out whether a study is trustworthy or not and how well-assessed results can be distinguished from bold claims that are not backed up by data. This is the first of three blogs that will give you an overview of criteria that you can use to find out if a study is credible or not. This blog focusses on things that can help you judge a paper even before looking at the data. In the two upcoming blogs, quality criteria for the methodological setup of a study and the representation of the results will be addressed.
Not all scientific data can be trusted equally. Sometimes it might be legitimate to “judge the book by its cover”.
To evaluate whether the research paper you are about to read meets some standard quality criteria, there are a few things, you can check even BEFORE reading the paper:
- Is the paper published in a peer-reviewed journal?
High impact scientific journals use a peer-reviewing process to ensure the quality of their publications. Before a paper gets published, it is submitted to several peer reviewers (experts in the respective field) who assess the quality of the manuscript. This process is either double blinded (the peer reviewer does not know who authored the paper and the author does not know who did the peer review) or at least single blinded (the author does not know who did the peer review). The experts may request further data, if they are not convinced by the results presented or changes to the manuscript if it is not clear enough. Only if all expert peer-reviewers agree that the presented content is worth publishing, the paper can proceed to publication.
Journals which do not require a peer-review process are generally less trustworthy and can contain non-validated data.
- Is it a preprint?
Besides submitting to a peer-reviewed journal, it is possible to publish manuscripts on a preprint platform. This means that the presented data has not been reviewed by other experts. It is possible that a paper gets declined by renowned journals due to lack of quality but still exists as a preprint.
- How many authors are involved?
A clinical trial involves a lot of work, often several different institutions are involved. If a paper only has one or two authors it is most likely rather an opinion, a summary of recent data (literature review) or a summary and comparison of recent data (review and meta-analysis). The latter can give valuable information on how different treatments available on the market compare to each other.
- Where and when was the study conducted?
Is the study up to date? Standards for clinical trials are currently evolving, an older study might not hold up to current quality standards anymore. Origin, ethnicity, and environmental factors can have a strong impact on the efficacy of certain treatments. The characteristics of the study participants should be comparable to you/your patients to draw conclusions.
- Is the respective trial registered at clinicaltrials.gov or a national registry?
A trial has to fulfill certain quality criteria to register at a national or international registry.
- Is there any conflict of interest from the researchers?
How was the study financed and are the authors involved with anyone who could benefit financially from the presented study outcome? Clinical trials are very expensive and therefore rely on the financial support from pharmaceutical companies. To prevent any possible exploitation and ensure complete objectivity, the communication in pharmaceutical industry is highly regulated by national and international authorities.
