Separating Fact from Fiction in Clinical Studies
RESEARCH
What’s ‘Good’ Research and What’s ‘Not-So-Good?’
Anthony Rosner
PhD
In today’s maelstrom of fake news, random tweets, obsessive tweets, and attacks on science by various government officials, it may be possible to lose sight of the importance of clinical evidence and reading it carefully and objectively. Yet we must seek objective truth in all that we read and view, no matter if—especially if—the truth is uncomfortable or contrary to our expectations. As Philosopher and radio host Stefan Molyneux says, “Truth has nothing to do with the conclusion, and everything to do with the methodology.”
The following quick guide is not exhaustive, but is designed to encourage critical thinking about the quality of evidence, and how, and when, you should consider applying it to your practice.
Red Flag 1: Alarmist and Unsubstantiated Statements.
Peer-reviewed journals as well as mainstream press seek readers, and salacious headlines grab them. Consider those attempting to link spinal manipulation with stroke. At best, they have demonstrated an extremely limited association that is at least an order of magnitude less than that caused by spontaneous arterial dissections.1 It has been shown that the probability of a vertebral artery dissection in symptomatic patients is at least as great if the patient visits a medical doctor’s office rather than a chiropractor,2 and that the vast majority of vertebral basal artery dissections are spontaneous, cumulative, or caused by factors other than spinal manipulation.3
^Peer-reviewed journals as well as mainstream press seek readers, and salacious headlines grab them. J J
Red Flag 2: Design Flaws in Research (here are few common ones that should raise your eyebrows).
a. An adequate description of manipulation is absent with occasional descriptions of mobilization rather than manipulation.
b. The clinical characterizations of the sample population are unclear.
c. The qualifications of those individuals administering treatment are not reported.
d. Physician-patient contact times are not uniform across compared groups.
e. Sample sizes are often too small to reach statistical significance.
f. There is the failure to observe or control characteristics at the baseline of a study.
g. Experimental bias may be introduced into a clinical trial, often implicit in the recruitment of participants.
Red Flag 3: Internal Biases.
a. A publication in the Journal of the American Medical Association described a series of meta-analyses which addressed the comparative abilities of two fractions of heparin to inhibit clotting. It turns out that these investigations delivered diametrically opposing conclusions, depending upon which criteria of scoring were chosen by the authors. The result was an entirely subjective study that is plagued by bias.4
b. A drug manufactured by a competing pharmaceutical company was delivered in a manner that it was bound to fail when compared to a product made by a second manufacturer that bore virtually all the expenses of that trial. In other words, the trial was rigged so that the product of the sponsoring organization was guaranteed to prevail.5 Promising results in a pilot clinical trial addressing
c. the ability of chiropractic to reduce both the prostaglandin levels and symptoms of dysmenorrhea6 were overshadowed by a subsequent full trial that relaxed the admission criteria of participants. Because of this, participants entered the trial virtually symptom-free, such that this investigation could not help but produce negative results7 through no shortcomings of the chiropractic treatment itself.
As you can see, there is more than meets the eye in any research—and as Molyneux so aptly said, methodology is
everything. One example that can claim such rigor is a study published in spring 2017 in the Archives of Physical Medicine and Rehabilitation showing that custom orthotics, particularly in combination with chiropractic care, significantly reduced Low Back Pain (LBP), by 34.5 percent alone and by 40.4 percent in combination. Function also improved by 18.5 percent and 32.3 percent, respectively. Although the study was funded by Foot Levelers, who also provided the custom orthotics used in the trial, it was undertaken by a respected independent institution, the National University of Health Sciences. Outcomes were assessed at 12 weeks, and at three, six, and 12 months, for compelling results.8 To my knowledge, Foot Levelers is the only custom orthotic company that can claim to be supported by this degree of detailed and persuasive methodology.
Quality research is painstaking and expensive, and can take a long time to work its way through the machinery of academia. It is also important to keep in mind that well-crafted individual case studies and case reports are often of greater value than deeply flawed randomized clinical trials, such as outlined here. But—as with any piece of information—they should be considered with the open, critical eye.
References:
1. Haneline MT, Rosner AL The etiology of cervical artery dissection. Journal of Chiropractic Medicine 2007; 6: 110-120.
2. Cassidy JD, Boyle E, Coe P, He H. Hogg-Johnson S, Silver FL,
Bondy SJ. Risk of vertebrobasilar stroke and chiropractic care: Results of a population-based case control and case-crossover study. Spine 2008; 33(4S): S176-S183.
3. Rosner A. Spontaneous cervical artery dissections and implications for homocysteine. Journal of Manipulative and Physiological Therapeutics 2004; 27(2): 124-132.
4. Juni P, Witsch A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analyses. Journal of the American Medical Association 1999; 282(11): 1054-1060.
5. Johansen HK, Gotzsche PC. Problems in the design and reporting of trials of antifungal agents encountered during meta-analysis. Journal of the American Medical Association 1999; 282(18): 1752-1759.
6. Kokjohn K, Schmid DM, Triano .1.1, Brennan PC. The effect of spinal manipulation on pain and prostaglandin levels in women with primary dysmenorrhea. Journal of Manipulative and Physiological Therapeutics 1992; 15(5): 279-286.
7. Hondras MA, Long CR, Brennan PC. Spinal manipulative therapy vs. a low force mimic maneuver for women with primary dysmenorrhea: A randomized, observer-blinded clinical trial. Pain 1999; 81(1-2): 105-114.
8. Cambron JA1, Dexheimer, Duarte,
Anthony Rosner is a champion of interdisciplinary research methodology in the health sciences, having previously served as Director of Research and Educapon at the Foundation for Chiropractic Education and Research, He was designated as Humanitarian Year in 2000 by the American Chiropractic Association and holds an honorary degree from the National University of Health Sciences. He obtained his Ph.D. from Harvard in Medical Sciences/Biochemistry.
