source: Glen Edelson. Copyright 2008. http://flic.kr/p/7EJYMw
By Andy Parsons, PT, DPT
I have seen plenty of plenty of news lately about people scrutinizing science. A recent piece on FiveThirtyEight.com highlighted some of the problems researchers have when creating and interpreting research. Specifically, FiveThirtyEight focused on the problem of p hacking. P hacking refers to "massaging" a data set after an experiment until the data show statistical significance. The data can also be affected by choosing or excluding which variables are studied (referred to as the "degrees of freedom" within research). These "degrees of freedom" can allow scientists to intentionally or unintentionally affect the outcome of their studies.
I stumbled onto this TEDTalk, "Battling Bad Science". It highlights a major concern in scientific literature- publication bias. Publication bias occurs when journals or authors report only positive results and fail to report negative or neutral findings. In his TEDTalk, Ben Goldarce paints this methaphor relative to publication bias:
“In fact, 76 percent of all of the trials that were done on this drug [reboxetine] were withheld from doctors and patients. Now if you think about it, if I tossed a coin a hundred times, and I’m allowed to withhold from you the answers half the times, then I can convince you that I have a coin with two heads. If we remove half of the data, we can never know what the true effect size of these medicines is.”
How do these concerns relate to the PT world? The PT profession is relatively young in the grand scheme of the medical world. Quality PT research is being published in journals like the Physical Therapy Journal, the Journal of Orthopedic & Sport Physical Therapy and Archives of Physical Medicine and rehab, but, in general, our professions body of knowledge is small in comparison to physicians. You must remember that bias is an inherent part of science, and you must critically analyzing the literature before implementing a new practice. (Or you should consider the lack of literature in the case the case of the "Next big thing") Remain skeptical and allow the scientific process vet the poor procedures and point you towards the good ones. You want to be "cutting edge" but no over-reactive to the "next big thing".
How do we fix science's woes? Transparency is one solution. More trial methods are being published prior to data collection, so methods can be strengthened through peer review (sort of an open source concept). Also, we expect to see results from a trial when we know it's being studied. Ideally, this will reduce authors and industries from withholding negative results. (aka publication bias). The Center for Open Science (COS) is one group that is leading the charge on data transparency. A portion of their vision statement states, "COS supports shifting incentives and practices to align more closely with scientific values." Cochrance is another source that focuses on creating highly-quality, synthesized data that raises the bar for research quality.
FiveThirtyEight.com asked the question, "Is science broken?" Answer: No. "It's just more difficult than most of us realize."
“The scientific method is the most rigorous path to knowledge, but it’s also messy and tough. Science deserves respect exactly because it is difficult — not because it gets everything correct on the first try. The uncertainty inherent in science doesn’t mean that we can’t use it to make important policies or decisions. It just means that we should remain cautious and adopt a mindset that’s open to changing course if new data arises. We should make the best decisions we can with the current evidence and take care not to lose sight of its strength and degree of certainty. It’s no accident that every good paper includes the phrase “more study is needed” — there is always more to learn.”
Banner image source: https//goo.gl/BF1dmg
** This information is not intended to replace the advice of a physician/ physical therapist. Andy Parsons, PT, DPT disclaims any liability for the decisions you make based on this information.