A note on peer reviewed studies

Many won’t give an idea credence unless a peer reviewed study exists that confirms it.

While this is fallacy, an appeal to authority, I am definitely not opposed to peer reviewed research. Quite the opposite. You’ve probably noticed that I’ve cited a variety of research sources in many of my pieces. I don’t make any of this stuff up. Often, when I perceive or find something to be true, I seek out research on the subject to learn more about what correlative evidence has been confirmed either way.

All of that said, I also worked in varying capacities in higher education for over 7 years, and much of that oversaw financial operations for sponsored research… actual, NSF/NIH/etc funded research. I not only reconciled expenses but also had to assist with the research reporting on those research projects.

So don’t take the following as a vilification of scientific research. I see for myself the value and process of that scientific research.

But it is worth noting that most of what we know about life hasn’t and probably won’t be scientifically researched and peer reviewed to officially stamp with approval.

To conduct peer reviewed research on a subject, any subject:

  • Your research has to be funded by a 3rd party. Universities almost never directly fund any research. In fact, every major university has a Sponsored Projects collective of Research Administrators whose sole job it is to help faculty members apply for and maintain external research funding.
  • You must work and be experienced specifically in the field or subject that you are going to research, or find and develop a partnership with professionals equipped to research in the field or subject of interest. Sponsors typically aren’t just handing money to anyone who wants to study something. You must demonstrate that you have the expertise and means to study a given topic of interest.
  • You must formally propose in a report to a desired sponsor a specific need to research a specific subject, one that can be observed and measured. Scientists aren’t just baseball-scouting the entire scope of their field subject researching, peer-reviewing and verifying everything that possibly could be studied in their field. There’s just no way and there just isn’t the resources available to do that.
  • As implied, you must have the infrastructure and organization available to conduct the research you state your plan to conduct. While sponsors will fund your research, they generally aren’t going to buy you a lab. They expect you to already have the infrastructure to conduct your desired research, and your sponsored funding will merely pay for incidentally relevant equipment, supplies, and personnel.
  • You typically cannot diverge from your proposed subject matter once you begin a sponsored project. If during research you discover a whole different relevant project to research, you can report that possibility as part of your findings, but your sponsor isn’t just going to pay for you to go off-topic and research that thread. You will need to apply for separate funding for a separate project to do so. The sponsor expects you to finish research and deliver a conclusion to the existing topic proposed within the stated time frame.
  • These research projects usually aren’t done in a week. The average project takes several years to complete.
  • A typical subject of research is often very granular and isolated, e.g. the effects of vitamin D supplementation on the red blood cell count of sedentary 40-something white adult males. To go any broader would so diffuse the quality of the research that your likely sample of data won’t provide any useful correlation. This gets into conceptually basic but potentially-brain-exploding-complex statistical data concepts like standard deviations and sample sizes. Basically, research projects have to be very specific and research conducted in very controlled settings, for the results to usefully matter.

I note all of this to point out that the stated results of a study often were carefully studied within a very particular set of circumstances. Within that set of circumstances, the correlative evidence is almost certainly sound. However, this is typically only true within that given set of circumstances.

For example, to do a study on the effects of almonds on digestive health, then find a correlative benefit from the almonds in the studied circumstances with the studied individuals… but then state, “Therefore almonds are always healthy for your gut!” is fallaciously inaccurate (possibly disingenuous, depending on the given perspective).

You only noticed a positive effect on the set of individuals studied, under those particular circumstances, and there may be various unintended factors behind what you observed.

For example, what else were they eating? What kind of diets did they practice in general? Were they all eating the same thing every day or did every person have their own individual diet? Did the study sample effectively represent the portion of the population that has a nut allergy or intolerance?

How active were the people studied? Did they all burn similar calories every day, or were some busier than others? Were they quarantined at a facility or just came in to visit? Were results physically studied from lab samples and measurements, or from anecdotal reporting and questionnaires?

All that context has an effect on what studied results were observed. And yet, that context is often completely omitted when people reference the results of studies.

So, even within the credible realm of peer-reviewed research, the circumstances alone behind any research can call the caveats of the observed evidence into question.

And all that never minds everything I previously brought up regarding complications and needs that go into any individual subject being studied in the first place. Research and what is now blanket-labeled science is a lot more micro-specific in scope than people realize.

So, now, the flip side: A lot of what we know to be true has not been objectively studied by scientists and specialists, let alone has any research been peer-reviewed. We’re going off common sense and anecdotal evidence for a lot of what we know about the world.

It’s not particularly smart to vilify a belief or principle due to a lack of peer-reviewed scientific evidence, unless the belief itself is so outlandishly out of touch from what we generally know to be consistently true that the burden of proof should fall on whoever claims that belief.

If in doubt, you’re more than welcome to propose a peer-reviewed study on any topic you believe has not been researched… and see for yourself exactly how hard it is to get peer-reviewed research for that subject.

Tagged , , , ,

Leave a Reply

%d bloggers like this: