A new study has come out in the journal the Lancet, and we have received many emails requesting that we address it. The study is this one:
The study type was a review; so it is not primary research, rather secondary research that sorts through and gathers all the research published to date.
Although the title of the paper claims that it was a systematic review, it’s not exactly a systematic review in the classic evidence-based medicine sense. Systematic reviews address a specific question (called a PICO question), asking how an exposure (like vitamin D) relates to an outcome (like cancer) or outcomes (like cancer and cardiovascular disease, etc.). That’s not what this paper did; there’s no clearly defined PICO question or questions.
What this paper did was collect all the prospective cohort studies to date that looked at vitamin D’s relationship to a disease. And they collected all randomized controlled trials to date that looked at vitamin D’s relationship to a disease. So in all, they examined studies that covered a plethora of diseases. They wanted to know the following:
- What kind of results are prospective cohort studies showing? Prospective cohort studies are observational, not controlled trials.
- What kind of results are randomized controlled trials showing?
- Is there a gap between what these observational studies are showing and what the randomized controlled trials are showing?
- If there is a gap, why is there a gap? Can we develop a hypothesis why there is a gap?
So they searched PubMed and Embase for prospective cohorts and randomized controlled trials. They had various criteria for whether or not to include a study in their analysis. One notable inclusion criteria for randomized control trials was that trials had to use doses of at least 2,000 IUs of vitamin D and participants had to start below a level of 20 ng/ml before they started the trial.
In all, they found 290 prospective cohort studies and 172 randomized controlled trials.
Here’s what they found:
- In general, prospective cohort studies showed more robust results than the randomized controlled trials.
- Many specific diseases that prospective cohort studies showed promise for, didn’t have the same kind of promise and proof from randomized controlled trials.
- Randomized controlled trials were generally considered failures for their lack of significant improvement in target outcomes. To give an arbitrary example, if a randomized controlled trial assessed 15 different outcomes (like multiple cardiovascular health markers, for example) and only had 3 significant outcome improvements, the authors’ general attitude was that vitamin D was not beneficial (more on this later).
- The type of outcome in randomized controlled trials that best matched the results of the prospective cohort studies was all-cause mortality. In other words, randomized controlled trials generally show that vitamin D protects against all-cause mortality, just like prospective cohort studies do.
The researchers concluded that since randomized controlled trials don’t perfectly match prospective cohort studies, then vitamin D deficiency must be linked to diseases via reverse-causality or confounding factors. An example of reverse causality is the following:
- Vitamin D deficiency doesn’t cause depression; depression causes vitamin D deficiency because it causes people to stay indoors.
An example of a confounding factor:
- Vitamin D deficiency doesn’t cause depression; a lack of UV exposure causes depression and a lack of UV exposure causes vitamin D deficiency.
The above examples are arbitrary examples, so please don’t take either statement to be necessarily true.
While it could be the case that vitamin D deficiency is a victim of reverse-causality or confounding factors and vitamin D interventions won’t prevent or treat some diseases, we’re not at the point in research to make that call. Here’s why:
To date, randomized controlled trials have been mostly small and exploratory. We’re not exactly sure how vitamin D affects the body. We think we’re very likely not getting enough, but how that affects us, we’re still trying to figure it out.
Since the randomized controlled trials we have setup to date are mostly exploratory, our trials will always look like failures to a certain extent. We’re not sure what kind of markers vitamin D will affect, so we list many markers as outcomes we’re interested in. In these trials, when vitamin D only improves 20% of the markers, some researchers say vitamin D is no good. Other researchers will say, “We’ve discovered which markers vitamin D is good for.”
These kinds of exploratory randomized controlled trials have paved the way for large randomized controlled trials now finally underway. These trials include:
|Name||Place||Participants||Amount of vitamin D||Outcomes||Year of results|
|VITAL||USA||20,000||2,000 IU/day||Cancer and heart disease||2017|
|FIND||Finland||18,000||1,600 IU or 3,200 IU/day||Cancer, heart disease and diabetes||2020|
|VIDAL||UK||20,000||60,000 IU/month||Longevity and others||2020|
When these trials conclude, we will better be able to compare prospective cohort studies and randomized controlled trials. But not until then. The reason is, these much larger trials will have robust enough data to look at simpler, more clinical outcomes, like cancer incidence, stroke incidence, as opposed to just having to look at markers for health.
And just because randomized controlled trials haven’t shown robust results like prospective cohort studies have yet, doesn’t mean there should be any “new doubts” on vitamin D. It is likely that not all randomized controlled trials will show the same benefits in vitamin D as prospective cohort studies do. If randomized controlled trials failed for nine out of every ten diseases, would you not take vitamin D? No, you still would, for prevention or treatment against every tenth disease.
The argument for supplementation and good sun exposure habits has always been that hunter-gatherers have much higher vitamin D levels than those living in developed countries. The logic is not based on observational outcome evidence. This remains the logic behind the pro-vitamin D crowd until the large randomized controlled trials above are published in 5-10 years.
Lastly, if you get in a debate over the merit of this study with friends or colleagues, don’t let them tell you this was a well-designed systematic review! As mentioned to start, this review didn’t have a PICO question for one, and for two, its methodology for searching for studies was not comprehensive because they did not use all the search terms they needed to for vitamin D. As a result, I’ve already noticed they didn’t include a very promising small randomized controlled trial by Derakhshandi H et al on multiple sclerosis and vitamin D. For a more comprehensive and thorough search term, see our post on methodology here.