17 min read
What is sufficient (scientific) evidence?
I have already spent dozens of hours and tens of thousands of words under water, with my diving mask on, immersed in the scientific literature that others put forward as evidence for their claims. People often make statements about a causal relationship, meaning that X (nutrition) causes Q (health (problems)). But to my regret, my efforts too often lead to a disappointing conclusion: the evidence is not sufficient in relation to the claim (if the evidence can be used to support the claim at all).
This rightly raises an important question for many followers: what is sufficient (scientific) evidence, then? I will share my ideas on this with you, and give two examples: the working methods of the Gezondheidsraad (the Dutch Health Council, GR) and the World Health Organization (WHO).
The aim of this blog is not to convince you about what is healthy or not. The aim is to give you insight into the scientific process behind establishing a causal relationship. So that you start to see establishing a causal relationship as a process, instead of taking over a conclusion from a random study on pubmed. Enjoy reading. (translated from Dutch)
The common thread:
If we want to demonstrate a strong causal relationship in nutrition science (and perhaps in all health sciences?), this process usually consists of several components:
The exposure (for example a dietary pattern, a food group or a nutrient).
An intermediate outcome (for example blood pressure).
A clinical outcome (for example coronary heart disease).
An important difference between blood pressure and coronary heart disease is that blood pressure is called an intermediate outcome measure, while coronary heart disease is a clinical outcome measure.
This is an essential difference, because improving intermediate outcome measures is never a goal in itself. You want to influence the health outcomes (the clinical outcome) that are related to the intermediate outcome measure.
In other words, you want to keep your blood pressure low to prevent developing coronary heart disease.
Examples of other intermediate outcomes:
blood glucose
insulin
LDL cholesterol
weight
inflammation
etc.
We prefer to always study the relationships between these components with experimental research. Experimental research provides stronger evidence for a causal relationship than observational research.
If you want to learn more about the different study designs and the pyramid of evidence...click here.
Ideally, we mainly study the relationship between the exposure and the clinical outcome in a large experimental study that lasts for decades. But in practice this is almost never feasible. That is why we often study intermediate outcomes with experimental research and clinical outcomes with observational research.
This is because the effect on intermediate outcomes occurs sooner, so you need fewer people and less time in studies to measure an effect.
You can improve blood pressure in a few weeks, for example, while preventing coronary heart disease takes years.
Demonstrating a strong causal relationship
If I have to sum it up briefly, demonstrating a strong causal relationship looks as follows:
X = Nutrition (exposure)
Y = Blood pressure (intermediate outcome)
Q = coronary heart disease
---- = plausible (biological) mechanism
O = Body of evidence
my own creation:
If we want to provide sufficient evidence for the relationship between nutrition and coronary heart disease, we want to:
- Provide evidence for the relationship between nutrition and blood pressure.
- Provide evidence for the relationship between blood pressure and coronary heart disease.
- Provide evidence for the relationship between nutrition and coronary heart disease.
Underlying these relationships there is always a plausible (biological) mechanism, which is often shown (beforehand) with, for example, in vitro studies (with cells, etc.) or with studies in animals. In other words, we form a hypothesis (theory) in which we describe how something possibly works (biologically), and then test it in humans.
When providing evidence for a causal relationship, we must always observe the rules of the pyramid of evidence. In the end, we use the total body of evidence as support for a claim.
If you want to read more about the pyramid of evidence and the body of evidence... click here.
Exceptions
Of course, there are exceptions. For example when we know for sure or strongly suspect that an exposure (at certain levels) can be very harmful. This makes it ethically unacceptable to do (experimental) research in humans.
You can find an example of this in my blog about chemical substances and metals in tap water. In such cases we more often choose research in animals (although you can raise ethical questions about that), cells or toxicological research. In the end, it will always have to be assessed per situation how much and how strong evidence is considered sufficient to demonstrate a causal relationship.
An example of this is tobacco. The fact that we have found a causal relationship between smoking and lung cancer is purely based on observational research. After all, it was and is unethical to carry out experimental research on smoking tobacco.
Examples of sufficient scientific evidence
I will take you through two examples of large organisations that work in their own way according to this approach. Let's look at how the GR and the WHO deal with providing sufficient evidence for a causal relationship.
The Gezondheidsraad
Source: Gezondheidsraad (2015). Werkwijze van de commissie Richtlijnen goede voeding 2015.
The GR bases dietary guidelines on research in which the exposure is measured first and the outcome is determined later. This means experimental research (only with a control group) and only prospective observational research (in which people are followed for years and people with the clinical outcome measure at the start are excluded from the study). This type of research has the strongest strength of evidence, because you know for sure that the exposure (such as high blood pressure) precedes the clinical outcome (such as coronary heart disease). This offers a stronger indication of a causal relationship.
The GR sees research in laboratories and in animals as hypothesis-generating research. In other words, it is used to generate a theory with a plausible (biological) mechanism that then has to be tested in humans.
In principle, the GR focuses on systematic reviews and meta-analyses of experimental research (RCTs) and observational research, so that the whole body of evidence is included in their advice.
The GR bases its guidelines (for a large part) on the relationship between nutrients, foods and dietary patterns, intermediate outcome measures such as blood pressure, LDL cholesterol and body weight, and the top 10 of diseases and deaths (related to nutrition) in the Netherlands.
The committee accepts an intermediate outcome measure when – besides indications from cohort research that it predicts disease risk – results from experimental research also convincingly show that a change in the intermediate outcome leads to a change in disease risk. According to the GR, only blood pressure, LDL cholesterol and body weight pass this test. In their advice, these intermediates are called causal risk factors.
For every advice you can find a background document in which the committee explains, for each dietary factor, the state of the science regarding effects (in the case of RCTs) and associations (in the case of cohort research). These advices always lead to a conclusion.
In the background document, the GR also discusses all the advantages and disadvantages of different types of research.
But they also give information about the quality aspects of measurement instruments for the exposure (dietary intake), how they approach the literature research and how they reach a conclusion. It is recommended reading for anyone who wants to learn more about nutrition science!
The GR uses four options for a conclusion:
- A high or low intake increases or decreases the risk of disease (based on RCTs), or a high or low intake is associated with a higher or lower risk of disease (based on cohort research). The strength of evidence is large or small. This is based on the availability of research, the presence or absence of heterogeneity, the strength of the association and any additional considerations that are specified in the explanation.
- An effect or association is unlikely. This applies if there is sufficient research that provides no indications of an effect or association.
-
The effect or association is not consistent. One or more of the following situations applies:
- In a meta-analysis, considerable and unexplained heterogeneity was found.
- There are considerable differences in the direction of effects or associations between individual intervention or cohort studies.
-
There is too little research to make a statement about the effect or association. This applies to evidence from only one original study, research in one study group, research of insufficient quality, or if too little research is available to make a statement.
Example: fruit and vegetables
Source: Gezondheidsraad (2015). Groente en fruit – achtergronddocument bij richtlijnen goede voeding 2015
The GR concludes that there is a causal relationship between fruit and vegetable intake and coronary heart disease. They do this on the basis of:
- A relationship between fruit and vegetables and an intermediate outcome (X -> Y).
- A relationship between the intermediate outcome and coronary heart disease (Y -> Q).
“In cohort research, blood pressure is associated with the risk of cardiovascular disease; the association is strongest for stroke. (sources: 25,26) RCTs show that a wide range of blood-pressure-lowering interventions (such as medication), both in patients and in people without cardiovascular disease, also lead to a reduction of the risk of cardiovascular disease. (sources: 27,28)"
- A relationship between fruit and vegetables and coronary heart disease (X -> Q).
Plausible (biological) mechanism:
Fruit and vegetables contain potassium (possibly helps directly with lowering blood pressure), fibre (possibly promote blood flow), antioxidants (protect blood vessels), nitrate (widen blood vessels) and are low in calories (helps with maintaining a healthy weight).
Conclusion: The intake of fruit and vegetables has a causal relationship with coronary heart disease.
The World Health Organization
The idea for this blog arose after the WHO stated that aspartame was placed in category 2B (possibly carcinogenic). The International Agency for Research on Cancer (IARC), an independent committee of the WHO, released a report (monograph) on aspartame. The Joint FAO/WHO Expert Committee on Food Additives (JECFA) also looked at the evidence.
The IARC chose to place aspartame in the category "possibly carcinogenic", which means that there is some evidence for a relationship. The JECFA, however, concluded that the current consumption of aspartame poses no danger to humans. They did not find the evidence strong enough and saw no clear plausible (biological) mechanism. So the two committees actually disagreed with each other.
A lot of discussion arose about the evidence cited by the IARC. I got the impression that many people on the internet ran off with this story without giving a correct picture of the actual evidence. For example, in the IARC publication you can find several (prospective) observational studies, and they do mention a plausible (biological) mechanism (something nobody mentions). The JECFA, however, did not find this evidence strong enough to actually change the safe limit for consumption. I too called the report insufficient evidence. After I brought this up, I got the important question: "But what is sufficient evidence, then?" Hence this blog.
I want to briefly explain what the IARC looks at as evidence for a causal relationship, and give processed red meat as an example. The WHO has placed this in the highest category: "Carcinogenic to humans" (Group 1). It is then very interesting to see how much evidence was needed for that.
The IARC's way of working
The aim of the IARC monographs is to investigate whether certain "complex mixtures, lifestyle factors and biological and physical agends" are possibly carcinogenic. And then, together with other international expert groups, to see whether more research is needed. A monograph is written when there is some evidence that people may be exposed to a carcinogenic "agend".
“A cancer ‘hazard’ is an agent that is capable of causing cancer under some circumstances, while a cancer ‘risk’ is an estimate of the carcinogenic effects expected from exposure to a cancer hazard.”
What is fascinating is that the categories speak of risks, but that the IARC clearly states in its document that it investigates hazards.
Hazards: A hazard (gevaar) is a potentially harmful substance.
Risks: Risk is the chance that a hazard actually occurs and causes harm, together with the severity of that harm.
They also state that these monographs do not give advice for guidelines or policy.
“These evaluations represent only one part of the body of information on which public health decisions may be based. Public health options vary from one situation to another and from country to country and relate to many factors, including different socioeconomic and national priorities. Therefore, no recommendation is given with regard to regulation or legislation, which are the responsibility of individual governments or other international organizations.”
The IARC works as follows:
They set up a working group of independent experts who bring together all the data on the subject. This includes:
-
Observational research
-
Studies in animals
-
Mechanistic research
-
Other relevant research
In addition to the working group of experts, there are a few more groups of participants who can be invited for the assessment of data:
-
Specialists on the subject
-
National and international health organisations
-
Scientists from the IARC
-
Observers with relevant scientific knowledge
All relevant data is then summarised and assessed for quality. Every monograph gets the following structure:
-
Data on exposure
-
Studies of cancer in humans
-
Studies of cancer in animals
-
Mechanistic and other relevant data
-
Summary
-
Evaluation and rationale
The AIRC describes extensively which data is or is not used per part. I will work you through it as quickly as possible. Brace yourself.
Whereas the GR focuses on studies in humans, and more specifically experimental and prospective observational research, the IARC uses all types of studies that are available. They do this because experimental research on cancer is almost never available, because of the long time cancer takes before it develops in humans. So the IARC bases a causal relationship mainly on the body of evidence, which mainly consists of observational research.
“The uncertainties that surround the interpretation of case reports, case series and correlation studies make them inadequate, except in rare instances, to form the sole basis for inferring a causal relationship. When taken together with case–control and cohort studies, however, these types of study may add materially to the judgement that a causal relationship exists.”
I understand that some people are sceptical about this. But let's not forget that we used this same way of working for tobacco. And I hope we are not going to have a discussion about whether smoking tobacco causes lung cancer.
The results from observational research are assessed for causality on the basis of the Bradfort Hill criteria such as:
-
Strength of the association
-
Do multiple studies find the same association
-
Dose-response relationship (the greater the exposure, the greater the effect)
Then the research in humans is assessed:
Sufficient evidence of carcinogenicity: (translated from Dutch)
The working group considers that a causal relationship has been established between exposure to the substance and human cancer. That is, a positive association has been observed between the exposure and cancer in studies in which chance, bias and confouding could be ruled out with reasonable confidence. A statement that there is sufficient evidence is followed by a separate sentence identifying the target organs or tissues where an increased risk of cancer was observed in humans. Identifying a specific target organ or tissue does not rule out that the substance may possibly cause cancer at other locations.
Limited evidence of carcinogenicity: (translated from Dutch)
A positive association has been observed between exposure to the "agends" and cancer for which a causal interpretation is considered credible by the working group, but chance, bias or confounding could not be ruled out with reasonable confidence.
Inadequate evidence of carcinogenicity: (translated from Dutch)
The available studies are of insufficient quality, consistency or statistical power to permit a conclusion regarding the presence or absence of a causal association between exposure and cancer, or no data on cancer in humans are available.
Evidence suggesting lack of carcinogenicity: (translated from Dutch)
There are several adequate studies covering the full range of levels of exposure that humans are known to encounter, and these studies are mutually consistent in not showing a positive association between exposure to the substance and any studied form of cancer at any observed level of exposure.
And the research in animals is assessed:
Sufficient evidence of carcinogenicity: (translated from Dutch)
The Working Group considers that a causal relationship has been established between the substance and an increased occurrence of malignant neoplasms or an appropriate combination of benign and malignant neoplasms in (a) two or more animal species or (b) two or more independent studies in one species carried out at different times or in different laboratories or under different protocols. An increased occurrence of tumours in both sexes of a single species in a well-conducted study, ideally conducted according to Good Laboratory Practices, can also provide sufficient evidence.
Limited evidence of carcinogenicity: (translated from Dutch)
The data suggest a carcinogenic effect but are limited for making a definitive evaluation because, for example, (a) the evidence of carcinogenicity is restricted to a single experiment; (b) there are unresolved questions regarding the adequacy of the design, conduct or interpretation of the studies; (c) the substance increases the occurrence only of benign neoplasms or lesions of uncertain neoplastic potential; or (d) the evidence of carcinogenicity is restricted to studies that demonstrate only promotion activity in a narrow range of tissues or organs.
Inadequate evidence of carcinogenicity: (translated from Dutch)
The studies cannot be interpreted as showing either the presence or absence of a carcinogenic effect because of major qualitative or quantitative limitations, or no data on cancer in experimental animals are available.
Evidence suggesting lack of carcinogenicity: (translated from Dutch)
Adequate studies involving at least two animal species are available which show that, within the limits of the tests used, the substance is not carcinogenic. A conclusion of evidence suggesting lack of carcinogenicity is inevitably limited to the species, tumour sites, age at exposure, and conditions and levels of exposure studied.
Finally, on the basis of the body of evidence (all the relevant research mentioned earlier and the assessment of it), the working group will come to a conclusion:
Group 1: The substance is carcinogenic to humans. (translated from Dutch)
This category is used when there is sufficient evidence of carcinogenicity in humans. Exceptionally, a substance may be placed in this category when the evidence of carcinogenicity in humans is less than sufficient, but there is sufficient evidence of carcinogenicity in experimental animals and strong evidence in exposed humans that the substance acts through a relevant carcinogenic mechanism.
Group 2 (translated from Dutch)
This category includes substances for which, on the one hand, the evidence of carcinogenicity in humans is almost sufficient, and on the other hand substances for which no human data are available but for which there is evidence of carcinogenicity in experimental animals. Substances are assigned to either Group 2A (probably carcinogenic to humans) or Group 2B (possibly carcinogenic to humans) on the basis of epidemiological and experimental evidence of carcinogenicity and mechanistic and other relevant data.
Group 2A: The substance is probably carcinogenic to humans. (translated from Dutch)
This category is used when there is limited evidence of carcinogenicity in humans and sufficient evidence of carcinogenicity in experimental animals. In some cases, a substance may be classified in this category when there is inadequate evidence of carcinogenicity in humans, but sufficient evidence of carcinogenicity in experimental animals and strong evidence that the carcinogenesis is mediated by a mechanism that also operates in humans.
Group 2B: The substance is possibly carcinogenic to humans. (translated from Dutch)
This category is used for substances for which there is limited evidence of carcinogenicity in humans and less than sufficient evidence of carcinogenicity in experimental animals. It may also be used when there is inadequate evidence of carcinogenicity in humans, but there is sufficient evidence of carcinogenicity in experimental animals.
Group 3: The substance cannot be classified as to its carcinogenicity to humans. (translated from Dutch)
This category is most commonly used for substances for which the evidence of carcinogenicity is inadequate in humans and inadequate or limited in experimental animals.
Group 4: The substance is probably not carcinogenic to humans. (translated from Dutch)
This category is used for substances for which there is evidence suggesting lack of carcinogenicity in both humans and experimental animals.
Processed red meat
Let's briefly look at an example from Group 1: The substance is carcinogenic to humans: processed red meat. That way we get a bit of an idea of how much evidence the IARC needs to demonstrate a causal relationship.
First I have to be clear about the definition of processed red meat.
“Red meat refers to fresh unprocessed mammalian muscle meat (e.g. beef, veal, pork, lamb, mutton, horse, or goat meat).”
“Processed meat refers to meat that has been transformed through salting, curing, fermentation, smoking, or other processes to enhance flavour or improve preservation.”
So this is not about a steak, but about meat that has been processed so that its shelf life and taste are improved.
Let's then look at the evidence that processed red meat is carcinogenic.
The working group evaluated more than 800 epidemiological studies that investigated the association between cancer and the consumption of red meat or processed meat in many countries, on different continents, with diverse ethnicities and diets. In the evaluation, the greatest weight was given to prospective observational research carried out in the general population. Case-control studies of good quality provided additional evidence. The focus was mainly on studies that considered red meat and processed meat separately, had obtained quantitative dietary data from validated questionnaires, had a large sample size, and controlled for the most important possible confouding factors for the cancer sites concerned.
- A relationship between processed red meat and an intermediate outcome (X -> Y).
An important intermediate outcome is the occurrence of adenomas (benign growths in the bowel). A meta-analysis published in 2013 reported a modest but statistically significant association between the consumption of red meat or processed meat and adenomas (preneoplastic lesions) of the colorectum that was consistent across studies.
A relationship has also been found between processed red meat, genotoxicity and oxidative stress. In humans, observational data showed slight but statistically significant associations with APC gene mutations or promoter methylation, which were identified in 75 (43%) and 41 (23%) of the 185 archived colorectal cancer samples, respectively. Consuming well-done red meat increases the bacterial mutagenicity of human urine. In three intervention studies in humans, changes in markers of oxidative stress (in urine, stool or blood) were associated with the consumption of red meat or processed meat. Red and processed meat increased the levels of lipid oxidation products in the stool of rodents.
- A relationship between the intermediate outcome and coronary heart disease (Y -> Q).
This was so much that I did not know how to summarise it. See the source if you are interested.
- A relationship between fruit and vegetables and coronary heart disease (X -> Q).
Positive associations between colorectal cancer and the consumption of processed meat were reported in 12 of the 18 cohort studies that provided relevant data, including studies in Europe, Japan and the USA. Supporting evidence came from six of the nine informative case-control studies. A meta-analysis of colorectal cancer in ten cohort studies reported a statistically significant dose-response relationship, with an increase of 18% (95% CI 1.10-1.28) per 50 g per day of processed meat.
There is insufficient evidence in experimental animals for the carcinogenicity of the consumption of red meat and processed meat. In rats treated with colon cancer initiators and promoted with a low-calcium diet that included either red meat or processed meat, an increase in the occurrence of colonic preneoplastic lesions was reported in three and four studies, respectively.
Plausible (biological) mechanism:
Processing red meat, such as curing and smoking, can lead to the formation of carcinogenic chemicals, including N-nitroso compounds (NOC) and polycyclic aromatic hydrocarbons (PAH). Cooking improves the digestibility and taste of meat, but can also produce known or suspected carcinogens, including heterocyclic aromatic amines (HAA) and PAH. High-temperature cooking by frying, grilling or barbecuing generally produces the highest amounts of these chemicals.
This would possibly happen through several ways such as DNA damage (genotoxicity), suppression of the immune system, promotion of angiogenesis and oxidative stress.
Conclusion:
“Based on the large amount of data and the consistent associations between eating processed meat and colorectal cancer in different populations, which make chance, bias and confounding (confouding) unlikely as explanations, the majority of the Working Group concluded that there is sufficient evidence in humans for the carcinogenicity of consuming processed meat.” (translated from Dutch)
According to the working group, this was not the case for unprocessed red meat. That is why it was not placed in the same category.
“Chance, bias and confounding could not be ruled out with the same confidence for the data on the consumption of red meat, since no clear association was observed in several high-quality studies and residual bias from other dietary and lifestyle risks is difficult to rule out. The working group concluded that there is limited evidence in humans for the carcinogenicity of consuming red meat.” (translated from Dutch)
Overall, the Working Group classified the consumption of processed meat as "carcinogenic to humans" (Group 1) on the basis of sufficient evidence for colorectal cancer.
In making this assessment, the working group took all relevant data into consideration, including the substantial epidemiological evidence showing a positive association between the consumption of red meat and colorectal cancer, as well as the strong mechanistic evidence.
Closing thoughts
So what is sufficient (scientific) evidence?
I have shown what I see as necessary evidence for establishing a causal relationship. I have also explained how large health organisations approach this. The common thread is that they look at the body of evidence, which is assessed for quality and shared with the world in a transparent way.
The common thread that I often see on the internet and social media, on the other hand, is that there is no transparency or quality assessment. People pick random studies from PubMed as support for their claims about causal relationships. This often happens with the suggestion that we have been lied to and that large health organisations are peddling nonsense.
I am 100% in favour of challenging (other people's) ideas; that is what ideas are for. As far as I am concerned, "the consensus" (if you want to put the large health organisations under that heading) can be attacked hard. But I am against cherry picking, misuse of science (such as citing studies without any testing of the content) and deception. Unfortunately, I see this come back all too often in alternative voices, such as influencers, news media, educational institutes and supplement companies. Sadly, that breaks all the trust I could have in such voices.
I will never claim that "the consensus" does everything right or delivers perfect work. But what I miss is substantive criticism of good quality and a well-founded counter-voice.
Of course, if you are satisfied with less evidence than these large organisations use and base your beliefs on that, that is perfectly fine. Everyone may believe what they want and do with their health what feels right. That is everyone's right.
This is okay: “Magnesium is important for more than 300 processes in the body, so I take magnesium for my health.”
But when someone makes claims about causal relationships and sells them as "scientifically supported", a burden of proof comes into play. And with the burden of proof comes responsibility. No burden of proof, no pleasure. The least someone must do then is be transparent about the support and how it was selected and assessed.
This is not okay: “Magnesium is important for more than 300 processes in the body, so taking magnesium will help you with health problems Z, X and Q.”
You now know which evidence is needed for such strong claims.
Adriaan ter Braack, Sjamadriaan, also wrote a good newsletter about this. Definitely recommended:
What did you think? Let me know in the form of a comment or an email: [email protected]
Have you come across a claim on the internet or social media and are you curious about an assessment of its support? Let me know and I'll dive in!
Do you want to support me and my nutrition science adventure? Share my articles or the podcast on your socials!
Everything you read on this website is my opinion based on knowledge and experience. There is a good chance that I sometimes overlook something, or that something could be better. I would love to hear it! We do science together.
Follow me or connect with me on:
Instagram: https://www.instagram.com/coenfirmation_bias/
Linkedin: https://www.linkedin.com/in/coen-dros/