9

votes

Can we trust ANY population studies?

Answered on August 19, 2014
Created October 25, 2010 at 12:10 AM

Patrik (or someone) has posted a link to a recent Atlantic Monthly article, "Lies, Damn Lies, and Medical Science" at the top of PaleoHacks.

Briefly, the article says that almost all research into into the efficacy of drugs, vitamins, and other interventions is highly dubious or simply fraudulent. We in the Paleo community like to cite lots of studies that support our contentions about diet, vitamin D, omega 6, and so on -- but are these studies really any better than those that underpin the SAD? It feels righteous to pick holes in "their" studies, but ours are probably similarly flawed. An excerpt:

When a five-year study of 10,000 people finds that those who take more vitamin X are less likely to get cancer Y, you???d think you have pretty good reason to take more vitamin X, and physicians routinely pass these recommendations on to patients. But these studies often sharply conflict with one another. Studies have gone back and forth on the cancer-preventing powers of vitamins A, D, and E; on the heart-health benefits of eating fat and carbs; and even on the question of whether being overweight is more likely to extend or shorten your life. How should we choose among these dueling, high-profile nutritional findings? Ioannidis suggests a simple approach: ignore them all.

For starters, he explains, the odds are that in any large database of many nutritional and health factors, there will be a few apparent connections that are in fact merely flukes, not real health effects???it???s a bit like combing through long, random strings of letters and claiming there???s an important message in any words that happen to turn up. But even if a study managed to highlight a genuine health connection to some nutrient, you???re unlikely to benefit much from taking more of it, because we consume thousands of nutrients that act together as a sort of network, and changing intake of just one of them is bound to cause ripples throughout the network that are far too complex for these studies to detect, and that may be as likely to harm you as help you. Even if changing that one factor does bring on the claimed improvement, there???s still a good chance that it won???t do you much good in the long run, because these studies rarely go on long enough to track the decades-long course of disease and ultimately death. Instead, they track easily measurable health ???markers??? such as cholesterol levels, blood pressure, and blood-sugar levels, and meta-experts have shown that changes in these markers often don???t correlate as well with long-term health as we have been led to believe.

My question: If it's foolish to rely on these studies, where do we turn for evidence? Are personal anecdotes and testimony as good as it gets?

21fd060d0796fdb8a4a990441e08eae7

(24543)

on October 26, 2010
at 06:56 PM

I can't give you an example, because it's a confidential project at my workplace! There's no magic though, it's just that, empirically, certain nutrients have population studies that are better correlated with randomized trial findings.

D13278772f6612432bf53413fad4e7af

(801)

on October 26, 2010
at 11:21 AM

You say that these studies are (sometimes but rarely) good at predicting causation rather than mere correlation. Can you give me an example of this? The article implies that all of this research must be disregarged because even the researchers make the jump too quickly. How does one identity the "good stuff"?

21fd060d0796fdb8a4a990441e08eae7

(24543)

on October 25, 2010
at 05:36 PM

Epidemiological studies ARE NOT only good for generating hypotheses. They are used often to ascertain adverse events, for the many exposures that cannot be randomized (e.g. asbestos exposure), and for other sundry purposes.

  • D13278772f6612432bf53413fad4e7af

    asked by

    (801)
  • Views
    1K
  • Last Activity
    1432D AGO
Frontpage book

Get FREE instant access to our Paleo For Beginners Guide & 15 FREE Recipes!

6 Answers

best answer

8
62ed65f3596aa2f62fa1d58a0c09f8c3

(20807)

on October 25, 2010
at 01:19 AM

Epidemiological studes are ONLY good for generating hypothesies. But they don't actually 'show' anything at all. The the hypothesies must then be checked out via numerous well controlled experimental studies.

Only then should scientists even think about saying they have any evidence for anything. The problem is that many scientists are in a hurry to say they have answers, so they can get attention and grant money, or impress big pharma and get attention and grant money from them by saying what big pharma wants to here. You don't get money by saying "I don't know yet, give me a few more years.. Let's face it, the only way to get money is usually to say something that someone wants to hear. So they say what is necesary to get money, even if the evidence is not there yet. Plus, many of these people may be so confident they are right that they assume later research will back them up and they don't feel it's needed to actually wait around for the actual good evidence. THere are many examples of scientists saying they need to dispense advice early on, before the actual evidence is hashed out, because they feel that they will be saving lives by starting earlier. And if the controlled studies don't come out the way they expect/want, then they usually change the parameters around and try again, hoping to get better results next time. (believe me, I have seen plenty of this in action when I was at UCLA helping with clinical trials)

So yeah, epi studies are not reliable. You need controlled studies that are well done and enough of them to show a clear pattern. And you need someone with an honest view and an intelligent mind for that kind of think to do a proper review of each study. Then you can get an idea, but even then, sometimes some important variable is left out or not thought of, and the whole thing turns out wrong anyway.

I think that is why the mental hueristic of using paleo eating works well. There is much still to learn, but knowing that we developed eating paleolithic foods helps give us a better idea of what is probably true. We use our natural eating styles as a shortcut to allow us to guess what is probably best for us. THen we use science as well as personal experience to continue to refine it as best we can. But I think you need all three legs to make it a stable plan: science, history, and personal experience of n=1.

21fd060d0796fdb8a4a990441e08eae7

(24543)

on October 25, 2010
at 05:36 PM

Epidemiological studies ARE NOT only good for generating hypotheses. They are used often to ascertain adverse events, for the many exposures that cannot be randomized (e.g. asbestos exposure), and for other sundry purposes.

10
21fd060d0796fdb8a4a990441e08eae7

(24543)

on October 25, 2010
at 01:02 AM

Dr. Ioannidis used to work at my office way before I got there, and I can assure you that the article is way more inflammatory than the general view of nutrition research from these Greek super-geniuses.

There's a few things "wrong" with cohort/population studies, in the sense that they do not show causation like randomized trials. But that is not really something "wrong" with the study design. Nutrition studies are arguably the hardest to conduct because of the complexity of effect, multiple interactions, difficulty in adherence, etc.

Ambimorph was right in that they are great for generating hypotheses. However, in certain nutrients/foods, they have a high correlation with findings from randomized trials. The trick is knowing which areas they are good at predicting causation in. Very few people have read enough of the literature and understand enough of the biology to accurately represent cohort study research findings. But they are always interesting to talk about, if you don't get too carried away :)

One of the more interesting areas of epidemiological research is systematizing subgroup research--in other words, some studies may be able to generate interesting hypotheses about subgroups, but you should technically announce before you go in that you are interested in that, or else you might be accused of fishing for data. However, subgroup analyses often don't have the statistical power necessary to avoid finding false positive correlations.

D13278772f6612432bf53413fad4e7af

(801)

on October 26, 2010
at 11:21 AM

You say that these studies are (sometimes but rarely) good at predicting causation rather than mere correlation. Can you give me an example of this? The article implies that all of this research must be disregarged because even the researchers make the jump too quickly. How does one identity the "good stuff"?

21fd060d0796fdb8a4a990441e08eae7

(24543)

on October 26, 2010
at 06:56 PM

I can't give you an example, because it's a confidential project at my workplace! There's no magic though, it's just that, empirically, certain nutrients have population studies that are better correlated with randomized trial findings.

2
100fd85230060e754fc13394eee6d6f1

(18706)

on October 25, 2010
at 12:38 AM

You can trust them to generate hypotheses. They can't establish causation.

2
F1e5ff10797e0e35cda081a4221cb614

on October 25, 2010
at 12:35 AM

The longer the study is, the more likely that confounding factors will lead to misleading results. The larger the study is (in terms of participants), the more likely it is that weak associations will become statistically significant:

http://healthcorrelator.blogspot.com/2010/07/china-study-with-large-enough-sample.html

Because of that, short term studies may often be more reliable than long term ones. For example, you may have a 3-month study with 30 participants showing that egg consumption is associated with improvements in health markers, and then a 30-year study with 10,000 participants showing that egg consumption is associated with decreased longevity.

The conflicting result may be due to some odd effect, like egg consumption making a small percentage of individuals more energetic and thus more likely to engage in risky (e.g., participating in a risky sport) or self-destructive (e.g., drug abuse) behavior. The effect may be very weak (e.g., a 1% decrease in longevity), but the large sample size will make it statistically significant.

Having said that, we need a combination of various types of studies, short term, long term, large and small, controlled and uncontrolled, etc. And we need, more than anything, better-quality research - where researchers actually think about the limitations of the research and report them.

0
Ef9f83cb4e1826261a44c173f733789e

on October 25, 2010
at 12:55 AM

Are personal anecdotes and testimony as good as it gets?

Personal experience it what it boils down to for me. Someone's studied and theories may get my attention, but it's my own results that count for me.

0
A3a5ffb13761bbc7951392b0e7235219

(0)

on October 25, 2010
at 12:45 AM

I don't have an answer, but I love the question.

Answer Question


Get FREE instant access to our
Paleo For Beginners Guide & 15 FREE Recipes!