Why do so many health studies contradict one another?

New data, inefficient studies, and if you like conspiracy theories, corporate/government meddling.

Corporate meddling because good money comes out.

Coffe is good but in small quantities [less than a cup per day]. Stimulates the I don’t know which systems from the body etc

From studies, people take the information they want to use. Others have already brought up the other point. When studies are done with corporate backers with an agenda, if the hypothesis they are looking for isn’t valid, they will scrap the entire thing.

Science isn’t a study. It isn’t two or three studies. Peer review and replication are part of the process. The internet has provided us with the ability to link to some abstract or a small review of some study. We don’t see the lifespan of this study, however. Was it attacked by peers because of inadequate sample size or methodology flaws? Were scientists able to duplicate this? We’re quoting scientific studies (I include myself here) without context, then complaining when the context wipes out our current understanding.

I thought about this for awhile after I saw your question.

I think the answer has to do with “nuance”. That is, the studies are of things that aren’t obvious. There’s no chance that a study of “health effects of getting hit in the head with an axe” are going to waffle on how good that might be for one. You’re also unlikely to see a study that says “getting 8 hours of sleep per night will be hazardous to your health in the long term” or “eating until you can’t eat any more day after day will affect your life in positive ways”. Some things are too obvious. But… could it be healthy to be “a little tired” or “a little hungry” every day? Would it be healthy to be “chased by a person with an axe” from time to time? That’s not so obvious.

So what has to be done is to make a hypothesis about “how much is a little” tired or hungry, figure a way to put metrics to that (because if it can’t be measured then it isn’t science anyway), come up with a testing methodology to determine some experimental results, measure the results with available technology, analyze the values determined through testing and finally draw a conclusion based on what the test / experiment was trying to prove in the hypothesis.

There’s a lot of room for error in each step of that entire process. Part of any good study (and published paper) is the peer review process, where other scientists and experts in the field of study review the experimenter’s entire written results. They’ll look for flaws and deviations from “known fact”, and failing to find obvious errors in logic or method (and with a basic trust in the truthfulness of the stated results) determine whether or not this was a “good study” or not. After publication, other researchers will attempt to replicate the experiments to determine the validity of the results.

Along the way, new measuring technologies and methods appear, new knowledge from completely different quarters colors the consideration of the results, new ideas and hypotheses emerge, new tests, etc.

And studies always continue at the margins of what is and what is not known. If you find any studies – no matter the results! – that attempt to determine whether or not getting hit in the head with an axe is good for you, then I’d sure be interested in that (as pure silliness).

Most health studies are simplistic and don’t take into account the complexity of human beings and how they interact with the environment. The experiments are often poorly thought out and the results inadequately analysed and so come to meaningless conclusions.

There is a voracious public appetite for ‘information’ on health and diet and so the media are always keen to report on any research however nonsensical it may be. The public consume the latest research findings on diet with the same eagerness as they buy the newest packaged meal from the supermarket.

Correlation does not imply causation. That’s a pretty big reason right there. Many times we see studies that ignore other factors, and therefore might erroneously lead someone to conclude a relationship that isn’t there.

That and corporate fraud, there’s plenty of that.

@incendiary_dan You took my answer. Although the second part I was going to just say that sometimes the people conducting the studies set them up to give the results they want, and then a PR department spins everything so it all sounds good.

@lostinparadise plus, human beings cannot be completely controlled, so medical studies are not perfect ever. Each person has their own set of genes, habits, environment they live in that can affect results of a study. Many studies are too small a sample or not double blind, but the results are still reported like it is fact or proven, but then a better constructed study comes along and has different results.

Not to mention sometimes there is no science at all behind a common belief. Sometimes we just think something is true because someone started the rumour and everyone seems to believe it. Then finally a study comes along to show it isn’t true at all.

The other line that goes with that is the Twain quote: “There are three types of lies: lies, damned lies, and statistics”

Because the human body is a dynamical system that is highly sensitive to initial conditions.

I have three answers to this. First, we sometimes make hypotheses based on ample evidence that simply turn out to be wrong. For instance, the evidence that hormone replacement therapy would be beneficial was overwhelming, and then when it was finally tested in a double blind, placebo controlled, randomized clinical trial, this was found not to be the case.

Second, when we do scientific research, it is impossible to discover the truth. We (usually) use parametric statistics to estimate the likelihood that, based on the samples we’re observing, an effect will be seen in the general population. When the likelihood is statistically very high, we can tentatively conclude that the effect is real.

But sometimes, even when the statistical likelihood of an effect being real is very high, the effect is still not real. It is impossible to know for 100% certain if an effect is real, or if it just looks like it is based on the samples you happen to be working with. So this is why we need to replicate findings before we can make meaningful conclusions.

Third, as technology gets better, we can make more accurate judgments about the things we investigate, and these might turn out to undermine things we believed previously.

I think there is a really big problem with the public perception of scientific research, and I’m not sure why that is the case (although I did once try to find out!)

Keep in mind that for every finding that is contradicted, there are dozens (hundreds? thousands?) more that are not. It seems unfair to damn all of science for doing precisely what it sets out to do: reject hypotheses that are ultimately not supported by the data.

I also want to try to address some of the misconceptions cited above:

1. “Corporate meddling.” It would be asinine for me to try to argue that no corporation has ever financed a research study, or that they have not attempted to influence the outcomes of these studies. However, this is extremely uncommon. A great deal of research is publicly funded, and privately funded research usually involves the granting agency handing you a pile of money and then asking you what you find. They do not control whether you publish your findings, or what the study says when you do publish it.

There have been a few high-profile cases that involved drug companies doing research on their own drugs and “massaging” the data. This kind of thing is very strongly frowned upon by the scientific community, to the extent that a very prominent scientist in my field is basically shunned now because she was involved with ghostwriting an article for a pharmaceutical company. She will not be fired from her university, but she has lost all scientific credibility.

2. “Most health studies are simplistic and don’t take into account the complexity of human beings and how they interact with the environment. The experiments are often poorly thought out and the results inadequately analysed and so come to meaningless conclusions.”

I see absolutely no justification for this statement. I read probably a dozen scientific articles per day and in my life have come across fewer than 10 that I would describe this way. Research studies are designed by experts in their field. Each study is designed and executed over a period of months to years by people who have been studying that issue (and only that issue) for years to decades. How and why they would waste their time creating a poorly designed study, failing to analyze it correctly, submitting it for peer review, and somehow passing peer review with their poorly designed, incorrectly analyzed study is completely incomprehensible to me.

3. “Correlation does not imply causation.” Actually, correlations do imply causation, they just don’t prove it. Cigarette smoking correlates with lung cancer, and cigarette smoking also causes lung cancer. Correlational studies were avoided for a long time in scientific research precisely because of this issue, but performed correctly they can be very informative. That said, correlations are not often used in scientific research. The most commonly used statistical tests involve an analysis of variance, which measures whether the variance between two groups is greater than the variance within each group.

I do not think statistics are a lie. I also do not think they are the truth. They are a tool, and they are the very best tool we have for determining causal relationships in behavioral and health sciences.

I agree with @nikipedia.

@nikipedia , There is something else, that you would know much more about than me. Scientists tend to be cautious in announcing their results. They take care not to overgeneralize. For example, climate scientists will talk about the harmful long term effects of CO2 emissions, but are hesitant to say that the extreme weather that we have been experiencing is directly related to CO2.

Unlike scientists, I think the press sometimes gets carried away with a scientific finding or perhaps is not sufficiently knowledgeable of the subtleties involved.

Because studying the human body, behavior, and health has so many, many variables that you can’t get rid of or control or often even know about.

Also, you may enjoy this comic.

With all due respect @nikipedia, you’re wrong on several counts. I think you’re a little too close to some of this to see it clearly. In terms of corporate meddling, there are numerous watchdog groups that have shown corporations routinely get involved on both official and less than official capacities. Further, the revolving door of corporate executives between the corporate and government jobs is well documented, particularly in departments that involve regulations and testing.

In addition, there are many cases in which bad research is held above overwhelming amounts of good research. A lot depends on what gets publicity, and the research community isn’t immune from public perception. Take a look at the still contentious debate over fats: vegetable oil manufacturers were at the core of demonizing certain fats and promoting theirs, even when the bulk of evidence says otherwise. But that bullshit is still taught to nurses and doctors in med school as fact. Then some of them go and do their own research based on faulty premises, further confounding the issue. See how one little corporate fraud can effect even legitimate studies?

Correlation does not imply causation. It suggests a relationship. That is all. That’s statistics 101.

@nikipedia I was referring to studies on diet and health rather than scientific research in general. I am very sceptical of these studies because, as the question suggests, they tend to contradict each other and the advice they give one year is overturned the next.

I was attempting to explain why this might be the case and I thought it might be because these studies, like the example quoted in the question, have a flawed methodology and because looking at factors in isolation, which is at the heart of the scientific method, might not always be appropriate when looking at very complex interactions such as that of man with his environment.

I don’t doubt that the people who carry out these studies are experts and persons of integrity but as their results are so often contradictory or mistaken there has got to be something wrong somewhere.

@incendiary_dan, if you’ve already made up your mind that corporations are controlling science, and you don’t want to listen to reason, I can’t fix that. I’m giving you information from ground zero, so to speak, as someone who actively performs this kind of research and is surrounded by it every day of my life, and if you don’t want to use this information in a useful way, that’s your choice.

I never stated that corporations are not involved in research; in fact I explicitly stated the contrary. What I did say is that this is fairly uncommon, and it is disclosed in the publication.

Finally, I disagree that there is a meaningful distinction between “implies” and “suggests” in this context. And for the record, I got in A in Statistics 101 as well as in Statistics 257, Advanced Biostatistics.

New information, better design. That’s the good reasons. Corporate backers making studies happen for profit’s sake, pharmaceutical companies doing same, different industries (food, dairy, etc.) pushing for more people to consume their products through ‘studies’. That’s the bad reasons.

@nikipedia You’re using straw man arguments; please don’t. I’m respecting your intelligence enough to read your points, please extend the same to me instead of debating points I didn’t make.

Oh yes, and I meant to clarify the Twain quote. It’s not an assertion that all statistics are lies, but rather that statistics are used to lie. Like lies of omission, which is often what they are; they mislead by sliding premises by people, rather than giving up front what was and wasn’t included. But that’s as often unintentional as it is intentional.

@incendiary_dan: Usually when an expert in a subject corrects me, I defer to their knowledge rather than insulting it.

@nikipedia Wow, the condescension. Seriously, grow up. You think because you’re a researcher you can use shit logic and disregard what I actually said and instead use your easy-to-knock down stereotypes? I gave you valid points that you disregarded and (likely purposely) misrepresented. If you’re not going to engage in serious discussion and take criticism honestly, don’t bother.

[mod says] Calm down, guys. No need to make this personal.

P.S. Telling someone they’ve used straw man arguments isn’t an insult. Not in my neck of the woods, anyway.

(1) “Imply” is a slippery word. It can mean “entail,” or it can mean “suggest.” Correlation does not entail causation; correlation does suggest causation.

(2) Calling clarification of one’s previous statements “shit logic” is an insult.

(3) Many health studies do not contradict one another. Instead, people misread the claims made by the studies as if they were stronger than intended. Six inconclusive studies are in agreement that the matter is not ripe for a conclusion, even if they all give different results.

@SavoirFaire No, actually that still isn’t. As rudely as that’s worded (my bad) it’s still not an attack on anyone or an insult. It’s a statement that poor logic was used, which isn’t an insult unless you’re a Vulcan. Seriously, I don’t want it sound like I’m griping, but on a site with purported intellectual focus, shouldn’t you mods have a grounding in formal logic? Just a suggestion. :)

And correlation suggest a relationship. Nothing more. That’s very important. I haven’t been involved in research in years, but that much sticks. Finding out what that relationship is is what empirical research is for.

Gotta love it when people just happen to know everything ~

@bob_ You just love to comment sarcastic crap anytime anyone happens to have a strong argument, huh?

Sorry for knowing how to engage in constructive discourse everyone. Thought scientists and those interested in science would appreciate that.~

@incendiary_dan Not at all. I also like to do that when anyone has a weak argument.

Seriously, how is asking that we use formal logic when engaging in discourse anything but appropriate, particularly in terms of science related questions? Because that is all I’ve asked for. I’m interested in @nikipedia‘s input on the subject because of her experience, and I don’t want her to just fall back on “I’m an expert so I know”, which is another formal logical error and, as I said, condescending. I have some legitimate concerns and points that weren’t addressed, but rather shooed away with straw men.

We can’t discuss things honestly if we’re not willing or able to do so. Formal logic is a tool for that. Dropping our egos and not getting insulted by minor criticism and disagreement is another.

@incendiary_dan Wow, I now see the error of my ways. I’m wrong, you’re right. Now, could you please post after me, so you get the last word?

Sure.

@bob_ Grow up already.

@incendiary_dan, let’s assume for now that you’re right and I’m wrong. I am saying this deliberately to try to defuse this so you don’t have to feel attacked and be on the defensive. I don’t think either of us is getting anywhere in the argument.

At my meditation class this Sunday, this exact issue came up. One of the girls in the class pointed out that when two people argue, what usually happens is that both sides walk away more convinced of their position, and angry at each other on top of that. This seems to be exactly what is happening here.

So I am wondering if there is a better way that we can communicate here so that we both feel heard, even if we don’t end up agreeing. What do you think?

@incendiary_dan I teach formal logic.

I sort of hate to say this, @SavoirFaire, but… so what? That’s not a real argument for anything, is it? Because if it is, then you should know as well as anyone that it’s a simple “appeal to authority” ... logical fallacy.

It’s not that I’m a big supporter of @incendiary_dan‘s arguments – I’m not – but this argument thread started at a low point, and it’s been going downhill.

@CWOTUS It’s not meant to be an argument. If you look at this response further up the thread, you will see that @incendiary_dan asked: “shouldn’t you mods have a grounding in formal logic?” My reply indicates that not only do I have that grounding, I have enough of it to teach others.

Got it. Thanks, and glad to hear that. Maybe I should have read more of the thread (or quit it entirely).

[mod says] Once again, there is NO need to make this personal, folks. Let’s lay off the sarcastic, insulting stuff. Please and thank you.

@nikipedia Thanks for responding. I’m not upset or angry at all (not at you, anyhow), merely pointing out that you didn’t actually respond to what I said, but something similar enough to make it appear that you did when you knock them down. I don’t think you did that intentionally, which is why I initially asked you to reread my arguments.

What can we do to continue constructive discourse? I think if you reread my argument and address what I actually wrote, that’d get it going again. That, and don’t appeal to authority, particularly since you’re not the only one here who’s done clinical research.

@SavoirFaire I’m glad you do. That means you understand that pointing out logical fallacies being used and asking someone to get back on track isn’t an insult.

@CWOTUS Yep, we don’t agree on much, but our disagreements are almost always logical! :P

@incendiary_dan I never said otherwise. I said that calling something “shit logic,” rather than actual pointing out the fallacies is an insult. When @nikipedia clarified her comments, she was in no way suggesting that you were endorsing the opposing views. She was clarifying her original comments so that you wouldn’t make the mistake of thinking she was saying something different than what she intended. Alas, it did not work. Thus you accuse her of building straw men, when she wasn’t really addressing you in the first place.

@incendiary_dan: Sounds to me like what you’re saying is that as long as I agree with you, we can talk. I’m gonna pass.

As a general rule, I ignore work until it can at least agree with itself. Still waiting for the psychologists to stop editing the DSM…

@Nullo If you’re waiting for science to stop changing, you don’t understand science. And if you value internal consistency that much, you better give up the Bible.

@SavoirFaire Oh, the Bible’s perfectly consistent, provided that you don’t go taking things out of context.
And thanks, but you can keep those words that you’re trying to stick into my mouth.

@Nullo I’m not sticking words in your mouth. First, I said “if.” Second, science changes because of inconsistencies and internal disagreements. As for the consistency of the Bible, it all depends on whether or not you’re willing to accept that 1 + 1 = 3 for large values of 1 and the level of literalism you want out of it.

@nikipedia Not at all, I’m asking you to address what I actually wrote. How is that asking you to agree with me? At this point there’s so little actual disagreement that it seems extremely silly we’re having this much trouble picking the conversation back up.

@SavoirFaire I asked @nikipedia not to use straw man arguments, because she did, and I find that a little insulting in addition to halting conversation. She then condescended to me. Do you not see that that was the point of contention? I think you’ve mixed up the timeline on this, because that context is a pretty important factor you seem to be overlooking. I referred to the use of straw man arguments as ‘shit logic’, because it is poor logic, and I wanted to continue the conversation, and frankly was a little annoyed that I was talked down to like that when I was trying to have a conversation (and because, as I said, she’s not the only one who has engaged in clinical research). Specifically, the phrase “if you’ve already made up your mind that corporations are controlling science” was not something I said or expressed. That’s suggesting that I was endorsing the opposing views, in your words. She then did not clarify. She condescended. Clarifying would involve reviewing what was said, and responding to how the conversation had been going before the point of contention.

I’ll start. As I said previously, I think the role of corporations is more prevelant than @nikipedia seems to believe, based on the reports of some watchdog groups. In particular, there is the documentable collaberation between several major corporations and government departments that oversee regulation and testing, as well as funding. Monsanto is chief among them, but logging companies like Weyerhauser often manage to get former execs into environmentally related positions. There’s also the drug companies, but they’re actually less sneaky than those aforementioned ones. While not controlling the entire field of research sciences, these often cloud good research, and because of the corporate interest, those studies that serve corporations’ agendas are often given more press.

The example I gave above of the controversy over fats is a good example of how even a little bit of tampering some time ago can skew results for some time after even among honest research, when people have a go at a subject with misinformed premises from those previous studies. That’s just something that happens, even when nobody’s meddling, because we are after all always operating on certain premises.

P.S. I also got A’s in statistics. I also have a 164 IQ and read at over 1000 words a minute with over 90% comprehension rate. What’s it matter? :P

@incendiary_dan The issue is that this post is a clarification of this post (which was not directed at you). Nor have you ever said what the putative straw men are. You’ve just said they exist (perhaps they are phantom straw men?) and that @nikipedia has used “shit logic” without even trying to point out where. That is disrespectful, and @augustlan had to ask you twice to stop. It’s time to put up or shut up: identify the straw men and prove that they were meant as representations of your position. Otherwise, you’re just making personal attacks and violating multiple moderator instructions.

@incendiary_dan Most IQ tests only go up to 160, so yours being 164 seems unlikely. And even if it was, it still doesn’t mean you’re always right.

@Aethelflaed I think that was his point. I’m pretty sure he was trying to demonstrate the irrelevance of claiming credentials.

@bobbinhood LOL