Why do people feel the need to pick apart research studies?

Never – EVER – take someone else’s word for something without questioning the motives behind it. Doing so is dangerous, and leads to things like thalidomide babies.

No one, epically when it comes to scientific data, will take the information at face value.

I think it is the nature of the beast. Since the process itself in producing a research study is comprised of blind tests and multiple testing in order to get as accurate a conclusion as possible.

I do it because it is second nature to me, though I imagine I have significantly more experience than the average person in dissecting methods and authors’ interpretation. I agree however that most people have no idea what they are doing when it comes to detailed analysis of an article’s strengths or weaknesses. All too often, people “go with their gut” or simply assume something is wrong because they “don’t believe it”. An ancillary question could be “what gives people the confidence to assess studies without significant training in analytical techniques?”

@shilolo: You also probably read the article before writing it off as garbage, which I have never (ever) seen anyone in one of these discussions bother to do.

@Seek_Kolinahr: That kind of reactionary logic is exactly what I’m so confused by. Certainly if this was something that had some chance of affecting your health or happiness on a personal level, I can understand the instinct for strong scrutiny. But I’m talking about things that are an interesting discussion piece, where the author’s interpretation of the data seems to be the most likely and most reasonable explanation (even if other potential explanations exist). Why can’t we just discuss the findings? Why is it so crucial that you first give it your personal stamp of approval?

I think in part a researcher/author can get so entrenched and immersed in the subject matter that during the writing process can miss some less than obvious detail an unattached observer may readily pick up on. Then of course there are those who live to nit pick at others mistakes.

@nikipedia

Because there is nothing stopping a company from not publishing findings that do not fit their agenda, and paying different people to perform the same study until they get someone who finds in their favor. It happens.

Of course I’d love to take every study subjectively (it would be much easier) but it just doesn’t make sense to not question a study funded by Phillip Morris that suggests nicotine use prevents Alzheimer’s disease. (random fictional example)

@Seek_Kolinahr: So (I can’t find a nice way to say this, but I don’t mean it so harshly) you assume bad faith on the part of the researchers until you can find some kind of definitive evidence otherwise?

No. I look at who funded the study, and work from there.

If the Alzheimer’s Association published a report stating that nicotine use prevents Alzheimer’s, I wouldn’t question it as strongly as I would if the same study were performed by a tobacco company.

The question is: What does the funding party have to gain by publishing this information?

There are two reasons, one good and the other not so good.

First the good. Peer review is what keeps science from getting snarled in false observations, flawed experiments and crazy theories. When a new research report is publiched in a peer reviewed journal, it is fitting that qualified scientists around the world challenge its findings, testing to see if the paper’s observations hold up in independent observations, if the experiments reported are repeatable, and if the experimental methods used have any flaws that might invalidate their conclusions. Only work that comes through this vigorous peer review process unscathed deserves to be taken seriously.

The not-so-good. Motives for challenging research are myriad. Research may conflict with cherished beliefs. White supremacists refuse to believe that their DNA is virtually identical to that of other races, or that their ancestors actually migrated from Africa in a diaspora from 70,000 to 40,000 years ago. Creationists reject carbon 14 dating because it proves that the earth is more than 6,000 years old.

Research may also challenge financial interests. Whalers insist that the whales they hunt are not an endangered species. They would deny it right up to the day that they hunt them into extinction. Global warming deniers may genuinely believe the disinformation put forth by the $27 trillion per year fossil fuel industry; but the driver behind the disinformation campaign is the cash. The same was true when science first showed that tobacco smoke was a carcinogen and nicotine was addictive. The tobacco industry used a massive disinformation campaign to fight the science. In fact, many of the same conservative think tanks and K Street PR firms that waged the war against tobacco science are using the tactics they honed in the tobacco fight today to try to defend continued and expanded use of fossil fuels.

As Upton Sinclair observed, “It is difficult to get a man to understand something when his salary depends on his not understanding it.”

@nikipedia This is not a bad practice. I actually took a class in college where we read through studies and determined if they had the data to back up their conclusions. Some of these studies were really terrible with their evidence and jumps to conclusions that I couldn’t help wonder how they got published.

The people who edit and run the science journal in question are supposed to go through these studies to determine if the evidence is there and if the data concludes as it should. But there are often… quirks in that step. As @Seek_Kolinahr has said, there are a number of scientists and publishers with money related motives, unfortunately. These editors also don’t know everything about the field and so, can make mistakes. That’s what makes “peer review” a good thing. (The editors call up scientists that they think would know about the subject to review the article). However, if a prestigious scientist is the author of the article, it will sometimes affect the reviewers judgment.

It is always wise to walk through articles and heavily analyze them if you’re going to take their conclusions for fact. Charts and data need to indicate a “significant difference” to make a point, and there’s a surprising number that don’t. And the experiments performed should be logically sound and lead up to the conclusion in a clear way. When we’re talking about science, we’re talking about replicatable facts. When it comes to fact, the proof should be there.

@ETpro GA

@Seek_Kolinahr: So if I understand correctly, you feel the need to pick apart the contents of research studies because you are concerned that the funding source may have manipulated the findings somehow? Are you okay with studies funded by the government (NIH, NSF, DoE, etc)?

@Iclamae: I certainly think people who have the background and ability should scrutinize research studies. But what I don’t understand is the instinct of people who clearly don’t have the ability to render a meaningful judgment to discount research studies wholesale, often without even reading them in the first place.

I don’t think people pick apart studies often enough. I have enough of a background in scientific study and research to know how many studies are fudged, are unchecked and what a pathetic process the peer-review one can be.

@nikipedia I agree with you on that. Just discounting a study without reading it doesn’t make sense. But to take the time to go through it and walk through all of the data and steps before you accept it as fact is healthy, imo. (Like with political news. If you have a journalist saying… the Gulf of Mexico is being ruined by the Jones Act not being repealed… you look up the information about the Jones Act and what it does. Then you decide if they made the right call. Because very very often they don’t and are simply making a statement to make it or because it gets them ratings.)

On another note, when I talk about research studies needing to be reviewed because of worries about over confidence or monetary aims, we are also talking about government funded studies. There are plenty of scientists in academia that abuse the system.

Something that particularly bothers me are cancer drugs. (I want to be a cancer researcher.) I walk through cancer drug studies very carefully. Many of these treatments don’t make a significant difference to their patients. They may increase your life span by 1 month, and you’ll pay some ludicrous amount of money for it and possibly suffer terrible side effects for that 1 month of life. In my opinion, that kind of drug is a waste of cancer patient time and money. It’s also a waste of the government’s funding. But there are a number of these drugs on the market because they make money (and to a lesser degree, because you never know what might cure 1 person’s cancer). And they make money because cancer patients put faith in the science and in the doctors recommending the drug.

I don’t want people to be super distrustful of scientists after all this. It’s just that they are human and often fall prey to bad judgment as well.

If a study is attempting to prove or disprove an opinion then it is usually flawed somewhere. The scientific method is to state an theory and then to conduct scientific experiments and scienticific observation against it. Then to come to a logical unemotional conclusion. There is much room for error because humans are contuctiong the experiments or entering the data. So to be truely scientific the theory must stand up under many experiments under different scientists to “hold water.” People love to edit and critique and if through all that the theory still holds then it is held as correct.
Studies are only that, just studies, and as so people love to find the errors. For the scientist, this is actually good because then the science becomes unemotional and more fact based. A scientic mind truely loves correction toward facts and an emotional mind loves to prove a point either fact or not. It is fun. It is interesting and drives discussion.

The fastest way to sound smart is to undermine the work of people smarter than you. I’m not making accusations, just an observation. If you’re not guilty of this, you know who you are.

“Picking apart” is, by another name, Peer Review. It is the“pickers apart” who are not the peers of the author who bother me, because essentially they don’t know what the fuck they are on about, nor do they substantiate anything they say, but there is a lot of that crap on Fluther. It used to make me so inflamed that I would misbehave in the threads; now I just try to shine it on: you can’t really teach a stupid person anything much.

What @nikipedia and @dpworkin said.

I think it’s a good thing. If the findings hold up to the nit picking, then you’ve got a valid research.

Part of the problem comes from the fact that there is a huge increase in non-scientific studies across the board, because every one has a theory, and they can pay people to do a study that will prove they are correct, no matter how flawed the so-called study is. This goes for the government agencies as well as anyone who produces a product they want to sell.

@YARNLADY -Well said :)

The peer review process in most disciplines is very rigorous. Amateur critiques may occasionally pick up some error that in fact was overlooked.

More often self-important pseudo-experts talk off the top heads (to be excessively polite) to try and impress others with their feigned superior knowledge. Others are far too easily taken in by such blow hards and their antics.

How do I know? I am an expert research methodologist with experience in a wide range of fields. I can differentiate genuine critics from those who just want others to listen to their voice and admire them. I did not become an expert by talking, but by reading, listening and discussing with those who are expert in their field.

Because they are studies. This is the search for the truth—everything must be questioned.

There is no such thing as a perfect research study so they all have strengths and weaknesses. Science cant advance if studies are not critiqued and their findings subjected to careful analysis.

Unfortunately, some organizations systematically distort the results of research studies. A good example is that of the Drug Abuse Resistance Education (DARE) program. Although not a single published scientific study has ever found the DARE program to be effective, the DARE organization continues to insist that it is a great success in spite of the unanimous evidence to the contrary. http://alcoholfacts.org/DARE.html