How much merit do you place in the concept of ideas/articles/claims being "Peer Reviewed"?

NONE, just because they showed it to their peers, means nothing.

Ditto ^^

I think peer review is a vitally important part of the scientific process. For people who are not familiar, peer review means that you have submitted a scientific article to a journal, and they have sent it to 1 or more experts on the topic who will anonymously review the scientific and intellectual merits of your article. If they see any major flaws, they will recommend that the journal reject your paper. If they consider your experiment scientifically sound, and they believe it will add something important to your field, they will recommend that the journal publish your paper. The final say rests with the editor of the journal, who decides to publish or reject based on all the reviews as well as their own expertise.

It is generally very difficult to get a paper past peer review. If there is a major flaw, you can rest assured the reviewers will pick up on it. If there is a minor flaw, you can still rest assured the reviewers will pick up on it. Often, they will require that you perform additional experiments before they will recommend publication. In my experience, nearly every scientific paper is either rejected outright or sent back for corrections before publication. It is extremely rare that a paper is just accepted. Based on absolutely nothing, I would estimate this happens <5% of the time.

So I think peer review adds a great deal of credibility to scientific articles. I would go so far as to say that if an article is not peer reviewed, I am 100% skeptical about it. If it is peer reviewed, I am much less skeptical of it. I wish I could rely on my own judgment to decide if an experiment was done correctly and the conclusions drawn are reasonable, but unfortunately, it is probably impossible to be an expert on every technique in every field and every analysis that can be done. I am inclined to believe expert peer reviewers when they say they were done correctly.

I do think there are flaws in the peer review process. Sometimes reviewers are unbelievable dicks. Sometimes their own biases lead them to reject papers, or to go a little softer on papers that maybe require more scrutiny. I think either the reviewers should not be anonymous, or the authors should be anonymous as well—unfortunately, there are petty rivalries among scientists that can influence reviewers.

I also think there are inherent problems in the scientific process that would be best solved by asking reviewers to act differently. Most studies that reach publication are statistically underpowered, fail to correct properly for multiple comparisons, and rely on arbitrary thresholds for determining significance. The way the system currently works unnecessarily increases both false positives and false negatives. But that is a whole other rant.

Summary—I put a lot of merit in peer review, but it is not always a perfect system.

I agree with @nikipedia ‘s great answer. What many people don’t seem to understand about scientific peer review is the degree of competition, whereby experts are encouraged to shoot down (or at least criticize) even good research, not to mention poor methods or faulty logic. If a published article passes peer review it says a great deal about the quality of the evidence. The more prestigious the journal, the tougher the peer review and the higher the standards.

The notion that organized science is some kind of big “club” that allows “members” to get by with winks and nods, if they merely have the right credentials, is absurd. Good researchers have to swim with sharks.

“Peer reviewed” means “You have given a whole community of people who know more than you a chance to call you on your bullshit.” (And “peer reviewed” doesn’t just refer to the hard sciences BTW!) So yes, I give such knowledge-claims a good deal of credence.

Until they are refuted or altered by later knowledge-claims, of course :)

Great answers and very informative. The thing I wonder about is “peer review” being used as a publishers tool to make things seem more legitimate. Who is going to call out a publisher/editor for typing that something was peer reviewed if it wasn’t? (of course it depends upon the subject matter) To me, the phrase is completely meaningless as I have no way of verifying that it was indeed peer reviewed.

Is there some kind of law against printing “peer reviewed” if it’s a lie?

@digitalimpression, I would think the authors submitting to the journal would call them out and stop submitting there.

@nikipedia I suppose an honest author would do that. (assuming they don’t already work for the magazine/publication)

I guess I just have doubts about placing extra confidence in an article/concept/announcement simply because of this little catch phrase. Life has taught me to be skeptical of quite a few things though. And no, there isn’t any one article I’ve read that I can point to.. This question came to my brain because of the novel Next by Michael Crichton.

It seems that there exists certain subject material where everyone on the print side (to include the author) of the thing wins with a “peer review” sticker attached. I’ve never researched the system so I’m not familiar with the checks and balances of it. Are there any? If not, I’m not very trusting of this sort of honor system.

@nikipedia I like the peer-review system, but it does seem like a good amount of articles with massive and obvious flaws get published anyway (like Priscilla Coleman’s 2008 article declaring that abortion leads to mental illness and almost everything on sex work and mental health). I dunno that I’d go so far as to say “If there is a major flaw, you can rest assured the reviewers will pick up on it. If there is a minor flaw, you can still rest assured the reviewers will pick up on it.”

I believe that peer review is essential to establish credibility… but “peer review” means that anybody with the appropriate skills can repeat the experiment. Of course, peers are human, so errors and corruption happen, but peer review lends credibility to even my skeptical mind.

That said, I like to know who those peers are.

I absolutely give ‘peer reviewed’ material more credence and I think @nikipedia‘s response explains why perfectly. It is no easy task to get your work published in peer reviewed journals.

@Aethelflaed, there are indeed some bad apples that make their way through. It can be lazy reviewers, or editors that think they know better than the reviewers (and are wrong), or out-and-out fabrications, as we have seen happen in more than one case recently.

Also, reviewers generally aren’t asked to look at the raw data and do the analyses over again, although that may be worth considering. Their job is largely to decide, assuming the authors are telling the truth, in terms of the methods and results, can their interpretation be believed? And ultimately, the only study that can ever really be believed is one that can be replicated.

@nikipedia Yeah, I know, but I was actually talking more about the methods and results. I see a lot of bad studies with really obvious flaws. Like, did you even take “how to conduct a study 101” obvious.

@Aethelflaed, I’m surprised. I don’t see that often (at least not in papers that make it through publication—I see it often when I am reviewing papers). And, often enough, there is a logical explanation. Experiments are subject to practical problems, and all too frequently you are forced to make some kind of tradeoff. Especially in the case of a very obvious problem—I think what usually happens is that the reviewer points it out, and the authors respond with an acceptable explanation that they just don’t put in the paper. Perhaps publishing reviews + responses would be useful.

Being half way into my PhD, I am finding out the joys of peer review, as well as doing some of it myself. I get the impression that some reviewers try to make constructive comments, while others just want changes made because they can.

I also don’t think that peer review necessarily means anything if research is only reviewed by like minded people. We would like to think that all reviewers are objective, but I’m not sure that is the case.

Regardless, it is essential until somebody comes up with a better system, which I doubt will happen any time soon.

To be honest, I find ethics committees more frustrating!