@Qinqu, I thought you might find this interesting for your research.
“During the last 13 months, the Republicans that have led the way with the largest number of Barely True, False, and Pants On Fire grades are Sarah Palin with eight, Michele Bachmann with seven, and John Boehner, Mike Pence, and the National Republican Congressional Committee with four each.
“Whereas Boehner received six “True,” two “Mostly True,” and one “Half True” ratings during this span, Pence and the NRCC received none in these categories, Bachmann only two, and Palin just four.
“What is particularly interesting about these findings is that the political party in control of the Presidency, the US Senate, and the US House during almost the entirety of the period under analysis was the Democrats, not the Republicans.
“And yet, PolitiFact chose to highlight untrue statements made by those in the party out of power.
“But this potential selection bias – if there is one at PolitiFact – seems to be aimed more at Republican officeholders than conservatives per se.
“An examination of the more than 80 statements PolitiFact graded over the past 13 months by ideological groups and individuals who have not held elective office, conservatives only received slightly harsher ratings than liberals.
“Half of the statements made by conservatives received ratings of Pants on Fire (12.5 percent), False (16.1 percent), or Barely True (21.4 percent), compared to 41 percent for liberals.
“These findings beg the central unanswered question, and that is what is the process by which PolitiFact selects the statements that it ultimately grades?
“When PolitiFact Editor Bill Adair was on C-SPAN’s Washington Journal in August of 2009, he explained how statements are picked: ‘We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it.’
“If that is the methodology, then why is it that PolitiFact takes Republicans to the woodshed much more frequently than Democrats?
“One could theoretically argue that one political party has made a disproportionately higher number of false claims than the other, and that this is subsequently reflected in the distribution of ratings on the PolitiFact site.
“However, there is no evidence offered by PolitiFact that this is their calculus in decision-making.
“Nor does PolitiFact claim on its site to present a ‘fair and balanced’ selection of statements, or that the statements rated are representative of the general truthfulness of the nation’s political parties or the elected officials involved.
“And yet…
“In defending PolitiFact’s ‘statements by ruling’ summaries – tables that combine all ratings given by PolitiFact to an individual or group – Adair explained: ‘We are really creating a tremendous database of independent journalism that’s assessing these things, and it’s valuable for people to see how often is President Obama right and how often was Senator McCain right. I think of it as like the back of a baseball card. You know – that it’s sort of someone’s career statistics. You know – it’s sort of what’s their batting average.’ (C-SPAN Washington Journal, August 4, 2009)
“Adair is also on record for lamenting the media’s kneejerk inclination to treat both sides of an issue equally, particularly when one side has the facts wrong.
“In an interview with the New York Times in April 2010, Adair said: ‘The media in general has shied away from fact checking to a large extent because of fears that we’d be called biased, and also because I think it’s hard journalism. It’s a lot easier to give the on-the-one-hand, on-the-other-hand kind of journalism and leave it to readers to sort it out. But that isn’t good enough these days. The information age has made things so chaotic, I think it’s our obligation in the mainstream media to help people sort out what’s true and what’s not.’
“The question is not whether PolitiFact will ultimately convert skeptics on the right that they do not have ulterior motives in the selection of what statements are rated, but whether the organization can give a convincing argument that either a) Republicans in fact do lie much more than Democrats, or b) if they do not, that it is immaterial that PolitiFact covers political discourse with a frame that suggests this is the case.
“In his August 2009 C-SPAN interview, Adair explained how the Pants on Fire rating was the site’s most popular feature, and the rationale for its inclusion on the Truth-O-Meter scale: ‘We don’t take this stuff too seriously. It’s politics, but it’s a sport too.’
“By levying 23 Pants on Fire ratings to Republicans over the past year compared to just 4 to Democrats, it appears the sport of choice is game hunting – and the game is elephants.”
http://blog.lib.umn.edu/cspg/smartpolitics/2011/02/selection_bias_politifact_rate.php