Bias Is In The Eye Of The Beholder, pt. 2

Not surprisingly, Tim Groseclose and Jeff Milyo's media-bias study took a thorough, mostly-deserved drubbing in the prickly provinces of Left Blogistan, while pundits in right field mostly devoured the results with little critical commentary on its methodology. The final instance of the former I'll be looking at comes from Media Matters, who make it their business to police conservative media spin and misrepresentation. After briefly stating their position on G&M's findings and mentioning the study's appearances in the MSM, they serve up a bit of ad hominem with a side of red herring:

None of the outlets that reported on the study mentioned that the authors have previously received funding from the three premier conservative think tanks in the United States: the American Enterprise Institute for Public Policy Research (AEI), The Heritage Foundation, and the Hoover Institution on War, Revolution and Peace. Groseclose was a Hoover Institution 2000-2001 national fellow; Milyo, according to his CV (pdf), received a $40,500 grant from AEI; and, according to The Philanthropy Roundtable, Groseclose and Milyo were named by Heritage as Salvatori fellows in 1997. In 1996, Groseclose and Milyo co-authored a piece for the right-wing magazine The American Spectator, titled "Lost Shepherd," criticizing the then-recently defeated member of Congress Karen Shepherd (D-UT) and defending her successor, Enid Greene (R-UT); when the piece was published, Greene was in the midst of a campaign contribution scandal and later agreed to pay a civil penalty after the Federal Election Commission found (pdf) that she violated campaign finance laws.

The fact that G and/or M may have formerly been on the take from conservative think tanks may tell us a great deal about their motivations and expectations, but it has no bearing on the validity of their current research. Partial parties can still conduct worthwhile scholarship as long as they maintain their intellectual honesty. Moving on:

Any quantitative study of this sort must take a complex idea -- in this case, "bias" -- and operationalize it into something that can be measured. But given its rather odd operationalization of "bias," it is perhaps unsurprising that the study's scheme leads to some categorizations no observer -- on the right or the left -- could take seriously, including the following:

• National Rifle Association of America (NRA) scored a 45.9, making it "conservative" -- but just barely.
• RAND Corporation, a nonprofit research organization (motto: "OBJECTIVE ANALYSIS. EFFECTIVE SOLUTIONS.") with strong ties to the Defense Department, scored a 60.4, making it a "liberal" group.
• Council on Foreign Relations, whose tagline is "A Nonpartisan Resource for Information and Analysis" (its current president is a former Bush administration official; its board includes prominent Democrats and Republicans from the foreign policy establishment) scored a 60.2, making it a "liberal" group.
• American Civil Liberties Union (ACLU), bête noire of the right, scored a 49.8, putting it just on the "conservative" side of the ledger.
• Center for Responsive Politics, a group whose primary purpose is the maintenance of databases on political contributions, scored a 66.9, making it highly "liberal."
• Center for Strategic and Budgetary Assessments, a defense policy think tank whose board of directors is currently chaired by former Representative Dave McCurdy (D-OK), scored a 33.9, making it more "conservative" than AEI and than the National Taxpayers Union.

We leave to the reader the judgment on whether anyone could take seriously a coding scheme in which RAND is considered substantially more "liberal" than the ACLU.

This interpretation is incorrect, because the numbers cited are not ADA ratings for the think tanks but rather average scores of legislators who cite each group. Thus, G&M avoid the rather elementary error of officially categorizing the ACLU as right-of-center or RAND as solidly liberal. However, what the list above does show is that citation information from the congressional record does not always coincide with the "conventional wisdom" about a policy group's ideological slant. (Indeed, it supports the contention that operationalizing bias in the news media may very well be impossible.) If this is true for think tanks, it probably also holds for news organizations as well.

(As a brief side note, I was surprised at how little space the authors devoted to defending the logic behind using think tank/advocacy group citations as a measure of media bias, especially considering some of the non-intuitive think tank ratings it generated. They seem to treat their method's relevance as self-evident, an attitude I find unwarranted given its novelty.)

But this is not the only problem with Groseclose and Milyo's study; they lump together advocacy groups and think tanks that perform dramatically different functions. For instance, according to their data, the National Association for the Advancement of Colored People (NAACP) is the third most-quoted group on the list. But stories about race relations that include a quote from an NAACP representative are unlikely to be "balanced" with quotes from another group on their list. Their quotes will often be balanced by quotes from an individual, depending on the nature of the story; however, because there are no pro-racism groups of any legitimacy (or on Groseclose and Milyo's list), such stories will be coded as having a "liberal bias." On the other hand, a quote from an NRA spokesperson can and often will be balanced with one from another organization on Groseclose and Milyo's list, Handgun Control, Inc. (Nonetheless, this reference is somewhat confusing, since Handgun Control was renamed the Brady Campaign to Prevent Gun Violence on June 14, 2001, and there is no reference to the Brady Campaign in the study or clarification of the name change; therefore, it is impossible to determine from reading the study if Groseclose and Milyo's score reflects post-2001 citations by legislators and the media of the group under its new name.)

This point is related to the WSJ's complaint about comprehensiveness that I cited in my previous post, but I would like to make one comment. I'm not sure if MM is correct about the NAACP above (one could imagine a story in which an NAACP representative alleges systemic anti-black profiling behaviors among police counterbalanced by a conservative think tank's research suggesting that the problem is isolated), but let's assume for the moment that they are; the same argument could also be applied to certain conservative groups. Journalists often cite Citizens Against Government Waste (average legislator score: 36.3) just for the pork-barrel statistics it maintains. Since there are no pro-pork think tanks, it's conceivable that the group's name could be found in an article without a liberal counterweight. Conceivable, but not particularly likely (or unlikely) in the absence of corroborating evidence.

A little further down, we encounter a variant of another argument the WSJ used:

Groseclose and Milyo's discussion of the idea of bias assumes that if a reporter quotes a source, then the opinion expressed by that source is an accurate measure of the reporter's beliefs -- an assumption that most, if not all, reporters across the ideological spectrum would find utterly ridiculous. A Pentagon reporter must often quote Defense Secretary Donald H. Rumsfeld; however, the reporter's inclusion of a Rumsfeld quotation does not indicate that Rumsfeld's opinion mirrors the personal opinion of the reporter.

I examined this view in my previous post but somehow neglected to mention its most glaring flaw. The reason that G&M restricted their purview to think tanks and not all sources is because in most cases reporters get to choose the think tanks they cite in articles. They cannot refrain from mentioning the views of Donald Rumsfeld or Al-Qaeda if the story demands it, but they can pick which think tank(s) will provide the analysis. This concept is so obvious that I have a hard time interpreting its omission as anything other than willful obtuseness.

Although the authors seem completely unaware of it, in reality there have been dozens of rigorous quantitative studies on media bias and hundreds of studies that address the issue in some way. One place the authors might have looked had they chosen to conduct an actual literature review would have been a 2000 meta-analysis published in the Journal of Communication (the flagship journal of the International Communication Association, the premier association of media scholars).

No argument here; G&M's lit review was rather skimpy, but that may be partly because they come from a political-science tradition and were not familiar with the relevant communication studies (not that that's an excuse).

Finally, of particular note is the way the study's authors toss about the word "bias" indiscriminately. We at Media Matters for America are particularly careful to make no accusations of bias, since saying a journalist or news outlet has a "bias" assumes that the one making the charge knows what lies within another's heart or mind. For this reason, most claims that the media are "biased" are problematic at best.

Dictionary.com's definition 2a for the word "bias" reads as follows: "A preference or an inclination, especially one that inhibits impartial judgment." Bias doesn't have to be conscious or intentional; merely pervasive (and demonstrably so). But MM do raise an important point; namely, that bias is an extremely difficult concept to quantify, and scholarly attempts to do so require thorough review and justification. The fact that the media research canon has failed to offer any consistent answers on the issue of bias invites the observation that we are perhaps asking the wrong question. It may be more fruitful to ask the people on both sides who accuse the media of bias whether objectivity in the 21st century is even possible. We should remember above all that although certain reporters may not always be objective, it does not follow that the organizations they work for are biased. As always, keeping a critical eye on the news and constantly seeking independent corroboration are the best safeguards against whatever hidden biases may be present in any given news source.