Ranking News Agencies

This article was originally drafted in 2014, but has become much more relevant in our current political climate.

There are many ways one can rank a news agency's reputability, and simple to perform the statistical analysis. It is easy to make a ranking based on a statistical measure, and even easier to do it wrong.
Based on data from the Pew Research Center^[1] from October 2014, I will be walking through a proper statistical ranking of major news agencies.

The first and easiest way to rank them is by simply subtracting the "distrust" from the "trust" to obtain an overall "trust difference", a simple measure of how many more people trust the station than distrust it.

=B2-C2

The problem with this approach is that it's not weighted by the total number of reactions. For example, when comparing CNN to The Economist, our approach results in a reputability score of 35 for CNN and 30 for The Economist. However, CNN has a much higher distrust to trust ratio than The Economist but still has the higher score because it has far more votes overall (79 vs 42).

Let's fix this by dividing the original conceived score by the total number of votes, normalizing it to the number of votes received.

=B2/(SUM(B2:C2))

In most cases, this is close enough. But when comparing results with a different number of votes on each, one needs to be careful of unpopular results that are more prone to outliers. For example, if we add this very blog (blog.adelqalieh.com) to the list, with only 1 trust vote and no distrust votes, it would dominate the list at 100%. Hence, the final thing we need to account for is uncertainty for news agencies with fewer votes.

To account for this error, we will model the proportion of positive votes as a Bernoulli parameter, and estimate the lower bound of confidence interval with a 95% confidence level. Since we have a discrete number of votes, we will use the Wilson score. The result will be the minimum real proportion of positive ratings given the polled votes with a 95% chance. The Wilson score interval is a binomial proportion confidence interval, but using the Wilson score rather than a normal distribution. This is a much better estimator of a binomial proportion distribution because it does not assume a symmetric distribution and maintains the same bounds as a binomial (0 ≤ p ≤ 1). It also has many other nice properties like intervals for samples (news agencies) with only 1 Bernoulli parameter, which is not possible with a normal^[2].

=((G2 + I2*I2/(2*SUM(B2:C2)) - I2 * sqrt((G2*(1-G2)+I2*I2/(4*SUM(B2:C2)))/SUM(B2:C2)))/(1+I2*I2/SUM(B2:C2)))

Although relatively few news stations have dramatically changed in rating, a hypothetical placement of "Adel's Blog" now only receives a score of 21%, putting it just above the Glenn Beck Program near the bottom of the list :)

Based on examination of individual rankings and affected news agencies, this appears to be a fair evaluation that balances the proportion of trust with the confidence in that value.

Conclusions

Overall, the results are not too surprising. BuzzFeed barely qualifies as journalism, while more prestigious and respected stations like BBC and NPR take the top spots. Interestingly, the top two agencies are foreign, as The Economist and the BBC are based in the UK. The closing of Al Jazeera America in 2016 is not surprising in retrospect with the level of distrust found in this poll.

One thing to qualify as well is that reputability is orthogonal to the political spectrum, but the extremes are much less trustworthy than the centrist news stations that are less likely to be judged. The bottom half of the list is almost entirely occupied by news outlets that lean so far heavily to one side that it is almost completely dismissed by the other side. The proliferation of news on the extremes is both a cause and a symptom of the political divide in the US, and has no clear end in sight.

Caveats

The biggest caveat one needs to be aware of is the role of perception in this poll. This analysis is not at a judgement on the accuracy of the news itself: rather, a ranking of the news stations in the American psyche at large. This is especially crucial when considering the polarization of the media and, by extension, the American public.

The other caveat is that the poll was conducted in 2014, which was a completely different political climate than 2017, albeit a scary prelude. The logical follow-up is to repeat this analysis with a similar dataset for 2017, though I do not anticipate media polarization shrinking.