#ContactForm1 { display: none ! important; }

Wednesday, November 30, 2011

What Makes Scientific Reports Suspect

There was a link running around Twitter last week that claimed watching FOX News made people less informed than people who didn't watch any news at all. It took me a day to track down the link people were referencing, and just reading it made me want to beat someone over the head with my keyboard, except I like my keyboard more than that.

Ignore the content of the study, that isn't necessarily the bad part. What is bad is what this represents as a whole, from the content of the study, to how the public views the results. If science were a gourmet meal this would be a moldy hamburger with roaches pulled out from under the dumpster. Sure, it's edible, there might even be people who enjoy roaches with their dinner, but it shouldn't have this kind of public approval.

Let's break down why this is Bad Science so that next time you too can tell if something is fact or fiction:

*NOTE* A poorly conducted study that qualifies as Bad Science does not mean the conclusion is untrue, it just means the conclusion is suspect and further research is needed.

1. Source Material

In the case of the FOX News article the source material most people referenced was an article in a rival news source. Huffington Post is usually a good source for news and they cite where they found their story fodder, in a poll done by Fairleigh Dickinson University which didn't just look at FOX News viewers but cited problems with other news sources as well (and praised a few).

For something as trivial as who knows more about the politics in Egypt this isn't a huge deal. I wouldn't ignore this information just because of the news source. But consider more serious things, like the choice to vaccinate your child or not. The doctor who originally published the study was barred from practicing medicine after being discredited, and yet someone will risk their child's life over that study.

When you see something touted as science, be skeptical. Ask for the link and the source material so you can see the research yourself.

2. Buzz Words

Notice the sensational headline "FOX News Viewers Known Less Than People Who Don't Watch Any News: STUDY" even though the first paragraph correctly acknowledges that this was not a study, it was a poll. There is a difference.

The use of buzz words isn't new, but it is a little scary. Over in the genetics community people use the word "sexy gene" to describe a hot button buzz-word gene that catches both attention and grant money. Everything for "the depression gene" to "the warrior gene" are researched, although probably not as well as they should be. What garners the attention isn't good science, it's buzz words.

3. Sample Size
This is the biggest problem with scientific studies: not enough data.

In fact, Genomes Unzipped has an entire article devoted to why small sample size is Bad Science. In short, the danger of small sample sizes is that they very rarely have accurate results. A study done with 500 people might show a correlation that ceases to exist when you have a sample size of 5000.

The FOX News poll has a sample size of 612. That might be a good sample size if the topic of study was a rare disease that only 1000 people worldwide have. It is not a good sample size for the category People Who Watch Televised News.

This is really the biggest red flag for Bad Science. When you see someone claiming "90% of Men prefer X to Y!" it's time to ask them what the sample size was. Percentages mean nothing. I can go ask three children if they like to eat Brussel sprouts and get all three to agree that they love them. My research would then prove "100% of children love Brussels sprouts!" Would that be accurate? Of course not. My children are weird. That's the danger of a small sample size.

If you want to see something really scary, consider the original Autism/Vaccine study had a sample size of twelve. Feel free to recoil in horror.

4. Controlling for Other Factors
The Fairleigh Dickinson University poll does get points for trying to control for some factors, they asked which political party the victims/test subjects aligned with. It's a start, but not good enough. In a perfect world researchers are able to control for all possible variables. In the case of the FOX News poll the only controlled variable was Political Party. What they didn't cover was: age, income level, education level, primary source of news, other sources of news, work place environment, and social levels.

The poll focused on what people knew based on what televised newscast they watch, which makes the erroneous assumption that the television is the primary source of news for everyone being polled. Because the poll didn't address where the people were getting their news from, and how much of their news they picked up from conversations at work, online news, social networking, or other sources, it's hard to say how much of an impact the news shows really have.

Not a huge deal because all this will do is confirm some people's beliefs of conservative newscasts while it's ignored by everyone else, but this is a huge factor in other studies.

Take the vaccine example above. Autism is still being studied, there are still some people who debate whether the whole spectrum is one disorder or multiple similar disorders, and it's possible that there isn't one universal cause. Factors considered before the study was published: age, vaccination history, was the kid crazy... The tenuous link formed of the criminally small sample size was that all the children had their regularly scheduled vaccines, ergo the vaccines were to blame.

5. Cognitive Bias
As my friend said when I questioned the FOX News poll, "But I want it to be true!" And why not? A good percentage of the population already believes the headline, so why would they question the research? It simply proves what they've known all along, and a researcher is in more danger of falling victim to this than someone reading the results. This is called Confirmation Bias. People want to prove their prejudices correct and will seek out information that supports their beliefs.

Gosh! Could that possibly relate in any way to our FOX News example? Why, gentle reader, it certainly can. Viewers are most likely to watch a news station that tells them what they already believe, and the researchers (suffering from Experimenter's Bias perhaps) are most likely to interpret the data to reflect what they already believe. Just typing that makes me shiver in fear.

The human brain is not programed to question preset prejudices. When a scientist sees something they automatically agree with they must train themselves to remain skeptical. This is the most insidious part of Bad Science because we do it to ourselves. We see something that supports our way of thinking and don't read past the headline.

Bad Science kills people.
Don't be a victim. Don't victimize yourself by passively agreeing with everything that supports your prejudices. Ask questions. If you see a fact thrown out in a discussion, check the data. Don't be content with the fluff news article, skip the hype and go straight to the scientific data. And don't be afraid to call shenanigans.

Questions? Comments? The box is just below, feel free to share your thoughts. All I ask is that you keep it clean and polite, and don't turn it into a political discussion.


  1. When I was taking a class in college on statics my teacher told us the results can be twisted to show what you want it to. Meaning none of the Scientific Reports are without the hand of man in them. News is news and we all read news on the internet besides watching the news on TV. Oh well, have a great day.

  2. With the right language you can twist any report. You shouldn't, but you can. That's what makes science reports in headlines so dangerous, and enticing. The Cognitive Bias bit grabs you. You want to believe, and the science is there, so obviously *science has proven X!" and the world is perfect again.

    Unless you can read and understand the original study you're at the mercy of everyone else's agenda. And that's not a good place to be.

  3. This probably explains why there are so many of conspiracy fanatics. Not to get off topic, but you sure move a lot. I did for almost forty years and now I am taking root and spreading out as I sit and meditate on the meaning of it all. I do like twitter thanks... I already have a couple of followers.

  4. I'm well known for telling people not to trust reports and opinions unless they have also checked out the source for themselves.

    A friend was telling everyone how much he agreed with a post by someone whose opinion he trusted on an article that my friend hadn't checked out. I read the poster's blog then followed the link to the original article. The original article never stated what the poster said was the point of the article. SO I told my friend to go read the original article.

    He was very surprised by the difference and has learned to check out the original sources.

  5. Ric - I've moved all my life, my parents moved more, the gypsy part of my byline is both genetic (Slavic/Mongol gypsy ancestry) and my nature. Once I know everyone in a town it's time to leave. I get antsy if I stay in one place too long. :o)

    Ilnara - Even trusted sources put a spin on things, whether they mean to or not is out of the question.