Tag Archives: how-to

How to Fact-Check Science

Part 1: The Scientific Method and Experiment Design

“Science” refers to a method for discerning truth, to a body of knowledge collected by generations of scientists, and in many ways, to a culture. The scientific method is the process by which scientists determine which facts are true.

The general steps of the scientific method are as follows:

Propose a question.
Do background reading about relevant known information.
Form an hypothesis (an informed guess as to the answer of the question).
Create an experiment to test the hypothesis.
Run the experiment.
Collect data.
Analyze the data.
Form a conclusion about the hypothesis, or go back to step 4 and repeat the process as needed to have enough information to form a conclusion.
Communicate the conclusion.

If you are new to the scientific method, check out this site for more details in an interactive format. It’s designed for kids doing science fair experiments, which makes it easier to read regardless of your age. In fact, science materials designed for kids are great for all ages for just that reason. If you want something more detailed, the kids version will give you enough context to be able to find and understand it.

Each one of the steps in the scientific method has its own standard set of rules intend to guide the scientist toward truth while circumnavigating the scientist’s own biases. Experiment design is a big part of this.

Controls are necessary. Scientists use an experimental group and a control group when doing experiments in order to have a basis for comparison.

For example, when I was in sixth grade, I tested my fifth grade teacher’s running program to see if it impacted lung capacity. I measured the lung capacity of her entire class of students periodically throughout the term. This showed a growth in lung capacity, but that alone was not enough information to determine whether there was a correlation between the running program and the changes in lung capacity. Other variables abounded, such as the changing seasons, the natural growth of children over time, and so on. In order to determine whether there was a correlation, I needed to use a control group which had all the same variables except the running program. I used the next door fifth grade class in the same building, which did not have a running program. This group did not show any growth in lung capacity, making the data I collected far more useful.

Note: If the class I used for my control group had been kids of a different grade, or at a different school, or if I had collected the information during a different part of the year, then it would not have been a good control group because here would have been more than just one variable that was different between the experimental group and the control group.

When evaluating scientific information, claims, and journalism, keep an eye out for problems with the scientific method or with control groups. If you can’t determine whether these things were done properly, it’s best to disregard the information as unverifiable. It may be true, or it may not be. Not knowing is part of science. Get comfortable with not knowing.

Part 2: The Reality of Science

The reality of science is a bit imperfect compared to what popular media has to offer. Have you ever watched a science fiction show where a character uses a handheld device or other instrument to “scan” something, and instantly finds out a whole pile of information about the object in question? That is pure fiction. The real world doesn’t have that kind of technology yet, and likely won’t in our generation’s lifetime. If a scientist needs to test a water sample for contaminants, for example, that scientist will need to run a separate test for each potential contaminant, and each of those tests takes time. The length of time ranges from minutes to days depending on the procedure. For bigger projects, such as studying a new phenomenon, science takes months or years, sometimes even decades, to produce reliable conclusions.

The reality of the timing involved in science is one of the key concepts you can use when fact checking science articles. Claims of detailed knowledge of brand new phenomena are probably not based in evidence, whether or not they end up turning out to be correct guesses. Think back to when Covid-19 first started. Remember all the firm claims that ended up being wrong? Be wary of science reporting about new phenomena, especially if the reporting doesn’t take into account the concept that “we don’t really know for sure yet” in the language.

In addition to limitations, science is subject to bias. This is true both of scientists when conducting science, and of reporters when engaging in science reporting. Scientists are products of our own cultures, which means our biases influence the way we think, and therefore the way we design experiments and interpret the results. This is especially prominent in anthropology (the study of humans), but all other scientific disciplines struggle with this as well.

In order to help prevent biases and other issues from harming the validity of the scientific body of knowledge, scientists participate in something known as “peer review.” When science journals publish scientific research, the process involves having other scientists who did not work on the project review the documents, procedures, and conclusions for proper scientific method and accuracy. This process is very rigorous. When fact checking science, look for peer reviewed articles.

And finally, be aware of the limits of scientific observation. There is a lot we don’t know about, due in part to simply not having the technology to allow us to observe it. Not knowing is part of science. Get comfortable with not knowing, with being willing to hold space for an unknown rather than trying to fill it in without having enough evidence to do so.

Part 3: Understanding The Numbers

Numbers are a big part of science. We use them to describe our observations so that other people can understand what we witness. We also use them to do calculations to figure out more information about our observations.

As you might already know, math can get very complicated. However, you don’t have to go learn calculus or statistics in order to be a discerning individual when it comes to evaluating scientific claims. That said, the most important math to understand for this purpose is probably statistics. Statistics are often twisted, misrepresented, and simply misunderstood in science journalism and by politicians.

Please pause after this paragraph to watch the 12-minute video embedded below to start to get a basic idea of how statistics works. If there are vocabulary words you don’t understand, then pause the video and look them up before continuing. This video contains example problems to do on your own. You can use them to evaluate your understanding of the concepts. If you can do them, then you have a grasp of this knowledge and can use it when evaluating scientific claims. If you can’t, or if you choose to skip them, that’s okay too – it means that you know you don’t have a strong enough grasp of these concepts to evaluate related claims. If this is you, then it is important to remember to put any statistical information you see in the “I do not know if this is true because I am not able to evaluate it” box in your head rather than immediately believing or disbelieving it. Again, get comfortable with not knowing.

When you are reading about science, activate your critical thinking whenever you see numbers. Are units provided? If something went 13 kilometres, that’s a lot farther than 13 feet. Are numbers presented in a way that makes sense? For example, looking at the total number of deaths due to Covid-19 between countries is not as useful as looking at the total number of deaths per capita. “Per capita” means “per person.” This is how to adjust for population size. Think of it this way: If 1,000 total people die in Rhode Island, that’s a lot different than if 1,000 total people die in California because California has so many people in it. If 1,000 people die per every 5,000 people in Rhode Island, then it is the same rate of death as if 1,000 people die per 5,000 people in California. This is why “per capita” numbers are often more useful than total numbers.

Part 4: Reading Scientific Literature

If you have never read a scientific study before, it can look daunting. Studies are filled with scientific jargon making them difficult to read, even for scientists. The key is to read them more than once, look up words you don’t know, and focus on specific parts of the study.

The abstract is a good place to start. This section summarizes the process and results of the study. Sometimes the abstract has all the information you need to fact check the article or meme which was supposedly based on the study. The other parts of the study are valuable if you wish to gain a detailed understanding of how the study was run, especially if you wish to evaluate whether it was done properly.

Part 5: Evaluating Articles

As mentioned above, research journalism is rife with bias and outright error. Be skeptical of headlines designed to evoke an emotional response. Here are some practical questions to ask yourself when evaluating articles that make scientific claims:

When was this article written? If it was written a long time ago, have there been new breakthroughs since then?
Who wrote it, and why? How might that bias the writing?
Does the article site its sources, including links to any scientific studies the article claims as sources? If not, disregard the entire article as it is not credible.
Do the abstracts of the original studies actually support the claims in the article?
If the article uses numbers, does this article use them in a way that makes sense?

Part 6: Sharing Information

If you struggle with holding space in your mind for the unknown, it may be difficult to read false information without absorbing some of it into your belief system. There is a big difference between believing something yourself and asking others to believe it. To ensure that what you share with others is true, it is a good idea to create a system for determining what you will share. Consider these questions:

Have I actually fact checked this, or am I only sharing it because it fits with what I already believe?
Am I sharing this because it is true, or because I would be allowing my anger, hope, or another emotion press the share button for me?

Sometimes you won’t be able to fact check something. Maybe it relies on statistics you don’t understand, or maybe there is a paywall between you and the original study. If you can’t fact check it, then what? What do you do with that informational meme or science article which makes a really good point, but which you are struggling to fact check? In my opinion, you scroll past it, or you find an expert to ask about it. Don’t share what you can’t fact check.

This post topic was selected by the author’s Patreon patrons.

Intro to Personal Anti-Racism in America

Leave a reply

I want to preface this by saying that I am a white American, and the target audience for this particular piece is any white adult in the USA who is against racism but may not have really dived into racial dynamics and anti-racism very much yet. If you’re a kid, you can still read this, but some things may not quite apply to you, and you may not have some of the information this piece assumes the reader has, which might make it difficult to understand in some places.

With that context established, I would like to invite you to take a moment to prepare yourself for this content. Remember that America was in a very real and literal sense built on the backs of slaves on land stolen through genocide of several different groups, many of whom are still here and fighting for the return of their ancestral lands, because unlike Africa, this continent was never decolonized. Remember that as history progressed, the powers that kept people of color down persisted, changed forms, and were enacted on every nonwhite racial group to arrive here.

These powers and influences are still all around us. We are steeping in them. Remind yourself that you are merely human, which means these influences have impacted the way you think in ways beyond your awareness. Remind yourself that it is up to us, today’s white people in America, to do whatever we can to correct the way we think about and handle race, because no one is going to do it for us. We can’t usually do that while feeling comfortable. That means this piece will likely be uncomfortable to read at times, and that’s okay, because that is part of how progress towards racial equity happens. This kind of discomfort doesn’t mean you’re a bad person. It means you’re an improving person, which is truly the best you can do. Ready? Here we go.

The first prerequisite for doing anti-racist work on a personal scale is this: We must acknowledge the extreme levels of racism swirling all around us in the USA.

Our culture, society, and government are racist in function and value white people above people of any other race. We can see this in the numbers, with black people being disproportionately turned away for jobs and promotions, indigenous women being murdered at an astounding rate, black men and boys being disproportionately murdered by police, and so on. We can see it in the actions of our government during the lifetimes of today’s population, such as the forced internment of Japanese-Americans during WWII, and of course the Trump administration’s despicable handling of immigrant and refugee families at the southern border, enacting atrocities which aren’t being applied to white immigrant and refugee families arriving from predominantly white nations.

We can also see the pervasiveness of racism by opening our eyes and following Americans of color on blogs and social media, and by opening our hearts to the truth of what people of color are saying about their daily experiences with racism of various scales both to our faces and all over the internet every day. Too many of us brush these things away as exceptions, as not true, or as not significant. Until we accept the fact that people of color experience racism daily on scales that range from racist comments, to systemic threats to one’s livelihood and home, to various forms of racially charged murder and other bodily harm, we cannot begin to do our necessary work because we are limiting ourselves through willful ignorance. The evidence is there; there is no valid excuse to ignore it now that we are adults and responsible for our own knowledge.

Most of us white Americans grew up not seeing any of this, perhaps even being taught that racism was a thing of the past. Indeed, many American high school history and sociology textbooks happily point out the “post-racial” nature of America as if this concept was somehow true. But, as white adults, it is our responsibility to educate ourselves and make sure we understand reality now. We have the most power out of any racial demographic to uphold or change the way systems work here. We have so much power, in fact, that both our actions and our inaction matter and have real consequences for ourselves and others. We must use this power responsibly and with intention. To do that, we need to learn about and acknowledge the reality of racial inequity in America. We cannot bury our heads in the sand, or our neighbors and community members of color will continue to suffer. Their continued suffering will be on us, just as the suffering of their ancestors was on our ancestors.

The second prerequisite is this: We must understand that we (white Americans) all enact racism, often without even realizing it.

The idea that we white people could grow up surrounded by so much suffering and not even see it astounds me, and yet this appears to be a frequent experience. We don’t suffer due to our race, so we don’t know what it is like and have no personal evidence that it happens to others. We weren’t adequately educated about racial dynamics in school, so we have little to no academic knowledge of it. We were raised by generations that believed racial blindness was the answer to fixing racism (it’s actually racist), so we have little to no immediate community culture that supports the concept of racial harm being real.

So, it makes sense that when we were growing up white in America, we didn’t really have any way to know about the existence of racial inequities, much less which specific actions or systems cause racial suffering for people of any given nonwhite race.

If we don’t know which actions or systems cause people of color to suffer, then it is impossible to conclude that we are avoiding doing or contributing to those actions or systems. We can’t fix what we can’t recognize, so we are given no means by which to prevent ourselves from inadvertently furthering the racial harms in America. In other words, the system we were born into creates racism within ourselves beyond our own awareness, regardless of our good intentions. This in turn means that inaction with regards to our own self-education as adults supports racism, both within ourselves and the structures in place around us.

On the occasion that a person of color tries to explain the existence or experience of racism to us before we have done the work to educate ourselves, and before we have accepted that racism real and “post-racial America” is a myth, that person’s words are so at odds with everything we think we know that we tend not to believe them. Nobody likes being treated as if their experiences didn’t happen, and it’s got an extra layer of oppression when there are racial dynamics involved like this. This is a common example of a way in which white people are directly racist to people of color without even knowing it. This particular kind of racism makes many people of color understandably less inclined to talk to white people about racism.

Here is a more specific example of unwitting racism from a study described in a post from Psychology Today:

Race can also play a role in evaluations of performance and achievement. In one experiment, law firm partners were asked to evaluate a memorandum supposedly written by a third-year associate named Tom Meyer. Half of the partners were led to believe the Meyer was black and the other half that he was white. The partners found twice as many spelling and grammatical errors in the memorandum they thought was written by “black” Tom Meyer than “white” Tom Meyer. And their comments suggested very different assessments of the associate’s capacity: White Tom Meyer was described as having “potential” and “good analytical skills”; black Tom Meyer by contrast, “needs lots of work” and is “average at best.” One partner stated he “couldn’t believe [the associate] went to NYU.” It is doubtful the partners who read and commented on the memorandum saw themselves as racist, but subconscious ideas about academic ability clearly guided their appraisals.
Excerpt From “Racial Dynamics in Education and Health Care” by By Rachel D. Godsil and Linda R. Tropp

You and I, being mere humans, are not immune to this phenomenon. Accepting that we have these internal biases allows us to seek them out and work to dismantle them. We can’t do that work without accepting and recognizing our biases for what they are.

With all that in mind, we can begin to do the work to discern and dismantle racism in ourselves and our immediate spheres of influence.

We cannot do this work until we give ourselves a basic understanding of what the heck is going on. We must give ourselves the education that the public system failed to provide. Once you begin to have an understanding of racial dynamics, you will be better equipped to direct your own re-education, and to have enough context to do things like look up answers to why a person of color told you that something you did or said was racist. This will be a lifelong process with a steep initial learning curve. Why not get started now, with everything shut down anyway?

Here is your initial homework to get you started:

Look up the racial demographics of your state, county, and city.
Find out which indigenous people lived in your area before colonization, and which do now.
Pick one of the racial groups you found above.
Look up “what is it like to be [x] in America” on the internet. Find something written on the subject by someone of that race, and read it.
Repeat steps 3 and 4 until you have an initial idea of the racial situation of your city.
Bonus: Repeat steps 3 and 4 for your the races in your whole county or state.

For those who prefer books, check your local library’s online catalog for downloadable e-books and audio books to maintain isolation to fight this global pandemic. A suggestion to get you started: “Racism in American Popular Media: From Aunt Jemima to the Frito Bandito,” by Brian D. Behnken and Gregory D. Smithers (part of the “Racism in American Institutions” series).

Check out the author’s Patreon