From deciding which soap to use or which light bulb to buy to discerning which politician is telling the truth about things like climate change or the proper approach to Covid-19 concerns, science is a big part of our daily lives. Most people, though, aren’t scientists. Chances are, you don’t have the time or money to just go out and get yourself a degree in a science field.

How do people go about fact checking scientific information when most people don’t have a background in science? In my experience as a scientist, frankly, most people do it rather badly. The worst part is how many people don’t realize that they are doing it badly and consequently both draw horribly incorrect conclusions and profess their false ideas to others, spreading misinformation.

And yet, as a tutor, I also recognize that people are genuinely doing the best they can with the information available to them. It is very easy to draw the wrong conclusion when the information one has is incorrect or insufficient. My intention with this piece is to provide information and context to support the non-scientist reader in spotting bad science, incorrect science reporting, and the outright lies so many politicians are fond of spouting.

Part 1: The Scientific Method and Experiment Design

“Science” refers to a method for discerning truth, to a body of knowledge collected by generations of scientists, and in many ways, to a culture. The scientific method is the process by which scientists determine which facts are true.

The general steps of the scientific method are as follows:

1. Propose a question.
2. Do background reading about relevant known information.
3. Form an hypothesis (an informed guess as to the answer of the question).
4. Create an experiment to test the hypothesis.
5. Run the experiment.
6. Collect data.
7. Analyze the data.
8. Form a conclusion about the hypothesis, or go back to step 4 and repeat the process as needed to have enough information to form a conclusion.
9. Communicate the conclusion.

If you are new to the scientific method, check out this site for more details in an interactive format. It’s designed for kids doing science fair experiments, which makes it easier to read regardless of your age. In fact, science materials designed for kids are great for all ages for just that reason. If you want something more detailed, the kids version will give you enough context to be able to find and understand it.

Each one of the steps in the scientific method has its own standard set of rules intend to guide the scientist toward truth while circumnavigating the scientist’s own biases. Experiment design is a big part of this.

Controls are necessary. Scientists use an experimental group and a control group when doing experiments in order to have a basis for comparison.

For example, when I was in sixth grade, I tested my fifth grade teacher’s running program to see if it impacted lung capacity. I measured the lung capacity of her entire class of students periodically throughout the term. This showed a growth in lung capacity, but that alone was not enough information to determine whether there was a correlation between the running program and the changes in lung capacity. Other variables abounded, such as the changing seasons, the natural growth of children over time, and so on. In order to determine whether there was a correlation, I needed to use a control group which had all the same variables except the running program. I used the next door fifth grade class in the same building, which did not have a running program. This group did not show any growth in lung capacity, making the data I collected far more useful.

Note: If the class I used for my control group had been kids of a different grade, or at a different school, or if I had collected the information during a different part of the year, then it would not have been a good control group because here would have been more than just one variable that was different between the experimental group and the control group.

When evaluating scientific information, claims, and journalism, keep an eye out for problems with the scientific method or with control groups. If you can’t determine whether these things were done properly, it’s best to disregard the information as unverifiable. It may be true, or it may not be. Not knowing is part of science. Get comfortable with not knowing.

Part 2: The Reality of Science

The reality of science is a bit imperfect compared to what popular media has to offer. Have you ever watched a science fiction show where a character uses a handheld device or other instrument to “scan” something, and instantly finds out a whole pile of information about the object in question? That is pure fiction. The real world doesn’t have that kind of technology yet, and likely won’t in our generation’s lifetime. If a scientist needs to test a water sample for contaminants, for example, that scientist will need to run a separate test for each potential contaminant, and each of those tests takes time. The length of time ranges from minutes to days depending on the procedure. For bigger projects, such as studying a new phenomenon, science takes months or years, sometimes even decades, to produce reliable conclusions.

The reality of the timing involved in science is one of the key concepts you can use when fact checking science articles. Claims of detailed knowledge of brand new phenomena are probably not based in evidence, whether or not they end up turning out to be correct guesses. Think back to when Covid-19 first started. Remember all the firm claims that ended up being wrong? Be wary of science reporting about new phenomena, especially if the reporting doesn’t take into account the concept that “we don’t really know for sure yet” in the language.

In addition to limitations, science is subject to bias. This is true both of scientists when conducting science, and of reporters when engaging in science reporting. Scientists are products of our own cultures, which means our biases influence the way we think, and therefore the way we design experiments and interpret the results. This is especially prominent in anthropology (the study of humans), but all other scientific disciplines struggle with this as well.

In order to help prevent biases and other issues from harming the validity of the scientific body of knowledge, scientists participate in something known as “peer review.” When science journals publish scientific research, the process involves having other scientists who did not work on the project review the documents, procedures, and conclusions for proper scientific method and accuracy. This process is very rigorous. When fact checking science, look for peer reviewed articles.

And finally, be aware of the limits of scientific observation. There is a lot we don’t know about, due in part to simply not having the technology to allow us to observe it. Not knowing is part of science. Get comfortable with not knowing, with being willing to hold space for an unknown rather than trying to fill it in without having enough evidence to do so.

Part 3: Understanding The Numbers

Numbers are a big part of science. We use them to describe our observations so that other people can understand what we witness. We also use them to do calculations to figure out more information about our observations.

As you might already know, math can get very complicated. However, you don’t have to go learn calculus or statistics in order to be a discerning individual when it comes to evaluating scientific claims. That said, the most important math to understand for this purpose is probably statistics. Statistics are often twisted, misrepresented, and simply misunderstood in science journalism and by politicians.

Please pause after this paragraph to watch the 12-minute video embedded below to start to get a basic idea of how statistics works. If there are vocabulary words you don’t understand, then pause the video and look them up before continuing. This video contains example problems to do on your own. You can use them to evaluate your understanding of the concepts. If you can do them, then you have a grasp of this knowledge and can use it when evaluating scientific claims. If you can’t, or if you choose to skip them, that’s okay too – it means that you know you don’t have a strong enough grasp of these concepts to evaluate related claims. If this is you, then it is important to remember to put any statistical information you see in the “I do not know if this is true because I am not able to evaluate it” box in your head rather than immediately believing or disbelieving it. Again, get comfortable with not knowing.

When you are reading about science, activate your critical thinking whenever you see numbers. Are units provided? If something went 13 kilometres, that’s a lot farther than 13 feet. Are numbers presented in a way that makes sense? For example, looking at the total number of deaths due to Covid-19 between countries is not as useful as looking at the total number of deaths per capita. “Per capita” means “per person.” This is how to adjust for population size. Think of it this way: If 1,000 total people die in Rhode Island, that’s a lot different than if 1,000 total people die in California because California has so many people in it. If 1,000 people die per every 5,000 people in Rhode Island, then it is the same rate of death as if 1,000 people die per 5,000 people in California. This is why “per capita” numbers are often more useful than total numbers.

Part 4: Reading Scientific Literature

If you have never read a scientific study before, it can look daunting. Studies are filled with scientific jargon making them difficult to read, even for scientists. The key is to read them more than once, look up words you don’t know, and focus on specific parts of the study.

The abstract is a good place to start. This section summarizes the process and results of the study. Sometimes the abstract has all the information you need to fact check the article or meme which was supposedly based on the study. The other parts of the study are valuable if you wish to gain a detailed understanding of how the study was run, especially if you wish to evaluate whether it was done properly.

Part 5: Evaluating Articles

As mentioned above, research journalism is rife with bias and outright error. Be skeptical of headlines designed to evoke an emotional response. Here are some practical questions to ask yourself when evaluating articles that make scientific claims:

• When was this article written? If it was written a long time ago, have there been new breakthroughs since then?
• Who wrote it, and why? How might that bias the writing?
• Does the article site its sources, including links to any scientific studies the article claims as sources? If not, disregard the entire article as it is not credible.
• Do the abstracts of the original studies actually support the claims in the article?
• If the article uses numbers, does this article use them in a way that makes sense?

Part 6: Sharing Information

If you struggle with holding space in your mind for the unknown, it may be difficult to read false information without absorbing some of it into your belief system. There is a big difference between believing something yourself and asking others to believe it. To ensure that what you share with others is true, it is a good idea to create a system for determining what you will share. Consider these questions:

• Have I actually fact checked this, or am I only sharing it because it fits with what I already believe?
• Am I sharing this because it is true, or because I would be allowing my anger, hope, or another emotion press the share button for me?

Sometimes you won’t be able to fact check something. Maybe it relies on statistics you don’t understand, or maybe there is a paywall between you and the original study. If you can’t fact check it, then what? What do you do with that informational meme or science article which makes a really good point, but which you are struggling to fact check? In my opinion, you scroll past it, or you find an expert to ask about it. Don’t share what you can’t fact check.

This post topic was selected by the author’s Patreon patrons.