The Limitations of the Results of a Statistical Analysis (Corporate)

What a Statistical Analysis Might Not Be Able to Explain or Show Us, and Why it’s Super Important to Know These Things!

• Validity and reliability: Validity asks the question of whether our data is actually measuring what we are interested in knowing. This seems straightforward for certain things like measuring distances or heights but is more difficult when you try to measure something like feelings or satisfaction. Reliability asks whether the tool you are using to measure something gives you the same results every time. Again, certain mechanical tools can be very reliable, like a ruler or scale, but other tools, like surveys, can be less reliable, being affected by things like time of day, whether it is in person or online, and the quality of the questions. Imagine this example. You have a scale that always reports people are 5 pounds lighter than they actually are. The scale is very reliable (same results every time) but is not a valid measure of weight because its measures are not accurate.
• Generalizability: Let’s say you found out that using blue paper makes your airplane fly the furthest. But can you say that all blue paper is the best for everyone or for every airplane design? We have to be careful about using our findings too broadly. This is where knowing the limitations of our study can help.
• Planning future research: So, you’ve figured out something about blue paper and airplanes. But what about red paper? Or green paper? Knowing what your study didn’t cover can help you plan what to try next.
• Ethics and transparency: We all know being fair and honest is important. And it’s the same with statistical analysis. We need to be open about what our study can and can’t show. It’s like when you play a game! You must follow the rules and be fair to everyone.
• Informed decision-making: Informed decision-making means making smart choices based on reliable information. Just like you wouldn’t pick a team for a basketball game solely based on who’s the tallest without considering their basketball skills! We understand that statistical analysis is just one tool in the decision-making process, and it’s important to consider other perspectives and sources of information as well.
  • Example of an informed decision: When we interpret the results of a survey-based observational t-test, we need to understand its limitations. While a statistical analysis, like an observational t-test, can provide us with valuable insights and help us understand patterns or relationships in the data, it has certain limitations. It might not be able to explain or show us everything about a situation or phenomenon. By being aware of these limitations, we can make smarter choices. We can consider other factors, gather additional information, and use critical thinking skills to make more informed decisions.

But How Do We Find Out What a Study Might Not Be Able to Explain or Show Us?

• Check how the study was set up: Just like how the rules of a game can change how it’s played, how a study is set up can affect the results.
• Look at the study sample: If only a few people are studied, we can’t say much about everyone else.
• Check for bias and other factors: We need to make sure nothing else was influencing the results.
• Look at what was measured: Did we measure the right things? (validity). Make sure the things measured in the study are relevant and accurate for what we want to know. Check if the measurements are reliable – would they give the same results if we did it again?
• Look for guesses or assumptions: Did the study guess something that might not be true? For example, we created a survey question that asked whether you took the bus, drove, or were driven to work but left out a category for walkers and bikers.
• Look at how the results were interpreted: Do the conclusions make sense?  Think about whether the conclusions made from the results make sense and are backed up by the data. Be careful of making too broad or too strong statements based on the results.

Navigating the Pitfalls: What to Watch Out For

• Don’t mix up “related” with “causes”: Just because two things happen together doesn’t mean one causes the other. Ice cream sales and crime are positively correlated, but it seems unlikely one actually causes the other (e.g., it is more likely that warm weather impacts both).
• Don’t forget where and how the data was collected: The way data is collected can affect what it tells us. Studying highway traffic from 1 to 2 a.m. will not tell us little about traffic from 7 to 8 a.m.
• Don’t forget what the study was designed to answer: It’s easy to get excited about interesting results, but remember to focus on the question the study was trying to answer.
• Don’t forget the time frame and scope of the data: The data collected only represents a specific period and area. For example, a study on polar bears in the Arctic in 2020 can’t tell us about polar bears in Antarctica in 2023. Remember to consider the time and place of the data when making conclusions or using it to answer questions.
• Think about how the results could be misunderstood: Results can sometimes be used in ways that the study wasn’t designed for. For example, a study showing that people who exercise more have less stress doesn’t mean that all types of exercise reduce all types of stress. Be cautious of generalizing results too much or using them to support unrelated ideas. It’s best to stick closely to what the study was really about.