Artificial Intelligence
How to Evaluate Research Articles and AI Information
A guide for critical thinkers in a changing information landscape.
Posted January 26, 2026 Reviewed by Devon Frye
Key points
- Critical thinking means examining rival hypotheses to spot alternative explanations.
- Experimental and researcher effects can subtly shape results through cues or study design.
- Examples like immigration and exercise and mental health show that multiple factors can influence outcomes.
- AI-generated information may reflect biases, outdated data, or overgeneralizations.
In memory of Robert Rosenthal
When I was a postdoctoral fellow at Harvard, I had the great fortune to be mentored by Robert Rosenthal, who passed away in 2024. His courses on Research Methods and Statistics were epic, and his research was groundbreaking. For over 50 years, he studied how experimenters’ expectations can influence research outcomes and how clinicians’ expectations can affect patients’ mental and physical health. He taught me how to think like a scientist, and I am writing this article in his memory to honor his remarkable contributions to science and critical thinking.
The Importance of Rival Hypotheses, Experimental Effects, and Researcher Effects
When reviewing research articles, it is essential to think in terms of rival hypotheses, which are alternative explanations for observed results. For example, if a study finds that immigrants are more likely to start small businesses, a rival hypothesis could be that the effect is actually due to local economic conditions or the availability of community support networks, rather than something specific about immigrants.
Other possibilities include reverse causation (perhaps areas with more entrepreneurial activity attract immigrants) or measurement errors (misclassifying what counts as “starting a business”). Using these questions systematically helps prevent us from accepting faulty conclusions at face value.
In addition to rival hypotheses, critical thinkers should consider experimental and researcher effects that can subtly influence results. Robert Rosenthal’s work highlights factors such as experimenter expectations, participant awareness, or subtle cues that may affect outcomes. These effects occur when the behavior, attitudes, or responses of participants are unintentionally shaped by the way a study is conducted or by the researchers themselves. Understanding these influences helps readers recognize that reported results may reflect not only the variables being studied but also the context and interactions within the research process.
Applying Rival Hypotheses to Real Studies
When researchers study attitudes toward immigration or mental health outcomes related to exercise, for instance, subtle factors can shape results in similar ways. In an immigration attitudes study, for example, participants may answer in ways that reflect researcher expectations—say, if pro-immigrant responses are unconsciously signaled—rather than their true beliefs.
In an exercise and mental health study, participants might overreport feeling better because they know that exercise is “supposed” to improve mood, or the social support of a fitness group could be the real factor driving improvements rather than the exercise itself. In both cases, survey design, question wording, or external context, like recent news coverage or public awareness of mental health campaigns, can unintentionally influence outcomes. Recognizing these effects helps assess whether study findings are robust or potentially biased.
Structured Critical Reading
By combining the practice of generating rival hypotheses with attention to experimental effects, critical thinkers develop a structured approach to careful reading. For instance, when reviewing a study claiming that immigration increases crime in a community, critical thinkers can ask:
- Could other factors, such as poverty, policing patterns, or demographic changes, explain the results?
- Could researcher expectations or survey design have influenced the findings?
This dual approach fosters critical thinking, encouraging readers to reason like scientists rather than passive consumers of information. It also helps build a habit of asking, “What else could be going on here?” rather than automatically accepting reported conclusions.
Quick Checklist: Rival Hypotheses and Experimental Effects
Rival Hypotheses:
- Could other factors explain the findings?
- Could reverse causation be at play?
- Are there measurement or classification errors?
- Are there confounding variables?
Experimental/Researcher Effects:
- Could researcher expectations influence participants?
- Could survey or experiment design shape responses?
- Could participant awareness affect behavior?
- Could external context influence results?
Example (Immigration Study)
- Claim: Immigration increases entrepreneurship.
- Rival Hypothesis: Local economic support, community networks, or pre-existing business culture.
- Experimental Effects: Survey wording or researcher cues could influence reported attitudes.
Example (Exercise and Mental Health)
- Claim: Exercise improves mental health.
- Rival Hypothesis: People who are already mentally healthy may be more likely to exercise, or social support from workout groups or community fitness programs may be the actual driver of improved mental health.
- Experimental Effects: Survey or study design could influence reported outcomes, for example, participants may overreport feeling better because they know exercise is “supposed” to help, or researchers’ expectations could subtly cue participants to emphasize positive effects.
These two examples illustrate how the same critical thinking framework can apply across different fields. Both studies involve human behavior and outcomes, yet in each case, multiple explanations are possible.
By systematically considering rival hypotheses and experimental/researcher effects, readers are better equipped to evaluate whether the reported results truly reflect the causal relationship suggested or whether other factors may be responsible. This also highlights the importance of thinking contextually: local conditions, participant characteristics, and external events all matter in interpreting findings.
Applying These Techniques to AI-Generated Information
The same critical lens applies to information from AI platforms like ChatGPT. For instance, if an AI suggests that immigration increases entrepreneurship, a rival hypothesis might be that the information:
- Reflects biases in the sources it was trained on.
- Uses outdated statistics.
- Overgeneralizes across countries or regions.
Other alternatives could be that local economic conditions, government programs, or community networks are the real drivers. Asking, “Could there be another explanation?” or “Could this information be incomplete or framed in a certain way?” helps ensure AI outputs are evaluated as rigorously as published studies.
Similarly, prompt phrasing, context, and model training can influence AI responses, just as experiment design and researcher expectations affect study outcomes. For example, if a prompt emphasizes entrepreneurship as a “positive trait,” the AI might highlight cases that confirm this perspective while ignoring others.
Recognizing these effects encourages critical thinkers to evaluate AI-generated content with the same rigor as scientific literature. Over time, practicing this evaluation strengthens the ability to identify potential bias, gaps, or oversights in any source of information, whether human- or AI-generated.
Key Takeaways
Whether reviewing research articles or AI outputs, systematically considering rival hypotheses and experimental/researcher effects strengthens critical thinking and helps readers interpret evidence with the discernment of a scientist. By consistently asking alternative explanations and examining study design or information context, critical thinkers avoid being passive consumers and instead become active evaluators of knowledge. Applying these principles broadly, across social science, health research, and AI-generated content, builds habits of careful reasoning that are essential in today’s information-rich world.
References
Rosenthal, R., & Rosnow, R. L. (2008). Essentials of behavioral research: Methods and data analysis.
Weissmark, M.S. (2025). Seeing the Other Side: Shifting perceptions. Amazon KDP.
