2013: Science's Seven Sins and a National Violence Project
Seven sins of psychological science and reflections in 2013
Posted Jan 10, 2013
1. The Seven Sins of Psychological Science, +/- Two.
Do we publish too much, a cacophony of too many small one-shot unreplicated very limited studies with an explosion of new online and offline journals? The media picks up on these and runs with them to the public without much attention to the sharp limits of study design and sample, generalizability, causal inference, and other issues. There are more than five to nine "sins" (Farley, F., Jennings, H., Jean-louis, G., Smith-Dyer, T., & Aulenbach, T., 2011) but let me outline some. 1) REPLICATION: Direct replication is something psychological scientists rarely do. 2) THE SAMPLE: Overwhelmingly convenience samples, particularly small samples of college undergraduate students in psychology classes, with published journal abstracts often reporting generalizations to people at large, rather than to comparable college undergraduates in psychology classes ( this heavy use of such research participants raises the question as to whether psychological science is to a substantial extent the science of college student behavior). 3) OVER-GENERALIZATION: Made from very narrow limited samples as noted above, and very limited conditions and measures. 4) MONASTIC RESEARCH: The high frequency of laboratory-based/university-situated studies with excessive use of simulations and participant self-reports with weak or unknown relationships to real-world behavior beyond the monastery walls ( n.b., most people don't live in labs, grow-up in labs, form a life and relationships in labs, etc.,--get out to where people live and use authentic real-life tasks and experiences). 5) FILE-DRAWER EFFECT: Much research never gets published, for various reasons, that can include a failure to find statistically significant results making publication acceptance more difficult, so it languishes forgotten in a file drawer, but perhaps if published could change the conclusions drawn by psychological science and alter the direction of the field and also what the public learns.6) LESS THAN PEERLESS PEER REVIEW: Where too few reviewers are used, or are weak on research design and statistics, and acceptance/rejection decisions can be biased or arbitrary or ill-informed. 7) THE "NEOPHILIA EFFECT": Reflecting a substantial focus on NEW findings, NEW concepts, NEW measures, frequently one-off studies with novel, unusual and sometimes counter-intuitive unreplicated findings that are often picked-up by the media because of the "NEW" and the unusual features. 8) THE NEW REDUCTIONISM: Where a veritable niagara of specific brain processes and structures are expounded as the base of and cause of a dizzying array of human behaviors, attitudes, motivations, social phenomena, economic behavior, etc., frequently derived from small limited laboratory studies of college students, self-report questionnaires or simple tasks with unknown real-world validity, or simulation tasks, and the ubiquitous fMRI—a possibly reactive test situation, with correlation frequently confused with causality. The foregoing are not characteristic of all psychological research of course, but are widespread features that need serious discussion in 2013. Psychological science has enormous importance in significant applied areas such as health, education, crime, personal life, and so on, and we want it to meet the highest standards. The public deserves no less. So as you read about psychological findings, see lists of things you can do to make yourself happy, motivated, successful, attractive, long-lived, etc., allegedly based on scientific studies, check those studies against the foregoing before you buy. In the incredible mass of information arrayed before us in this information and internet age, I believe some skepticism is a survival skill.
2. A National Violence Project
There is a need in 2013 for a National Violence Project to bring together the greatest minds and most experienced basic and applied researchers, thinkers, practitioners and stakeholders to conceptualize effective means to reduce violence in America. Any ideas with strong potential to help would be on the table.Piecemeal approaches to the problem of violence aren't working; a NATIONAL coordinated organized collaborative approach is needed, similar to the Manhattan Project of WWII, the Moon Mission of the 1960's where President Kennedy asked science to get us to the moon and in a few years we were there, and the more recent Human Genome Project. This proposed National Violence Project would generate the best solutions we have, and outline needed areas of further research. It could be funded by government and the private sector, including foundations, and certainly given the topic--crowd-sourcing. I believe development of such a National Violence Project should be a priority for the human sciences and policy makers in 2013, and should be a White House backed endeavor and a priority. Let me add at a grander level, the global need for psychology to confront "The Horror", to humanize an inhumane world. It must become psychology's job #1. The problem of horror is the greatest challenge facing psychology--murder, torture, terror, slaughter of the innocent, rape, slavery, starvation, absence or miscarriage of justice, false imprisonment, and on and on.
3. Re-Evaluating American Education
Education reform in America is a long-term mantra of educational policy, education research , the science of testing and assessment, etc..America wrings its hands over the dire straits of education, looking to anyone for help--John Legend, Lady Gaga, and other education experts.Its clear that fresh thinking on this topic is needed in 2013. In my view, American education is bad in places, we know exactly where those schools are, but in the majority of American schools, education is good to excellent. American public education is necessarily messy, as one would expect in a democracy where public schools are mostly controlled locally by thousands of local school districts and school boards of local citizens, and the financial base depends primarily on local real estate values and real-estate taxes. Decrepit real estate with little or no value (think inner city Detroit, Philadelphia, etc) means no means to support those resource-needy public schools. Our much touted low international standing on science and math scores, far behind a tiny dictatorship city-state like Singapore where there is central control over education, or a tiny democracy like Finland where there is, again, central control over public education, looks bad, but look closer. Firstly, Singapore and Finland are small precious boutique school systems compared to the behemoth USA 100,000+ schools across a vast land of a highly diverse population. Secondly, we will never be able to achieve such central/federal top-down control. Third, these international comparisons are overly based on to some extent inappropriate assessments where the tests used in the rankings are concerned. These assessments fail to sufficiently tap America's signature strength--creativity and innovation. If one takes a bigger view, stands back from these tests of science and math, and instead examines the global scientific and mathematically-related innovations of recent decades and currently, the U.S. wins hands down! These international test comparisons over-measure convergent thinking and under-measure divergent thinking, the latter being America's great strength, the fuel for innovation and creativity. I'm not saying we can't improve American education overall--we can and should. However, I hope to see in 2013 our educational-psychological research and public policy focus explicitly on the well-known sub-set of schools that are failing and give them the help they need, the resources they need. And lets scale way back our concern for these inappropriate international comparisons.
Diagnosis is an extremely important issue in much psychological practice, and applied research. 2013 is a watershed year on this topic as the 5th revision of the "bible" of diagnosis--the Diagnostic and Statistical Manual (DSM) will be published as DSM-5. The DSM has not been revised since the 1990's. Schizophrenia, serious depression, autism ADHD, etc., usually require a diagnosis, and the DSM-5 as a system of choice has been the subject of wide-spread debate and controversy.A major criticism has centered on the quality of the science in the DSM-5. A document with such high impact on America's health must be based on the very best science, and a number of scholars and professionals are skeptical of the science. Looking to 2013, an international movement has arisen toward re-conceptualizing diagnosis from the ground up (Farley, 2012) and I look to significant progress on this important topic.
Farley, F. (2012).Reboot diagnosis: DSM-5 goes live, nascent movement arises. PsychologyToday.com
Farley, F. Jennings, H., Jean-Louis, G., Smith-Dyer, T., & Aulenbach, T.(2011). Examining psychological science--some inconvenient truths in the New Monastic Order. American Psychological Association Annual Meeting, Washington, D.C., August 7.