Verified by 
In August, it will be 10 years since I received my Ph.D. and I will no longer be considered an early career researcher. Ear hairs. Prostate screening exams. Black socks pulled up to my knees. Scary things are on the horizon…
Before I cross over into old age, letting the younger generation take over the creative reigns, I need to do a retrospective. Let me share the lessons learned from 5 research publications that don't sit well with me. This is my confession.
For those of you who aren't scientists, you need to know an important truth. Just because research is published in a peer-reviewed journal by a reputable publisher does not mean the science is good. There are so many journals that exist. Any time a research study gets rejected, it can be resubmitted somewhere else, and if it happens again and again, scientists can keep on resubmitting until some horrendous outlet takes it (see Traumatology as an example of a journal that published papers that better outlets rejected until eventually everyone realized this and it folded). Think about this the next time you scroll random journal articles to find scientific evidence for your belief (ignoring the counter points) that vaccinations cause autism and hands-free phones are the answer to safe driving (by the way, I had to read through 40 articles to find one that suggested hands-free phones are not that hazardous to driving…but I did it, and now you too can tout scientific evidence that the hazards are overblown!).
But back to a naked assessment of 5 studies from my own research laboratory. Each of the psychological studies below is questionable in its own unique way. Rather than just tell you why, I want to provide reinterpretations and fresher insights about human beings…
1.
Kashdan, T.B., Collins, R.L., & Elhai, J.D. (2006). Social anxiety, positive outcome expectancies, and risk-taking behavioral intentions. Cognitive Therapy and Research, 30, 749-761.
At first glance, this seems like a really useful contribution to science. We thought to ourselves that there is no way that every single person that is socially anxious, worried about being criticized and rejected by others, is going to be shy and inhibited. There must be some socially anxious people who cope with their anxiety by doing outlandish, risky things. If I worry about being rejected and hurt by you then one way to cope is to hurt you first. That is, I deal with my fears of being rejected by being aggressive and violent towards other people. Sounds awesome. Too bad this study was a horrible test of this idea. Instead of measuring how often socially anxious people are aggressive, violent, use drugs, or engage in risky sexual activities (if you can't get love, you can buy it with prostitutes), I asked them how often they THINK they will act this way over the next six months. Worse than that, I didn't give them a calendar to put a mark down each day they plan to have sex without protection, smoke crack, or punch somebody in the face. No, I handed them a single piece of paper with a list of 30 risky behaviors and asked them to put down a number for often they will do each one over the next 180 days. Trust me, the research participants did not reflect on these questions for more than 5 seconds. These participants wanted to move on to the next item and then the next, to get through a 10-page survey as quickly as possible. Think about this—if you tend to get into arguments with co-workers, can you really tell me how often it is going to happen over the next six months? It would take you some time to think about who you are going to see, what you are going to talk about, and whether there is anything that will increase the likelihood that you will unleash your fury (perhaps an open bar with bottom shelf whiskey). And thus, this study is nonsense.
But this was the first study to test this question and it led us to move away from what people expect to do to asking people what they actually do. And this led to a growing body of research showing that in fact, approximately 1 in 5 people with social anxiety problems is disinhibited (instead of inhibited), aggressive (instead of passive), risk-prone (instead of avoiding danger).
An important lesson is that if you want to understand what people do, make sure that you measure what people do. Be skeptical about research that only measures what people expect to do because people are bad at predicting how they will feel in the future and what they plan to do.
2.
Steger, M.F., Hicks, B., Kashdan, T.B., Krueger, R.F., Bouchard, T.J., Jr. (2007). Genetic and environmental influences on the positive traits of the Values in Action classification, and biometric covariance with normal personality. Journal of Research in Personality, 41, 524-539.
With the completion of The Genome Project in 2001, interest in the nature versus nurture debate has surged. A personality trait feels real if you can get a specific number that captures the contribution of genetics verus the contribution of unique environmental factors. This is the only study of genetic differences I have been part of and there is something about the approach we took to this study that never felt right. We had a great sample, collecting data from 336 sets of twins, with a mean age of 49, who completed the most popular measure of 24 psychological strengths and virtues. But I have no idea how anyone can make sense of our findings. Of these 24 psychological strengths, humor has the smallest genetic contribution (14%) and spirituality has the largest genetic contribution (59%), and in the middle lie strengths such as humility/modesty (25% genetic contribution) and integrity (38% genetic contribution). You might want to ask me—how do you measure humility by asking people on a questionnaire how modest are you? My answer—I have no idea. You might want to ask me—does knowledge about the contribution of genetics to strengths and virtues help a person? My answer—no. You cannot apply any of these findings to a single individual and the sample size isn't big enough to make sufficient statements about the population (in the United States? in the Midwest of the United States). If there is one thing we have learned it is that there is no battle between nature and nurture. There are genes that act like switches, turning on or off depending on the presence of particular environmental or biological factors (the field of epigenetics). Our brain is an intricate machine and our hardwiring influences our decision making, such that someone who is high in sensation seeking might be more apt to choose friends who are also impulsive, novelty seekers and with this group, delinquent behaviors might become common, setting off a path that increases the likelihood of academic problems, being underemployed, and greater financial problems and stressors as an adult. It is overly simplistic to separate the environmental and genetic contributions of particular aspects of our personality because our sensitivities and preferences influence what we do and over time, this determines what we become. Everything is mingled together in a tapestry. This study gives the illusion that we can surgically separate nature and nurture, which we cannot do with the numerical precision presented.
3.
Post-traumatic growth is what all the cool kids are talking about. The notion that a traumatic event does not always break a person and that sometimes, a person rebuilds their life with gifts such as a greater appreciation of life and refined sense of priorities; strengthening of significant, close relationships; recognition and elaboration of personal strengths; recognition of new possibilities or a sense of purpose for one’s life. Sounds awesome. Those of us who have seen clients know that this happens but as for the science, its sketchy at best. And our study only added to a sloppy literature.
Essentially, to know whether someone grows in the aftermath of a trauma you need to know what trajectory they were heading towards prior to the trauma and whether the onset of the trauma altered that trajectory such that they bounced forward or upward toward greater health and prosperity. Unfortunately, like far too many people that study post-traumatic growth, we did not study people over time. Nope. We just gave 176 people who suffered a traumatic event a survey packet including the 21-item Post-traumatic Growth Inventory (where they had to give an answer from "0" - I did not experience this change as a result of my crisis to "5" - I experienced this change to a very great degree as a result of my crisis for items such as "I have a better understanding of spiritual matters"). Do you even realize how complex of a question this is? These trauma survivors have to calibrate what path they were on before the trauma hit, what path they are now on after the trauma, compare these two paths to determine whether the latter one is better, and then determine the degree to which that the difference between these two paths can be explained or caused by their crisis. They have to compute this complicated algorithm 21 consecutive times in a row and then the questionnaire is done.
Here's a not-so-hidden secret about research—if you ask a question, people will answer it, even if the question doesn't make sense or is far beyond their computational capacities. Don't blindly trust the measures that researchers use, take a close look at the exact items. Then and only then can you adequately assess the quality of the research.
4.
I am conflicted about adding this study because I think the findings are interesting. In this study, we collected data from 75 military veterans who were diagnosed with post-traumatic stress disorder (PTSD). It is wrong to assume that the presence of PTSD means that everything in life is problematic. Decades of research have shown that the use of certain coping strategies such as cognitive reappraisal are generally effective in reducing anxious thoughts and feelings. We thought that this storyline seems too simplistic. We figured that whether or not the regular use of cognitive reappraisal is effective might depend on whether you are skilled at recognizing, understanding, and clarifying what you feel in a given moment. And this is what we found. Military veterans with PTSD only benefitted from the regular use of cognitive reappraisal if they also excelled at being aware of their emotional experiences. Veterans with PTSD who were less knowledgeable about their emotional life experienced zero benefits from their use of cognitive reappraisal. Sounds great until you take a closer look at the methods used…
If you were interested in intelligence, would you find it acceptable for scientists to simply ask children to answer the following question: "Please give an answer from a scale of 0 (not at all) to 10 (absolutely), in general, do you tend to be an intelligent person?" Of course not. You would want them to solve problems and complete tasks. Intelligence is a skill that can be observed. The same goes for driving ability, pain tolerance, and video game playing. If you are low in intelligence then by definition, you are probably less accurate in answering questions about your intelligence. The same goes for emotional clarity. If you find it difficult to be aware of, understand, and describe what you are feeling then by definition, you are going to be less accurate and reliable in estimating how you excel at this skill. This is why studies that use self-report questionnaires to assess intelligence, emotional intelligence, and in this study, emotional clarity may be problematic. Its worth asking whether what you are interested in is being measured in the way it should be measured. Would you be confident police officers can give accurate answers to a questionnaire of whether they show biases in treating all citizens equally, regardless of race, sex, religion, or sexual orientation? Would the results of fMRI tests be useful to determine who is best friends with whom in an elementary school? Remember what you are interested in and what questions are trying to be asked. These will determine the appropriate methods.
5.
Kashdan, T.B., Rose, P., & Fincham, F.D. (2004). Curiosity and exploration: Facilitating positive subjective experiences and personal growth opportunities. Journal of Personality Assessment, 82, 291-305.
For a long time, this was my most widely cited paper. We measured two dimensions of curiosity and I can tell you now that one of them is wrong. Yes, curiosity is about recognizing and seeking out new information and experiences, a dimension that we refer to as Exploration. The problem is with the second dimension that we refer to as Absorption - the tendency to lose track of time, be tough to interrupt, and fully engaged in activities. This often happens when we are curious, but it also happens when we pay close attention to something or someone. You might be feeling fear, anger, or awe, and not necessarily curiosity. This is why we created a second version of the scale with two dimensions that are central to curiosity—the motivation to seek out knowledge and new experiences (Stretching) and a willingness to embrace the uncertain and unpredictable nature of everyday life (Embracing).
Stop using the CEI! It was my first attempt at creating a personality scale and let's be honest, it is horrid. We improved the assessment of curiosity with the CEI-II. Let's view the first version as a nice stepping stone, a sloppy footnote that helped me create something to be proud of. And then I realized that I was still ignoring a large portion of this thing called curiosity. It required 14 years of reading, thinking, and synthesizing after the publication of my Curiosity and Exploration Inventory that I finally nailed down the construct. I recently published a vastly improved measure that you can download here:
(but stay tuned, we found a tiny flaw that we fixed and are now writing up for publication. with the right amount of intellectual humility and perseverance, science can be self-correcting.)
I am fortunate to have great collaborators. I'm surrounded by creative minds. Often our research program starts off slow and I am not confident about each finding that comes out of my laboratory (or from other laboratories). I stay attuned to the main objective of why I am a psychologist…understand some of the mysteries of human behavior and in some small way, reduce the amount of suffering and increase the amount of well-being in the world. This cannot be done with a premature commitment to being right. This cannot be done by blindly accepting theories, research, and treatments that other people promote. But the key is to be skeptical, not cynical. Be curious, keep experimenting, keep learning, and most importantly, keep asking questions. And part of this storyline is to be naked, exposed, and vulnerable every once in awhile.
Dr. Todd B. Kashdan is a public speaker, psychologist and professor of psychology at George Mason University. If you're interested in learning about and downloading additional work, go to: toddkashdan.com
