Recently, a large company asked my group to work on a demonstration project of cognitive skills training using the ShadowBox approach we have been developing. But what content area should we use for this demonstration? The company works on a wide range of military activities. However, the company wanted to keep the project unclassified, which ruled out the top contenders.
The company decided that we should focus the training on the engineering specialty of systems integration, which made a lot of sense except that I don’t know anything about systems integration, nor did my team members. Over the years, we have gotten used to jumping into a totally new specialty area, yet we were very uncomfortable with working on systems integration because it seemed to be very technical and likely to require engineering experience and sophistication far more complex than anything we had done before. Despite our uneasiness, we agreed to give it a try.
We showed up for a week of in-depth cognitive interviewing and discovered that the company already had a training program for systems integration. Moreover, it had a 42-page manual (plus appendices) detailing the phases and steps of systems integration. We thumbed through the manual and it seemed extremely comprehensive, covering every part of the systems integration process. If we felt inadequate before, we felt even less confident now.
What in the world could we add?
And then, as we thumbed through the manual, we realized what was missing:
- What are the tough decisions here?
- What makes situations difficult?
- What can go wrong?
- Where do systems engineers get confused?
- What kinds of mistakes do new systems engineers make?
- How can you recover from mistakes?
- How does the mindset of a new systems engineer change with experience?
- What are the difficult tradeoffs in a systems engineering project?
- What tactics have experienced engineers learned for managing risks?
- What do highly experienced systems engineers see and understand that their less-experienced colleagues miss?
None of these topics was covered in the 42-page manual.
By the end of the week, after conducting eight in-depth cognitive interviews, we had identified a set of cognitive training requirements, including some critical mindset shifts, and had sketched out a number of ideas for ShadowBox scenarios. We ultimately worked with the company to formulate six scenarios and tested three of them, showing that the ShadowBox training could potentially substitute for some of the expensive instructor-led training they were currently using.
Afterwards, I was struck by the changes in attitude, my own attitude and that of the sponsor, regarding what we might contribute. The sponsor was justifiably proud of the manual on how to do systems integration — it captured the processes and procedures. Yet it was clear to everyone what was missing — the cognitive dimension. The questions listed above illustrate the cognitive dimension but you shouldn’t assume that all it takes is to ask these questions.
Moving into the cognitive dimension takes experience and a mindset shift, from believing that complex tasks can be performed by carrying out the steps, to a mindset that appreciates the subtleties and the ways that context affects which steps you perform, how you perform them, and how to adapt beyond the steps when necessary. It is a mindset centered around curiosity so that you can ask the next question and the question after that, excited to learn more about the tacit knowledge that underpins expertise.
