Failure: a critical element on the path to success

Several years ago, when I was a novice astronaut, I was asked to participate in a training simulation. The training session lasted about eight hours and involved mission managers, flight controllers and crew. As a participating crewmember, I was stationed in a high-fidelity simulator of the International Space Station in a building at the NASA Johnson Space Center in Houston. The Mission Control personnel (i.e. ground team) were in a replica Mission Control Centre elsewhere on the JSC campus.

This was my first ISS simulation and I was excited. I had been asked at the last moment to participate since there were no experienced astronauts available on that day. Even though I had not yet completed all my training on Station systems, the experience would be useful.

The first seven hours of the simulation were relatively benign. The instructors who had written the script had introduced several malfunctions into the sim. The flight control team and I had successfully managed them all. The standard practice for these simulations, however, is that the lead instructor wraps up the day by introducing a major contingency situation (e.g. fire, depressurization) into the last hour of the sim. These situations are designed to stress the flight controllers and crew, and evaluate our ability to work together as a team to solve major problems.

Toward the end of this day, the Capcom at Mission Control called me over the space-to-ground radio communication loop. I sensed urgency in his voice.

“Bob, we have detected a leak in the cooling loop of the port-side thermal control system. This means that we will soon lose cooling for half of the Station systems and payloads. We need you to immediately implement the procedures in the red emergency book to shut down the portside cooling loop and reconfigure the starboard loop to provide cooling for all the high-priority equipment.”

“Copy all,” I responded.

“Be aware,” the Capcom continued, “that in one minute, we will begin a 20-minute loss-of-communication with the Station. The ground will be unable to help you with the procedures. You will be on your own. Any questions?”

“No questions,” I responded. “I’m working the procedures.”

“We’ll talk to you again in 20 minutes. Good luck,” responded the Capcom as radio and data communication with the Station went dead.

A complex Thermal Control System keeps ISS hardware cool and astronauts comfortable in our orbiting home.

Loss of cooling for half of the Station is a serious failure. Without cooling, equipment will soon overheat and fail. Fortunately, I thought, thermal control was one Station system for which I had recently completed training. I felt confident to deal with this situation.

My task was to quickly shut down the leaking loop and transfer the equipment (e.g. computers, furnaces) that had been cooled by that loop to the good loop on the starboard side. The procedures were long, convoluted and nested within each other. A crewmember had to be sharp to properly execute them.

I got down to work. I shut down the starboard loop and reconfigured the port loop. As I got further and further into the procedures, I heard warning tones on the overhead speaker and I noticed several messages on the caution and warning panel. I got a sick feeling in my stomach that I was doing something wrong.

Twenty minutes later when we regained communication with Mission Control, my greatest fear was confirmed. Capcom reported to me that I had erroneously shut down the good (starboard) loop. Station equipment was overheating and failing. Aargh! We had to end the simulation – there was no point continuing.

Each astronaut training session concludes with a debrief. A debrief is a constructive lessons-learned exercise amongst the crew, flight controllers, instructors and simulation supervisor that captures what went right or wrong during the session. This particular debrief was long and thorough. My incorrect shutdown of the thermal control loop dominated the discussion. I acknowledged my error and learned a million lessons from the experience. As a team, we considered how this situation could have been prevented and how we would do things differently next time.

My dismal performance in this sim was the low point in my astronaut career. If this had been an actual day in space, we could’ve lost the entire multi-billion-dollar International Space Station because of my error. I would’ve become infamous. On the positive side, I learned lessons that benefited the rest of my career. I never repeated that mistake again. And I learned the importance of openly acknowledging and analyzing failures. Failure is a painful rite of passage on the trajectory to success.

A book entitled Canadian Failures: Stories of Building Toward Success has recently been published by Dundurn Press in partnership with Ingenium. The book is uncommon in that it examines the failures of individuals and organizations, rather than their successes. Some chapters were written by prominent Canadians who candidly reveal uncomfortable, professional failures – failures that shaped their identity. I am one of the authors and use my chapter to describe methodologies that have helped me as an astronaut to learn. Speaking candidly about failures is not easy. But by doing so, we hone our skills, improve our performance and enhance our chances for success.

The main message of the ‘Canadian Failures’ book is that success in life and business is not automatic, nor easy. It often results after a series of missteps and mistakes.

I am suspicious of individuals and organizations who speak only of their accomplishments. I have more respect for those who have the courage to speak candidly of their failures. There is much that we can learn from those who boldly went where no one had gone before … even though they did not initially succeed.

I hope that Canadian Failures will spark conversations about a national culture that views failure differently – as something to be acknowledged and built upon. Proceeds from sales of the book will go to the Ingenium Foundation which supports children’s programming at our three national science and technology museums.

•