In previous chapters, I have tried to show that discovery and invention depend at least partly on mental models. Similarly, ethical behavior depends on what my colleague Patricia Werhane calls moral imagination (Werhane, 1991). She uses the Challenger case as an example.
This shuttle was launched on January 28th, 1986, carrying a crew of seven including the first teacher into space. Seconds into the launch, the rocket booster exploded in front of a national audience.
The night before the ill-fated launch, a group of engineers at Morton Thiokol, including Roger Boisjoly, an expert on O rings, objected to launching the Challenger on the grounds that they had no data on the behavior of these rings at low temperatures, but projections from existing data suggested they would fail. Temperature at the launch was estimated to be 26 degrees Fahrenheit, and the previous lowest launch temperature had been 54.
NASA asked the engineers if they could prove the O rings would fail. They could not. Their data suggested that the probability of O ring failure increased with lower temperature, but the relationship was less than perfect. Finally, Robert Lund, the Vice President of Engineering, was asked by his immediate superior to "take off your engineering hat and put on your management hat" (Werhane, 1991, p. 606). He switched hats, the launch was approved, and all seven of the crew perished.
There have been thousands of pages of analysis of this tragedy, from all kinds of ethical, scientific and political perspectives (for example, see Davis, 1989). What I want to do is focus on how a lack of moral imagination might help explain what happened. Lund, the VP who switched hats, switched perspectives as well--from an engineer concerned with problematic test results to a manager who had a schedule to meet and a customer to satisfy. One could argue that Lund was still a virtuous person--he did not become evil, or immoral. But he engaged in what Davis (Davis, 1989) calls 'microscopic vision'; he accepted the Marhall Flight Center's definition of the problem, which was that the shuttle was safe until proven otherwise.
According to Patricia Werhane, moral imagination involves "at least four things: (1) that one disengage oneself from one's role, one's particular situation, or context; (2) that one becomes aware of the kind of scheme one has adopted and/or that is operating in a particular kind of context; (3) that one creatively envision new possibilities, possibilities for fresh ways to frame experiences and new solutions to present dilemmas; and (4) that one evaluate the old context, the scope or range of the conceptual schemes at work, and new possibilities" (Werhane, 1994).
Lund certainly failed to do (1): he switched from one role to another, but did not 'disengage' from both roles and evaluate his decision from yet another perspective--say, that of one of teacher Christa McAuliffe's students. Would such a student say go ahead and fly if there were any concerns about safety? To put it in terms of ethical theories, Lund adopted a utilitarian cost-benefit view when he put on his manager's hat, and from that standpoint it looked like the risk of damaging Thiokol's relationship with NASA outweighed the uncertainties involved in flying at low temperatures. If he had shifted to an RP perspective, the potential loss of human life would have become the primary consideration.
Point (2) is related: it involves becoming aware that one has adopted a role that works well in certain situations, but may not in the present one. We all assume roles, but we also need to be able to get distance from them, to realize we are also human beings with moral responsibilities, not just actors playing out a script.
In the 1950s, the psychologist Stanley Milgram (Milgram, 1974) decided to test a commonly-held theory about the rise of the Nazi party in Germany. Imagine you were a subject in Milgram's experiment. You responded to an ad calling for volunteers. You are met at the lab by a scientist in a white coat and another volunteer--an ordinary, middle-aged man who seems nice enough.
The scientist flips a coin. You are assigned the role of experimenter, and the other fellow is the learner. He is strapped into a chair next to you. Your job is to hold his hand down on a shock plate and press a key delivering a shock every time he gets an answer wrong. The shocks increase in voltage with every mistake. The shocks are clearly painful--the other fellow shouts "Ow!", then complains of a heart condition and demands to be let go. When you ask the experimenter, he says you must continue, and that he will assume responsibility. Eventually, the other fellow refuses to respond, then loses consciousness.
A variety of experts predicted that only a 'pathological fringe' of one or two percent of ordinary citizens would continue to shock the learner all the way through the 450 volt limit. In fact, Milgram found that 30% administered the highest level of shock when they had to force a victim's hand onto the plate, and obedience was much higher when the victim was in another room. Milgram related this to Eichmann's defense that he was just obeying orders when he murdered thousands of Jews.
Philip Zimbardo (Zimbardo, 1972) did a similar experiment where he used a coin flip to assign student volunteers to prisoner or guard roles in a fake prison. The students identified so strongly with their roles that Zimbardo had to cut the experiment short. Zimbardo himself became more of a prison administrator than a researcher. When he heard that one of the former prisoners, released because he fell apart under the stress, was organizing a break, Zimbardo worked hard to foil it--instead of realizing he had a golden opportunity to study rumor transmission.
What surprised Milgram and Zimbardo is that participants in their experiments did not simply realize they were playing a role they had little incentive to maintain, and walk out--or even call the police and demand an end to the whole experiment!
Points (3) and (4) regarding moral imagination are also connected to this issue of roles. In order to be able to envision new possibilities, one has to be aware that one is operating within a context, or view. Before Kepler, planets moved in circles. That was not a view--that was the way the world was. But Kepler turned this fact into a 'conceptual scheme' or mental model--it became only one way of looking at the universe. Other views became possible. Similarly, once you realize that you are acting a role, you can imagine what would happen if you stepped out of it, and behave differently.
In 1970, Ford introduced the Pinto, a compact that had been designed in record time with a 'limits of 2000' constraint: it could exceed neither 2000 pounds in weight or $2000 in cost. Because of the speed of design, a potential problem with the fuel tank was identified only after the design was frozen: the tank ruptured when hit from the rear at a relatively low speed (about 30 mph). It would have cost $11 a vehicle to fix this problem. Ford used the National Highway Traffic Association's figure for the cost to society of each traffic fatality ($200,000) and did a utilitarian, cost-benefit analysis which showed that it would be more expensive to fix the car than to live with the possibility of over a hundred fatalities. In this case, Ford should have exercised moral imagination and considered the same problem from a rights perspective.
Dennis Gioia was a relatively new employee at Ford in charge of the recalling of defective automobiles. When exposed to evidence that a number of Pintos had exploded, including "graphic, detailed photos of the remains of a burned-out Pinto in which several people had died" (Gioia, 1992, p. 382) he did not issue a recall. Looking back on his decision, Gioia concluded that,
My own schematized knowledge influenced me to perceive recall issues in terms of the prevailing decision environment and to unconsciously overlook key features of the Pinto case, mainly because they did not fit an existing script. Although the outcomes of the [Pinto] case carry retrospectively obvious ethical overtones, the schemas driving my perception and actions precluded consideration of the issues in ethical terms because the scripts did not include ethical dimensions (Gioia, 1992, p. 385).
Schema is a term some cognitive scientists use to refer to the expectations a problem-solver brings to a situation. Kuhn's paradigm is a kind of higher-level schema that tells scientists where to look for interesting research problems, how to investigate them and what results to expect. A mental model is a kind of schema that is especially important for inventors and designers, because their expectations are embodied in imagined objects.
Gioia's schema or set of expectations led him to look for a high frequency of incidents or a clear cause. At this time, there was no clear cause (Gioia was unaware of the test analysis that showed a problem with the fuel tank design) and no pattern of incidents--just occasional flaming crashes. Part of Gioia's schema was the assumption that all small cars are more prone to serious crashes of this sort and that people accept risks when they drive--'safety doesn't sell'. Gioia was also experiencing cognitive overload--he had lots of cases and reports to attend to. So to be noticed, a potential problem had to fit a pattern predicted by his schema.
One outcome of a schema is a script that dictates how one ought to act in certain situations. For example, most of us have scripts for restaurants that include waiting to be seated, getting a menu, ordering, and paying a bill (Schank & Abelson, 1977). After seeing one particularly gruesome wreck, Gioia activated one of his standard scripts and called for a preliminary review of the Pinto case. He agreed with the unanimous decision to leave it on the market. In this case, Gioia knew that the Pinto was a popular, best-selling auto and that other sub-compacts had similar accidents; therefore, it was easy for him to engage in confirmation bias and dismiss evidence that would have violated his schema and activated his recall script. Had he stepped back and realized that he was making assumptions consistent with a certain role, he might have seen able to imagine and evaluate other possible responses to the initial, anecdotal evidence that there were problems with the Pinto.
It is too much to ask Bob Lund or Dennis Gioia or anyone to go through all four steps in moral imagination when an immediate decision is called for. One must already be able to do this kind of moral reasoning, which calls for special training. One of the problems with taking roles too seriously is that they lead to compartmentalization. Bob Lund switched hats from engineer to manager, effectively compartmentalizing a decision that needed to be made from both perspectives.
In Dennis Gioia's case, it might have meant integrating his pre-Ford suspicious-of-corporate-America values with his role at Ford: apparently inconsistent roles which, could they have been integrated, might have helped him see how reports of flaming cars and burning teenagers would have been perceived by most people. In turn, Gioia might have been able to help Ford think of new scripts for responding to warning signals like this.
The point here is not to dump on Mr. Lund, claiming that he is responsible for the whole Challenger disaster, or criticize Mr. Gioia, who was a conscientious employee trying to identify problems serious enough to warrant a recall. It is rather to suggest how an engineer/manager with a different sort of training might have been able to step out of the context in which she or he was operating and evaluate these problem differently. Gioia calls for integrating ethical decision-making into schema and scripts through the use of "vicarious or personal experiences that interrupt tacit knowledge of 'appropriate' action so that script revision can be initiated. Training scenarios, and especially role playing, that portray expected sequences that are the interrupted to call explicit attention to ethical issues can be tagged by the perceiver as requiring attention" (Gioia, 1992, p. 388).
Gioia training scenarios resemble the cases we will be presenting in the rest of the chapter. Moral imagination should be exercised at the discovery and invention stage, not just when problems occur. Charles Perrow (Perrow, 1984) argues that complex systems like nuclear power plants are accidents waiting to happen. He might make a similar argument about the space shuttle, or the design of the Pinto. Avoiding Challenger and Pinto incidents requires more than moral imagination at the time of a problem--it requires moral imagination from the beginning, when a new technology is being created.
Niels Bohr exercised this sort of moral imagination when he foresaw the two sides of the atomic bomb. Bohr's coat of arms bore the inscription "Contraria Sunt Complementa" *(Holton, 1973). He had argued that there were two, complementary ways of looking at light; depending on how one set up an experiment, it behaved as a wave or as a particle. It could not be reduced further--physicists would have to accept light's complementary nature.
Similarly, he saw the atom bomb as both threat and opportunity: a monstrous weapon that threatened mass-destruction and an opportunity to make war--and even nation-states--obsolete (Rhodes, 1986). Bohr felt that the development of atomic weapons meant that, "We are in a completely new situation that cannot be resolved by war" (Rhodes, 1986, p. 532). Bohr tried to communicate these views in various ways to Roosevelt and Churchill, emphasizing the need for international cooperation in dealing with this threat and opportunity after the war--cooperation that would of necessity involve the Soviets. Churchill would have none of it--he thought of atomic weapons as bigger bombs, not qualitatively different weapons, and he also thought that the U.S.-British monopoly on this new technology could be preserved, at least for a time. Roosevelt was initially more sympathetic to Bohr, but was persuaded by Churchill to adopt the Prime Minister's view that this whole effort of Bohr's was subversive and dangerous.
What Bohr advocated was the kind of open sharing of information that was characteristic of science, but not of relations between nations. Once everyone clearly understood the danger represented by fission and fusion weapons, then there would have to be cooperation.
In the late 1940s and early 1950s, Muzafer Sherif and a group of colleagues, including his wife Carolyn, set-up a series of summer camps at different locations to study inter-group conflict and how to resolve it. Groups of boys arrived at the camp and were randomly assigned to two cabins, and each cabin was deliberately isolated from contact with the other. Group identities soon emerged, with groups taking names like the Rattlers and the Eagles. These groups were brought together for competitive events like Capture the Flag. Intergroup hostility rapidly emerged, leading to insults and fist-fights. Competition heightened intragroup solidarity.
On a small scale, this sounds like the evolution of international hatreds--but compressed into a period of a week, with escalating violence toward the end. How to bring the groups back together?
The Sherifs first tried a common enemy--a sports event in which both groups would have to pool their best players in order to defeat an outside group. The two groups cooperated long enough to beat the other team, but then intergroup hostilities resumed. The threat of a common enemy produced only temporary cooperation--witness the post-war conflict between the Soviets and their World War II allies.
Then, at another camp, the Sherifs tried the equivalent of desegregation. The groups were brought together for a series of pleasant activities--better meals, movies, fireworks. No cooperation was required--just occupying the same space as equals. Unfortunately, these occasions merely served as opportunities for further insults and fights.
Finally, at still another camp, the Sherifs tried using superordinate goals: "problem situations in which goals compelling and appealing to each group could be attained only through the efforts and resources of both groups" (Sherif, 1976, p. 133). First, the staff simulated a failure in the water supply. The boys had to organize themselves to inspect over a mile of pipe, then fix a problem at the reservoir that demanded teamwork. When they returned, the old hostilities resumed.
Second, the boys went in separate trucks on a camping trip. But the truck assigned to get their lunches broke down. (This simulated failure required great skill on the part of the driver). The boys had to pull it up a hill, and hit on the idea of playing tug of war against the truck. Afterwards, they exchanged mutual congratulations and ate together. These sorts of experiences were repeated until mutual sharing was the rule, rather than the exception.
Bohr's moral imagination was to see in nuclear weapons a superordinate goal--a threat so huge it would mandate cooperation. Despite many close calls--including the Cuban missile crisis--the Soviet and American superpowers avoided another world war and instead conducted a ruinous arms race while battling through surrogates. In the end, the Soviet empire crumbled. Nuclear weapons are now possessed by more countries than ever and remain a significant threat. A nuclear war could still break out between countries like India and Pakistan.
What constrains nations is the complementary of this weapon--it promises overwhelming, indiscriminate slaughter against which there is currently no defense and therefore is useless, if the other side possesses it, too. Perhaps Dr. Frankenstein's biggest mistake was to reject his creation, to run from it in horror, thereby helping to turn it into a monster. Bohr did not shrink from the bomb; he embraced its contradictions, and tried to help others see that it had changed everything.
A recent example of this kind of moral imagination is provided by a former Commander-in-Chief of the Strategic Air Command, General George Lee Butler, who announced two days after assuming command in 1990 that with the end of the cold war, SAC's mission was complete--it was time "to think in terms of less rather than more." A series of small epiphanies led him to this view. One was looking closely at the complex, mathematical formulae the Strategic Air Command used to decide on targets. Nuclear warheads were aimed not just at Moscow and missile silos but also at bridges, highways and railroads. There was no real consideration of the enormous 'collateral damage' that would occur--the fact that radioactive fallout, firestorms and the direct effects of the explosions would kill millions. Butler realized it would make just as much sense to aim warheads at evenly-spaced points on the Soviet map, and eliminate the huge planning infrastructure. Butler cut the target list to 3000, and worked to down-size SAC (Smith, 1997).
A trip to the Soviet Union in 1988 also influcence his views. He had seen the May Day parades on television, knew all about the Soviet juggernaut. But his plane landed on a runway pockmarked with holdes and broken lights. his motorcade bumped over ragged roads and passed crumbling buildings. The May Day parades were the facade of a Potemkin village.
After retiring, Butler called for the elimination of nuclear weapons--not a unilateral, sudden disarmament, but a careful, negotiated reduction in global nuclear weapons, with the United States taking the lead. Now in retirement, he continues to worry about the same issue as Bohr--will governments remember that nuclear weapons threaten such slaughter that they are in effect useless?
In the words of my friend, Jonathan Schell, we face the dismal prospect that The Cold War was not the apogee of the age of nuclear weapons, to be succeeded by an age of nuclear disarmament. Instead, it may well prove to have simply been a period of initiation, in which not only Americans and Russians, but Indians and Pakistanis, Israelis and Iraqis, were adapting to the horror of threatening the deaths of millions of people, were learning the think about the unthinkable. If this is so, will history judge that the Cold War proved only a sort of modern-day Trojan Horse, whereby nuclear weapons were smuggled into the life of the world, made an acceptable part of the way he world works? Surely not, surely we still comprehend that to threaten the deaths of tens of hundreds of millions of people presages an atrocity beyond anything in the record of mankind? Or have we, in a silent and incomprehensible moral revolution, come to regard such threats as ordinary--as normal and proper policy for any self-respecting nation?
This cannot be the moral legacy of the Cold War. And it is our responsibility to ensure that it will not be (Butler, 1997, p. C2).
This page was last edited: Wednesday, July 14, 1999