Disasters are always studied in retrospect. We will not have an experimental science of the subject any time soon. Disasters range from the personal - your wife tells you she is leaving you for the postman - to the global - your country invades the wrong nation.
Disasters, of course, are closely linked to self-deception. There is nothing like being unconscious of reality to make it intrude upon you in unexpected and painful ways.
Let's look at the tragedy of the 9/11 attacks. It had many fathers, but few have been as consistent in this role as the airlines, at least in preventing the aircraft takeovers on which the disaster was based.
Their actions were typical of US industrial policy: any proposed safety change came with an immediate threat of bankruptcy. In previous decades, the automobile industry claimed that seat belts would bankrupt it, followed by airbags, then child-safety door latches, and what not.
The airlines' lobbying organisation, the Air Transport Association, has a long and distinguished record of opposing almost all improvements in security, especially if the airlines have to pay for them.
Between 1996 and 2000, the association spent $70m opposing a variety of sensible (and inexpensive) measures, such as matching passengers with bags (routine in Europe at the time) or improving security checks for airline workers. It opposed reinforced cabin doors and even the presence of occasional marshals (because they would occupy non-paying seats). It seems likely that much of this was done "in good conscience" - that is, the lobbyists and airline executives convinced themselves that safety was not being compromised to any measurable degree, otherwise they would have had to live with the knowledge that they were willing to kill other people in the pursuit of profit. From an outsider's viewpoint this is, of course, exactly what they were doing. The key fact is that there is an economic incentive to obscure the truth from others - and simultaneously from yourself.
Only four years after 9/11, the airlines were loudly protesting legislation that would increase a federal security fee from $2.50 to $5.50, despite numerous surveys showing that people would happily pay $3 more per flight to enhance security. Here the airlines did not pay directly, but feared only the indirect adverse effects of this trivial price increase. Note that corporate titans appear to increase their own chances of death slightly to hoard money, but with the increasing use of corporate jets, even this is not certain.
Regarding the specific event of 2001, although the United States already had a general history of inattention to safety, the George W Bush administration even more dramatically dropped the ball in the months leading up to the attacks: first by downgrading Richard Clarke, the internal authority on possible terrorist attacks, including specifically those from Osama Bin Laden. The administration stated it was interested in a more aggressive approach than merely "swatting at flies" (Bin Laden here being the fly).
Bush joked about the August 2001 memo saying that Bin Laden was planning an attack within the United States. Indeed, he denigrated the CIA officer who had relentlessly pressed (amid code-red terrorist chatter) to give the president the briefing at his Texas home. "All right," Bush said when the man finished. "You've covered your ass now," as indeed he had, although Bush left his own exposed.
His administration had a particular interest in focusing only on the enemy, not on any kind of missed signals or failure to exercise caution. Absence of self-criticism converts attention from defence to offence.