also a lot of product names associated with these scams, and are tainted. consumers need to do their due diligence in this area. also, be ware of where you're purchasing the product. the internet is like the wild wild west. be careful and do your searching widely. >> we heard a lot about the various challenges of law enforcement to combat scamsment what are some of the difficulties or obstacles you or your agents have seen in combating. >> it's been mentioned panel

after panel this morning, it's a global problem. so anytime we're trying to conduct a criminal investigation, where the criminals are located overseas, it makes it that much more difficult to run the investigation. we rely on collaborating with other law enforcement agencies, and we rely on collaborating with other law enforcement and regulatory components of foreign governments in addition, fda is a public health and state enforcement engine, and to conduct a criminal investigation involving a public health and safety matter, where the public needs to be warned, it makes it very difficult at times to conduct a criminal investigation, because once an alert goes out, notifying the public of another tainted supplement, the hat is tipped. but it's the right thing to do.

>> there may be viewers out there, consumers, saying to themselves, look, i have a disease that i'm suffering from, and i need something that is going to cure me. they've gone to their physician, gone to different sources, and they are lacking something that's going to cure their ailment, and they want to say to -- they say to themselves, i'm not going to listen to some government bureaucrat telling me i shouldn't take this product because it might cure my otherwise incurable disease. why shouldn't i go on the internet? why shouldn't i go someplace else and look for this cure that some government bureaucrat is telling me i shouldn't be taking. >> that's a very difficult question. i've been a federal bureaucrat and a private sector as a healthcare professional. the only thing i can say to that is consumers need to do their research, and at the end of the day, you need to trust your physician and your pharmacist.

>> thank you very much, jeff. i'd like to turn now to our third speaker, patrick is a trial attorney in the consumer protection branch of the department of justice. he has been in that position since september 2009. prior to that he was clerk on the u.s. court of appeals for the sect circuit, and he is going to address specific subjects related to dietary supplements, the subject of manufacturing practices and he's going to talk about it globally and also through the lens of a particular case he handled. >> thanks. i'm a trial attorney in the consumer protection branch of the department of justice and we handle among other things cases on behalf of fda and that includes dietary supplement cases. first i'd like to talk about the rules applicable to dietary supplements and manufacturing practices. up until around a few years ago,

the only rules that were applicable were the same rules applicable to food production facilities and those were the general food good moving practices regulation. in the harder part of 2000s they needed to ensure the safety and purity and quality of dietary supplement at every step throughout the manufacturing process, and those new rules were made official in 2007, and they went into effect in 2008 and 2009 and 2010, depending on the size of the manufacturer. now, compliance with nose new rules has been by and large a very difficult challenge for many manufacturers. fda allotted in 2012 inspections that had been conducted over the past four years, several hundred. fda found in about half of the inspections they had identified

manufacturing practice issues with dietary supplements. so, the fda can't inspect even half of the type of facilities that are out there. so, fda has to inspect two companies, food facilities, cosmetic manufacturers, so it's impossible for the fda to inspect all the dietary supplement manufacturers out there, and according to the fda we have half of those supplement manufacturers not even get though baseline of being able to ensure their products are what they say they and are have been manufactured under the proper conditions. so more specifically -- we have been talking generally but i want to give a specific example of when things can go wrong, and i'm going to talk about a case i litigated, and i brought a little show and tell today. so, speaking of products not being manufactured under the proper conditions i'm going to

talk tot quality laboratory a firm in new jersey i litigated. the quality formulation laboratories was not aa fly by night company. a large dietary supplement manufacturer in the newark area and had been around 20 years. it made a lot of products, but its flagship products were workout products and if you walked by a gnc or the dietary supplement section of any food store you probably see these, and these are protein powders people make shakes out of in order to get extra operate teen. qfl made milk protein and egg protein powders that were sold all throughout the country. most of the products were sold as third-party brand. they were labeled with other company's brands so you would have no idea where they were

manufactured. qfl also sold products under it own brand, american sports knew. they said qfl was not a fly by night company, and at the time the loft was filed, qfl had a distribution deal with costco in which products were going to every or almost every costco warehouse in the united states that was extraordinarily lucrative, worth $10 million of product every year. so, i'm going to tell you the story but i don't mean to paint every dietary supplement company with this brush. there are plenty of do it supplement companies trying to do the right thing and i'm sure there are many facilities that are perfectly clean, but qfl was on the fda's radar in 2007 when there 'twas a consumer in the pacific northwest, a young man, a college student, and he had a

life-long severe milk allergy and he knew he had it, and his parents bought him a tub of protein powder that was supposed to be egg proceed teen powder. it was labeled with a different company's brand so they had no way of knowing it was qfl. he took a half scoop of the egg protein powder, and within a very short time his heart stopped, he throat had swollen shuts and he was lucky his room mates were there -- administered an epipen and got him to a hospital where he was stabilized. his parents complained to fda, and told them about the situation. fda found that the product had been manufactured at qfl's facilities and inspected the facilities. in early 2007 when they conducted the first insection, the investigators found a number of issues that were of some

concern to the agency. first, qfl did not have proper controls over cross contamination between products, and what that means is that qfl would produce an allergy-containing product, followed by a product that was supposed to not contain that allergen without making sure none of the made its way into the other. and one would be because of inadequate cleaning, and powder blowing around and residue all over the plant. there was another problem which was indiana kuwait batch formulation voids was unclear what ingredients and how much of what ingredients had gone into what product. there will also labeling issues. the fda noticed some labels did not reflect what was in the

product, and that's a few of them. but fda as it always does tried to work with the company to bring the company into compliance. so, the agency pursued with the firm through letters and other submissions ways to get the firearm into compliance. so fda went back in the beginning of 2008 and the issues of cross-contamination had not been recognized and the fda surmised maybe the issued of cross-contamination were the reason why the consumer was injured in the northwest. so the fda continued to try to work with the company, but when fda went back to the company in december of 2008, the inspectors saw something that was truly extraordinary. this is when things win off the rail. they found an active roach infestation that had not been contained in any way, shape or form. rats running through the facility. there were bags of raw ingredients, such as the protein

powder, that had been gnawed through by rodents and urinated on by rodents and rodent feces in the ingredient. and when the investigators went into the blending room there was still residue all over the place and this time there were hundreds of rodents in the residue that was on the blending platform, and most troublingly, the investigators found a half of a rat that had been cut in half sitting next to the scoop used to scoop the powder into the blenders. they knew it was half of the rat because they saul the other half a short distance away. that was fresh kill. also, there was bright red blood coming out of it, and obviously that was a severe concern to the inspectors. so, -- the other thing is that the inspection -- there was some indication that qfl product had

been intentionally misbrand and intentionally but milk protein in the egg protein product because milk protein is cheaper. so an attorney in my office filed a lawsuit against qfl on july 1, 2009, seeking to shut the company down, and that lawsuit was filed on fda's behalf. as soon as the lawsuit was announced to its credit, costco dropped the products immediately and returned all of it to the form and that was about -- at least a million dollars worth of products. so, qfl chose to litigate the case for nine months. even with the evidence we had. but eventually the firm agreed to a consent degree deck career. -- deck career requires firms to do a shutdown. upon entry of the decree by the

court, and it provides a path forward with which the firm can submit sanitation plans, hire experts, get a new fda inspect, put their plan into effect, and they have to get written authorization from fda to resume operations. so, that's the kind of consent decree that qfl agreed to and now we're up to march o2010. in that year the court entered the con seen the decree, and in most case that would end the case, but probably as you can tell by me seating here that did not end the case. the consent agreement, forbid them from receiving, manufacturing, preparing, packing, labeling, and distributing food at their plant located in patterson, new jersey, or any other location or any new location at which they operate. but as it says, the case is just getting started. so, shortly after that consent decree was entered.

fda got a tip that qfl had not shut down but moved its operation somewhere else and they were every day instructing employees to rave at the patter union plant and were putting employees in a veterans driving them a certain distance to an undisclosed location, and from there they were putting powder into tubs and sending it out to consumers. so, fda's office of criminal investigations, the new york field office, conducted a criminal search warrant of the new location in may 2010. the new location was in concord, new york, which 30 miles north. when they got there they found qfl employees sitting around tables putting powered into tubs. they saw a large operation that was basically hand-packingpackid there was powder everywhere ask the products were being produced

there at the exact same time the search warrant was been excused, the inspectors win to the qfl facility in new jersey to conduct an inspection and there they asked the owner, who they found there, to -- for record relating to qfl. the owner of the firm and his son said there were no records. the investigator said we know about the other plant. something is going on there, and then there were records, 100 cases of records and in addition, the inspectors found some pregnancy vitamins on their way to the united kingdom, ready for shipment, and also found some other products that had been shipped to that facility during the time that the firm was supposed to be doing absolutely nothing at all. so, to make an extraordinarily long story short in july of 2010 we petitioned the court in new jersey to hold the companies and

the owner of the companies and those involved in the secret operation in criminal contempt of the conceit agreement. the court accepted our position and started proceedings against the company and individuals. there will a number of other things that happen e happened during the litigation of the case but one of them reef lates to this tub right here in january of 2011, which is going to haunt people that were supposed to go to trial, we were calling around for various witnesses to see if we could find some other companies that got products from qfl at the time they weren't supposed to be operating. we called a company here, performance food sneer in allentown, pennsylvania, fairly large distributor, and we asked them if they had got 'products from qfl during the time period they were supposed to be shut down. and they said we're supposed to pick up a shipment tomorrow. and that was interesting. and the agents conducted an undercover operation where we put an oci agent on the

performance food center's truck, and videotaped them distributing pallets of products out of the paterson, new jersey, warehouse, and this very tub, exhibit 805, is one of the tibet distributed on that truck. so, ultimately the defendants went to trial in may of 2011 in trenton, new jersey, convict on all charges of criminal contempt. the companies were find a total of a million dollars. the owner got a 40-month prison sentence. his two sons received 34-month prison sentences and they're in federal prison right now. when they were sentenced in november 2011, the judge said this criminal contempt was unique in its persistent and scope, and i could not disagree. i do have to say they appealed their conviction and in january of 2013, the third circuit court of appeals in philadelphia affirmed their convictions and remanded the two son's sentences

back to district court for additional fact finding. so we're still litigating the case. it's like live they will ask the supreme court to review their conviction. >> patrick, for those in the audience and those who bay be watching on television, many people may be frankly horrified, and should be horrified by the things you said. when consumers are researching dietary supplement, trying to figure out if they're safe according to best manufacturing processes, what can they do other than maybe when they open the turk looking for half rat on top, what other steps can they do to find out whether the dietary supplements they're taking have been manufactured as they should be? >> well, i'm going to echo the other speaker here. one of most important things a person can do is research online possible other complaints against the company there are making the dietary supplements

they're considering to ingest. the fda's web site is a very important resource in terms of companies that fda has taken regulatory action against. fda has a large database of warning letters sent out to companies identifying manufacturing violation so it's rare for a case to get through our office it's a long boy, the fda issues many more warning letters that advise companies they're engaged in practices that are potentially unsafe. >> i want to thank the panelists very much. very educational for me and hopefully others as well, thank you. [applause] [inaudible conversations]

[inaudible conversations] >> thanks four hanging in there everybody, we're almost doesn't one more panel to go. so, i love the month of march. we got spring training has begun, march madness is about to start, spring is around the corner, tax season is here. well, of course, with tax season comes a lot of fraud scams and our next panel is happy to tell you all about it, and hopefully arm consumers with additional information they can watch out for to avoid becoming victims of common tax fraud schemes. we have carol, a counsel in the tax division of the department

of justice, and barbara, the act director in the global financial crime section in the internal revenue service criminal investigation division. so, thank you. why don't you begin, tell us what to watch out for. >> okay. well, i think what most taxpayers are -- should most be aware of is two different types of scams that might be perpetrated on them during tax season. the first is something that happens without their knowledge. their identity is stolen and a tax return is filed in their name and they have no idea that it happened until they either try to file a tax return or get some kind of notice from the irs that something is going on with their account, that they don't feel -- they needed to file. the second one is situations where taxpayers have untrusted their information to a professional, a paid tax preparer, and sometimes they're

taken advantage of. this is a very small minority of instances but it's something that once -- in either situation, there's a return preparer involved, it's very hard situation to try and rectify. and so we're here to try to talk about some of them. the most important one, one that received the most press of late is identity theft through a number of avenues. taxpayers, individual, private, personal identifying information, is stolen, and information is used to file tax returns. the consumer has no idea their information is compromise ordinary a tax return has been filed in their name, and this starts early. as soon as the filing season opens up in early february, and because the fraudsters in this

case, they know when we start processing returns and they have stolen information and try to get that information into our systems as soon as possible. >> robert, where are you seeing the bad guys stealing the identities the most? >> i couldn't quantify where they're seeing it the most but about anyplace you would deliver up your name, your direction and your social security number, it can be stolen. we have seen healthcare institutions, car dealerships, financial institutions, even seen it in lawyers' offices. >> so, it's very difficult and a culpersome process to prepare your own tax return and file it, but once somebody has done that for you and you have no idea how you find out about it. the irs may issue a notice saying, we got a tax return, and we have some questions on it. you may be trying to file your

own return and you'll get a notice from the irs saying we already have a return in your name. you might start getting a notice that you owe additional money on your tax return. you might get a notice from either a federal or state or local benefit agency saying, we're going to start taking your benefits because the irs told us you filed a tax return and reported x amount of income that and disqualifies you. irregardless of how you're told about it, the best thing to do is contact the irs. we have a special unit, the identity protection specialized unit, which deals solely with the theft of identities and there is an 800 number, 1-800-980-4490, where you can call and report that information. if you have been given notice by

the irs that your information has been stolen, it will give you complete instructions on how to work through the process to advise irs of the correct information that should have been on your tax return and how to, in a essence, collect your accounts with the irs. >> robert, i understand that a lot of these scammers are using prepaid cards in order to help effect wait their scheme. they're getting the fraudulent refunds and then asking the treasury deposit load those on prepaid cards. is that something you're aware of? >> yes. the refund on the prepaid card, we have seen those refunds go into bank accounts. they've even -- regards to the old fashioned method of getting a treasury check and then cash ing the treasury check.

all of those are valid ways to receive your refunds. and so the fraudsters use any of those means they can to get the cash in their hand, and really that's their whole objective, is to -- this is just another income stream for think another fraud scam they can run and get money that they really didn't earn. >> and that is just generally one of the concerns we have about mobile payment, emerging payment mechanisms, making the flow of money easier and more anonymous, and it's something we're paying attention to. so, thank you, robert. carol? >> i was just going to add, on the enforcement end, which is where the department of justice comes in, we have just had a number of prosecutions of individuals that have been convicted of stolen identity refund fraud.

there's been a stolen identity. they used it to get a tax refund or multiple tax refunds to which they're not entitled to. some of those prosecutions have received sentences ranging anywhere from a year to over ten years. identity theft as a crime has a very, very high sentence, and these individuals don't just steal in general, they don't just steal one or two i'ds and file one or two tax returns. what we are seeing and what is being vigorously prosecuted are individuals or groups of individual that do hundreds and hundreds of these. so, each taxpayer is injured to the tune of a couple of thousand dollars, which is certainly significant, but the perpetrators are getting refunds of hundreds of thousands of dollars. so, when they're prosecuted, each instance of identity theft is a -- has a sentence and those

are multiplied. >> as a matter of fact, carol -- sorry to interrupt -- identity theft carries a mandatory minimum sentence of two years, and that is very significant to have a mandatory minimum sentence in a white collar crime. >> right. and so a lot of these individuals having schemes going, they're -- the months add up quickly when they are sentenced, and in addition, though, to criminally prosecuting people who are stealing identities and then committing tax refund crime, once the department of justice does on the civil enforcement side is try to find these people and shut them down, whether they are tax return preparers -- your regular corner tax return paper that may occasionally do a legitimate run, they're also doing ill legitimate tax refund

they can be shut down with save injunction. over the past ten years the department of justice has enjoined over 500 bad return preparers, some of whom also combined their tax preparation services with identity theft and they can be shut down. on the department of justice web site there are a number of press releases. that's a good resource if you think your return preparer is doing something shady or you're not sure you want to see if they're allowed to prepare returns and you can look on the department of justice web site and look for tax division press releases, injunctions, and you can into whether the person is under an injunction to stop doing that. so we're sort of hitting this from two tasks in terms of enforcement. we have the prosecutions, which have hefty criminal sentences, and a restitution component, but we also have stopping the process through civil process.

but while -- let me step on your toes -- what can a taxpayer do once their identity is stolen, and they're entitled to a refund but their refund has been paid out. ...

you're having financial hardship you can't get around because of that an additional resource in the irs is something you can go to the taxpayer advocate service. kind of an on butsds man -- ombudsman in the irs. that is a 877 number. 877-7774778 and they can help, kind of cut the red tape if you're having a hardship situation. but, unfortunately, because you're dealing with trying to unravel a fraud scheme and we want to make sure once we've been scammed again, because that happens, they do in fact use the same mechanisms, the fraudsters do, that the public does to

try to determine where is their refund or hey, my address is changed. they use all those same legitimate tools a taxpayer uses to try and game the system. it does take a while to sort that out unfortunately but, but, it is, there is a mechanism and it will get resolved. once you are identified as a victim of identity theft the irs is also assigning you something called an identity pin, or ip pin and what that does, now irs systems in order to file a tax return in the future, you have a pin number every year the irs will notify you in advance of filing season so that you will confirm not only the information that is on the tax return but you will confirm the piece of information the irs already sent you, regarding like a bank atm pin, if you will. >> before we wrap up, you want to just mention "the

dirty dozen", pick out a couple that you think ought to be mentioned and at least reference it? >> sure, sure. each year the irs puts out a dozen tax schemes that the public should be aware of. these are schemes generally perpetrated on people trying to comply with the tax system and enterprising fraudsters figure out how to prey on the taxpayer. we talked a little bit about return preparers and basically what you want to know there is, no matter who you pay to help prepare your tax returns, without a tax return it is taxpayers responsibility. always get a copy of the return. sign only a completed return. go to a return preparer that's reputable. you can check with local consumer agencies or if there are professionals like a cpa or tax attorney you

can check with the state associations. when you pay a tax return preparer their fee should be based on the complexity of the return, not a percentage of the refund. and in any circumstance the return preparer, your refund should not be coming from the return preparer, it should be coming from the irs. >> let me just jump in here. my favorite website of as a doj attorney representing the internal revenue service, is www.irs.gov. it is number one on the book mark. all information on i.d. theft and protection specialization unit and tips on protecting the taxpayer as well as all 12 dirty dozen schemes are on the website. i commend it to you for your perusal. some of the other schemes

not necessarily perpetrated on consumers but consumers can get sucked into are schemes to hide your money offshore. to create a, yourself as a church. to file zero returns or basically tax defier constitutional kind of arguments. other things for, you know, free money. those are things to watch out for. you know the old addage, if it seems too good to be true, follow that. and a lot of those are detailed on that irs dirty dozen page. >> i would like to identify one more. it is a common scheme that's not focused solely on the irs but there is something called phishing schemes where the fraudsters try to contact you through social media and get your personal information because they're portraying themselves as the irs the irs does not initiate taxpayer contact by

e-mail. and if you get an e-mail from something that looks like it is from the irs and you have not been in contact with them through the mails previously, again, you can go to irs.gov, there is an e-mail address where you can forward that e-mail and investigators will look into that. a lot of times those schemes are not, are becoming actually from offshore. once your information is out there, even that information is valuable to fraudsters to sell amongst themselves but the big key that the irs does not initiate contact with taxpayers through e-mails. >> all right. well, thank you both, very much. i appreciate it. >> you're welcome. thank you. >> [applause] >> and thank you all again for attending. that wraps up our second annual consumer protection summit. thank you to georgetown university. natalia, thank you so much for all the hard work you did putting this event together. i want to make one last plug for a website.

i love the irs website. my favorite is the tax force's website, stop fraud.gov. irs is close second. thank you you all very much and hope to see you next year. thank you. [inaudible conversations] >> if you missed any of the consumer fraud discussions from earlier the panel discussions, they're available in our video library at c-span.org.

we have more live coverage coming up today here on c-span2. george washington university hosting an all-day conference on the 10th anniversary of the space shuttle columbia disaster. they're back at 1:00. we'll pick up live coverage on c-span2. the event features nasa officials that investigated the explosion and made testing and safety procedures on future missions. this afternoon over on c-span, secretary of state john kerry and first lady michelle obama honor the 2013 international women of courage award recipients. they recognize women around the world that have shown courage and leadership in advocating for women's rights. that is at 3:00 p.m. on on c-span. earlier today in the washington the labor department releasing the jobs numbers. unemployment rate dropping from 7.9% to 7.7% in february and the economy adding 236,000 jobs. this is the lowest level of unemployment in four years. reaction from washington, some reaction.

senator harry reid, the majority leader said today's numbers show our economy is growing and is is toed to grow even faster in the months to come. on the other side house speaker john boehner said in part, any job creation is positive news but the fact is, unemmoment in america is still way above the levels the obama white house projected when the trillion dollar stimulus spending bill was enacted. you can find a link by the way to that february report on our website, c-span.org. and news from rome today. catholic cardinals set tuesday as the start date for the conclave to elect the next pope. the vatican press office said the decision was made this afternoon in the vote of the college of cardinals. as we mentioned, the 10th anniversary of the space shuttle discovery, shuttle disaster discussion is underway. they resume at 1:00. we'll have that live for you here on c-span2. until then we'll show you a discussion on the latest u.n. sanctions against north korea from this morning a's "washington

journal". >> host: for a look at some of the latest developments in the world of u.s. foreign policy we're joined by blake ounce sell, the managing editor of "foreign policy" magazine. we'll start in northeast asia where north korea this week has threatened to undertake a preemptive nuke clear strike on the united states and south korea. what prompted that threat and how seriously does the united states need to take it? >> guest: well, i have a 1-year-old kid who is just about to have his first birthday and i look at this sort of north korea flinging its food basically. you know, when my kid is eating dinner and done with something or doesn't like it, he drops it off the tray and north korea is basically saying, we don't like these new sanctions that the u.n. just passed and we're going to though our food basically. we want you to pay attention to us. we'll ratchet up threats no matter how realistic they are. for instance, today they

said or yesterday they said they were going to launch a preemptive strike against the united states and its interests. well, you know, as far as we know north korea doesn't have the capability to do that. they can't put miniaturized nuclear weapons on missiles. so they're calling for attention. they want the united states to come to the table and offer some concessions and they want to embarass the new south korean government that just took power. >> host: you talk about the u.n. sanctions that have been levied against north korea. want to play a little bit, this is susan rice, the united states ambassador to the u.n., announcing these sanctions and explaining a little bit about what they are. >> first, resolution 2094 imposes tough new financial sanctions. when north korea tries to move money to pay for its nuclear and ballistic missile programs, countries must now block those

transfers, even if the money is being carried in suitcases full of bulk cash. likewise north korean banks will find it much harder to launder money for the dprk nuclear programs this resolution imposes new travel restrictions. if a north korean agent is caught making arms deals or selling nuclear technology, countries will be expired to xcell that agent. countries must prevent the travel of people working for designated companies involved in the nuclear and missile programs. >> host: blake hounshell of "foreign policy" magazine. we hear a lot about sanctions over the years with north korea. what is different about these and will they work? >> marcus nolan, who is a north korea watcher with the peterson institute here in

washington said yesterday that he doesn't think these sanctions are going to have much of an effect at all on north korea's behavior. north korea is already one of the most sanctioned countries in the world, if not the most sanctioned. i for one am surprised that some of these loopholes weren't closed already. north korea is extremely isolated. it has been able to, you know, it is kind of an incredible story it has been able to get this far given how closely the country is watched. so i don't expect to see much of an effect on north korea's behavior or its nuclear program. you know, a lot of north korea watchers are now saying, look, this whole strategy of ignoring north korea and then adding new sanctions every time it issues a provocation just isn't working and the obama administration needs to come up with some kind of new diplomatic strategy, whether working more closely with china or whether it is coming to the table and talking to north korea and

telling the north koreans directly what we think of them and what we're prepared to do. >> host: we're taking your calls this morning on a variety of foreign policy issues. we'll get into this segment of the "washington journal" from north korea to venezuela, to the capture of the son-in-law of usama bin laden. you can give us a call. democratic line is 20-5853-880. republicans 202-858-2881. independents, 585-238-88882. staying on north korea, some optimism out there in reports that china might be coming around to help with these, this round of sanctions. here is the story from "usa today", china crucial in sanctioning north korea. talk a little bit about china's role. >> that was very interesting, chris hill that used to be the u.s. envoy to the six-party nuclear talks said that, you shouldn't pay attention to the words of

this resolution but rather the music and the music is that china is finally seems to be coming on board with a strategy of pressure. that's knew but what i think, china's bottom line is that china doesn't want to see unrest on the korean peninsula. they have got worries about the border of north korea. about contraband activities going on. about refugees streaming across the border. their bottom line they don't want the north korean state to collapse because they're worried about a pro-american, unified correa. so, you know, they will do what they have to do to stay in the international community's good is graces and pressure north korea. but the bottom line they will not let this country collapse. >> host: and the relationship between china and north korea's, i he is g not new leader anymore, kim jong-un. he has been in for what

about a year now? >> yeah. after his father died. apparently the relationship between china and north korea has been described as like lips and teeth. that is the phrase the chinese use but the chinese for years have been trying to get the north koreans to behave basically. they brought kim jong-il on numerous trips to parts of china. they show him factories. tell him, this is how you need to reform your economy and i think the chinese perspective the north koreans haven't listened an they're condition standly issue these provocations which frankly isn't good for business and the business of china right now is business. >> host: taking your calls on this segment on north korea and a variety of other foreign policy issues. cass up first from grand junction colorado, on the democratic line. >> caller: thank you for taking my call. i've been an obama fan for a very long time and i voted for him but i'm very pleased

what he did. no one does everything exactly right, when you're fighting your own country as well as having to fight foreign wars, it gets to be pretty hard. i think he handled it in a very courageous and very gentlemanly way. and what he has, i have never seen a speech being given when he was on his way being in office, where they showed a picture of a man with a sidearm plainly strapped around his waste just a few yards away from the president. i don't believe we would have seen that on any other president ever. >> host: how would you rate the president's handling of north korea, a historically very tricky nation to deal with on the international level? >> caller: well, i think he is doing the best he can considering that he is being, he is being firsted into making decisions and he has got a lot of people telling

him what to do and he has a tremendous bag of rocks on his back and i think that he is doing the best that he can do under the circumstances. >> host: blake hounshell, "foreign policy" magazine, how is the about the being rated on the international level, specifically to north korea? >> guest: well i think you have to look at the results. when it cops to north korea the results of this administration's policy of what it calls strategic patience, just aren't good. north korea just launched another nuclear test. their program seems to be improving. they have threatened to conduct missile tests which would be another provocative action. you know, north korea is elf did he the land of no good options. i'm not saying there are easy choices here but if you look at the results of what this policy basically ignoring north korea for the last four years has wrought,

it is not impressive and i think that it is time for some more creative, you know, innovative approaches. we have on our website, foreign policy.com, an article now by a former north korea negotiator who says, you know what? why not appoint dennis rodman the special envoy to north korea? maybe he can break to these guys to get something accomplished. >> host: paul up next from enterprise, alabama, on the independent line. good morning, paul. you're on with blake hounshell. >> caller: good morning, gentlemen. i would like to say that everybody knows north korea is in no position to be throwing its weight around in this manner and that it is, i think biggest negative motivator in the world. everybody also knows that they are a proxy of china. thank goodness for china agreeing to the sanctions. but more importantly for me, is the, as governments spar,

the citizens suffer and, i would like to know, do you have any specific means that we can actually reach the sit currency -- sit -- citizenry of north korea. everybody is fed propaganda there and here and just, builds animosity that doesn't need to exist? thank you very much. >> host: that's a great question. you know, as closed as society of a society north korea is, over the last decade or so it has opened up just a little bit. the most recent thing that happened foreigners are now allowed to have 3g cell phone coverage. hundreds of thousands of north koreans have cell phones. they're limited and i'm sure there being monitored very closely but there are a lot of north koreans, especially

that live along the border area with china can pick up, you know, reception and 3g signals from china. there is a lot of smuggling, cross-border trade that goes on. contraband dvd's from south korea, that sort of thing. there are tons of north korean refugees who live in northeast china who go back and forth smuggling. that's an opportunity for the united states or the international community to start getting, you know, news filtering into north korea so that people inside that very closed, very repressed society can know what is going on in their own country and the reality of the world. >> host: question from amber on twitter. the american media is under the assumption that north korea is a dictatorship. the truth is it is run by a cadre of seniors and not the boy. talk a little bit about north korean leadership these days. >> guest: well, you know, we don't really know to any

great level of detail how exactly north korea is run. we're kind of reading tea leaves. we're looking at goat entrails. we're, you know, coffee grinds but from what we do know, kim jong-un's you uncle and his aunt, his aunt being kim jong-il's sister, are still very powerful and what i think has happened over the last year or so since kim jong-un came to power he tried to become not just a figurehead but also someone who's powerful in his own right. some analysts will say that these kind of provocative actions that the north koreans do, like the nuclear tests, like threatening the united states and south korea, those are designed it consolidate kim jong-un's control over the military and i think that's probably the right analysis. >> host: duane up next from jamaica, new york, on the

democratic line. good morning, duane. you're on with blake hounshell. >> caller: how are you? >> guest: good. >> caller: i'm watching and following on the north korea issue for years. my father was actually in the korean war, and just the stories, basically a war with that country would be catastrophic. is, are we prepared to deal with the worst-case scenario? i mean we're saying we should try other ways of dealing with north korea, short of war but we, as you say we have used, we imposed all the sanctions that we can possibly imagine. so i mean, bottom line, are we prepared to take that final step in dealing with north korea? and if not, then shouldn't we really kind of go for one-on-one, you know, meeting with this country and to try to resolve the matter diplomatically and stop pretending we're ready to really take the final

step? >> guest: you know that's a great question. the white house said yesterday after this north korean threat that the united states was quote, very capable of dealing with anything that the north koreans throw at us. i heard one person make the, sort of macabre joke, that probably there is one, u.s. submarine off the quote of north korea somewhere that could wipe out pongyang with one missile. so militarily i don't think that we are, i wouldn't say, i think -- [inaudible] but if it came down to that, we would be able to handle it. of course it would be catastrophic as you say. it wouldn't be something that anyone would want or welcome but the united states is prepared for this. they have had, you know, decade to watch north korea and to prepare for anything they throw at us. the thing that people really worry about with north korea the thousands of artillery

pointed at the south korean capital seoul. north korea periodically threatens to turn seoul into quote, a sea of fire and they really mean that and they are capable of that and that is something, to me it suggests, let's try diplomacy. let's try a little harder to come up with some kind of a framework that if can not solve this problem, at least put it back into some kind of manageable box. >> host: turn now to the subject of the capture of osama bin laden's son-in-law, abu, sulaiman abu ghaith. here he is is, a spokesman for al qaeda, with his father-in-law in the picture on the front page of "the wall street journal" how big of a blow is this to al qaeda? is this more symbolic than anything else? >> guest: well, you know, i think the fact that most of us probably hadn't heard of this guy before suggests it is probably not a huge blow. the interesting thing that he is probably the most

senior al qaeda operative to be tried in the united states and that's already proving controversial. the administration says it wants to, you know, bring him to court and some folks on capitol hill said hold on a minute. let's send him to guantanamo instead. so i think the real debate is going to be over, legally what does the united states do with this guy? you know, as far as we know, he is more of a pop grandist. he is not, planning attacks. so he probably doesn't have that much to tell us, you know, about plots that are in the works or other operatives that are out there. the one really interesting thing about him is that he spent time in iran and there are these senior al qaeda level, al qaeda lead he is -- leaders who have been living in iran for years. terrorism watchers have been seeing these guys kind of filter out into different arab countries over the last few months and that's one

reason why this guy was picked up because iran apparently kicked him out. he ended up in turkey. we got wind of it and then the cia picked him up in jordan. >> host: it was a joint operation with the jordanian authorities as well? they were in on this? >> guest: well the u.s. has very close ties with jordan's intelligence services which by all accounts is probably the most professional and competent in the arab world and it looked like they, they helped us pick him up. >> host: go to brian from colonial heights, virginia, on the independent line. brian, you're on with mr. hounshell. >> caller: yes, sir. how are you gentlemen doing today? i just wanted to talk about foreign policy for a minute. i think that senator paul out of kentucky really did a great thing yesterday, standing up for all americans civil liberties, democrats, independents, republicans across the board. i think this should draw

much attention to senator paul and his father, dr. ron paul. their foreign policy needs to be reexamined. it needs to be stopped cut down by the media in general and chopping on it. they have very valid points what they talk about. they have very valid points to the problems we have today. if we don't look at the petrodollar, the thing that dictates all our foreign policy and the reason why we're in these countries, these wars will never end. soldiers wounded will never end, the welfare will never end and destruction of the family unit will never end. we need to get serious about this. only when our real liberty abouts and freedoms in this country are challenged will they then stop being so divided and we will come together as one. >> host: brian brings up senator rand paul, that filibuster that he was part of on the senate floor was mostly about domestic drone use.

what are the biggest implications for drone use on foreign soil? >> guest: well, you know, i thought that was a real interesting moment. it seems that the politics of terrorism have really changed in the united states and you saw, you know, rand paul get support not only from the republican caucus and the leadership of the republican party, but also from people on the left. people like, you know, ron wyden, democratic senator, who is a big, advocate of civil liberties. you know, i think the specific points that rand paul was raising were probably questionable. i don't think anyone really thought seriously that the united states is going to launch drone strikes against you sitting in a cafe. i think that's, a little bit of paranoia. i think john mccain, you know, aptly spoke to that but the real question about the drone policy that a lot of people have, is there a better way to go after these

guys in places like yemen, somalia, the border areas of pakistan? you know, drones have caused civilian casualtis. there's a lot of political blow back in these countries. they're not happy about it. and we're dealing with governments that haven't been transparent with their own people about what they're doing. that is part of the problem where you have in pakistan a military that on the one hand is privately telling the u.s., go ahead, drone these guys. then publicly in their own press they're saying, we don't know anything about this. this is terrible. we don't want this happening . .

and if the government makes the determination of whether they think they are safe to release into the open or whether they negotiate with their home countries to incarcerate them there. now the scary thing that has happened, particularly in yemen and saudi arabia, you have these former guantanamo detainee is that have been sent back to those countries. in yemen and there was a famous prison break a few years ago that looked very suspicious we're a bunch of guys tunneled out and it seemed like there was some kind of complicity from the government. in saudi arabia they went into this quote on quote rehabilitation program where supposedly, you know, they had been -- their - had been changed

and they renounced terrorism. but a couple of these guantanamo detainees went on to the leaders of al qaeda and the arabian peninsula which is probably the most dangerous of these al qaeda franchisees. >> we are talking with blake hounshell, foreign policy magazine editor of foreign policy magazine since 2006. and before that studied on the senate for development studies in cairo. what's the name of the center, pronounced it for me. what did you study their? >> guest: i looked at civil liberties issues in the arab world particularly egypt. it was an interesting time and i learned a lot, and is served me well when i was able to go back there for the revolution in 2011. >> up next susan from new jersey on the democratic line. good morning, susan. >> caller: good morning. how are you? >> host: good. you are on with mr. hounshell.

>> caller: i appreciate what you did with your guest as far as going to iran. president obama is doing what he can after bush screwed up the state. can you understand that? >> host: go ahead, susan. >> guest: on foreign policy issues? >> caller: absolutely. he wouldn't be elected -- i am a democrat -- he wouldn't be elected if people didn't feel he was doing the right thing. >> host: are you concerned about human rights issues in iran, explain a little bit more. >> caller: of course i am. i've lost friends there. my teachers stepson is in iran fighting the war. i don't know what to say. yes, i do feel he's doing the

right thing but he is only doing it because he's got to take over the leftovers from president bush. >> host: blake hounshell, talk about how president george w. bush -- does he still loom over the u.s. foreign policy? >> guest: yes and i think the call is talking about in iraq, not iran. i think the consensus is overwhelmingly that the iraq war mack was a huge mistake and it crossed well over a trillion dollars and a lot more in terms of lives and injuries and people with lost limbs. people's livelihoods have been destroyed pity it was very costly intervention. and the white house will tell you they've spent the last four years digging out of the hole the bush administration left. there is an iraqi delegation in washington this week.

they have lobbyists for the first time and they are buying u.s. weapons systems, but they also seem to be aligning themselves uncomfortably closely with iran which is an adversary of the united states. so i don't think too many people would argue this is a big victory for the united states. >> host: let's turn now to venezuela and talk about u.s. relations -- >> you can watch this on the video library at c-span.org. we take you back live to george washington university. it's been ten years since the space shuttle columbia disaster and they are hosting a discussion today with nasa officials and people who were part of the investigation into the explosion that killed all seven crew members on february 1st, 2003. >> the next panel is a discussion of -- we talked at the immediate response to the accident in the previous panel

coming and now we are going to cover nasa and the columbia accident investigation interactions between the board and nasa. we have to people who are expert in that discussion. one is doug cook who was a nasa adviser commesso nasa person dealing with technical issues. and we have dr. john with us in the space policy institute and he was a member of the columbia investigation board who came in to the board membership after it was formed as it was realized that there were other deep organizational institution issues that needed to be looked at in the course of the investigation. without further ado, doctor? >> thanks, scott. i not the best person to describe the day-by-day operations of the columbia board, because i joined a month after it started operations. and with the admiral's

permission since most of the work i was doing had to do with what was going on in washington, i based my investigations year rather than hanging out with the people in houston as they went about their technical work. but i heard enough about what went on. some of it i will talk about, not all. so, what do i push here? in one of the slides this morning, the point made that one of the things happened after challenger was the development of contingency plans should there be another accident. and those contingency plans were put into effect on february 1st, 2003 after was clear that columbia wasn't coming back. and part of the plan called for a seven person investigation board where the people were predesignated by their possession, not by their

identity. with the nasa administrator name of the chair of the board, which he did that day with contacting the admiral. he had run the investigation of the terrorist attack on the destroyer in yemen and had the experience there for of running a rather elaborate investigation. i think sean o'keefe was the navy to the core and probably wouldn't have asked a non-handyperson to take over -- non-navy person to take over. the board quickly assembled by the afternoon of february 2nd they held their first meeting and there were pictures in scotts video of the first meeting at the air force base in louisiana. and quickly they decided that they needed to be near nasa in houston so they moved from louisiana to houston and a

couple of days and then when the investigation was done, the core team moved up here to the building where we did most of the report writing to get a remarkable experience, intense to the people that did it from day one to the report released on august 26th, 2003, working at least six days a week if not seven and certainly more than eight hours a day. first thing the board this figure of the way the plan had been set up there wasn't provision for adequate independence so there were iterations to guarantee that it could carry out a totally independent investigation with its own budget, its own staff, the staff of nasa maintaining its own independent records.

this wasn't going to be a backup kind of committee because it was constant, but from the start with the important exception as the admiral noted this morning, information about the crew conditions and the ability to take privilege witness statements on a cost of eventual basis. this was a very public and a very open kind of undertaking and the new charter included the possibility of new members beyond the now eight person board. originally people in the program were involved in the investigation. the board felt that wasn't a good idea investigating itself. i should add a historical by comparison after the apollo fire in 1967, jim was able to

convince the president lyndon johnson to let nasa run the investigations of the different models of how the investigation should go forward that were at play. i'm trying to think of what is on the next slide. in its first few weeks, the board decided to go beyond the traditional investigation. first of all to look at nasa as an organization and how it operated with its culture was and have its communication channels worked or didn't and the next step in broadening was to place the accident in the long-term historical context of the shuttle program. to focus on the accident but casting a broad context of what

is it about the future of the u.s. space program. that kind of input came from outside and i should say they want little to do with this. the staff level were under orders not to talk to it. they were afraid that somehow the white house, president bush and his associates would get blamed for the accident. so, my meetings have to be at starbucks on the corner, not in the executive office building. here is the board to remind you who was there. admiral, major general, all of them were original members.

scott hubbard it named a nasa official to be a member of space at that point the director scott got the honor. i was asked to join in early march after the decision the board was going to look at the history of the budgets and the political history of the shuttle program. doug osheroff was added at the same time, namely i think because the board needed a nobel prize winner. osheroff won his nobel in the cold temperature physics and was pretty clear by then that the tank was part of the accident

chain. sally was added at the same time because she was sally ride, very smart woman because she had been on the challenger investigation because she had a lot of good contact internal to nasa. she was a spy in terms of what was going on roger was an ase engineer and a weak leader sheila widnall, a highly qualified engineer was added to the board, and then steve turcotte were original members. 13 members incidentally the same number as the challenger investigation board. >> this is an elaborate process that is emerging army. the board and its own staff and a lot of relationships to nasa

and its investigation activity that is what doug is going to talk about. is it part of the team? >> i was assigned -- you were part of us rather than them? >> we worked in parallel with nasa doing a lot of work to understand the accident you can get a sense of this diagram, and the board itself organized itself into four groups. uzi the names of the group and the board members assigned to each group. we were not allowed to hire our own staff. the generals in the crowd and the admiral brought along their captains and majors and their

tenants so there was a slight uniform look to the people working on the investigation and some of those people were very good. one of the lead investigators there's a total of 13 board members and 30 investigators were just kind of senior staff. it's an independent analysis team to run our checks on the analysis that nasa had been given us. so, total people connected probably about 150. a lot of people anyway. i miscounted by the way 30

investigators and significant support staff for administrator, documentation photography, government relations, budget travel report writing. it's been a lot of compliments over the years about the quality of the report itself. and i think that from the start there was an intent to make this a high quality piece of literature by writing well illustrated retelling the story a. we had a professional team to do that to be about the first draft of all of the chapters were written by the board members at maybe a few of the very senior staff. this was an extensive and expensive. this was a $17 million undertaking, which is a fair amount of change for a six month effort. we wanted to get it right and we took every provision to make sure that we did.

here is the charter on the four teams, just to see the range of things the were looked at. this was the team that i lead, the organizational policy with sally ride and scott hubbard involved. everybody got involved on this one because i think we can to the conclusion that the specific technical accident was really a embedded in a lot of history in both politically and programmatic and organizational history. here was some of the people that were involved in the team to the a couple of them would be familiar to you share in washington. on your left, the now head of state department, space and

missile office. harry of syracuse, dennis jenkins, who we brought in to team for and the admiral quickly procured to be his kind of person that really new every nut and bolt on the shuttle. dennis had written a very extensive book on the technical development of the shuttle roger from the air and space museum. dwayne who is here someplace yet who is a ph.d. graduate by that time from gw now working on this base board come into the master's candidate here at the space policy institute who is probably the most successful graduate. he runs a brewery in the virgin islands. [laughter] better than being the deputy at nasa i think. [laughter] to me out of the whole report

this is the single most damaging that nasa and indeed the country didn't keep its part of the bargain to offer this complicated fragile system to the maximum of its potential for safety so we were careful not to point fingers at individuals, but we certainly pointed fingers at almost everybody that was involved in the program. as i said, we were asked not only to look at the accident, but place the accident in the broader context of our now what. here are some quotes from chapter 9 of the report, and you can read them.

what's interesting to me is that almost every one of these could be said of the situation today so there was an attempt to change things. the bush administration read our report and agreed with these kind of conclusions. president bush set forth a vision for space exploration in january of 2004 but for reasons we don't want to go into here, that wasn't sustained. notice i didn't say that it wasn't sustainable, but it wasn't sustained, and we are in a situation today where there is a continued failure in my view of national leadership that set out directions for the u.s. and space. and that's all i have to say in the formal remarks.

doug? i'm going to go down here so i can watch you >> anybody want to help? thank you. there we go. i will go ahead and start while he is fixing what i did from. so, this afternoon i want to give my perspectives to be it's

actually a perspective that is based on my a years in a mass of and the programs i worked on and my experience with this so going back on the program, i was in the program starting in 1973 right after it became a true demand work on it through this development, so i had experience for about 11 years leading the flight testing on the shuttle home. i worked on the enterprise challenger where we had instrumentation weeded flight test work, and then after the challenger accident i was asked to work in the program office and ended up working in the program office during the return to flight activities after challenger. and so that was my early

experience. i volunteered to work on this. i remember calling brian o'connor right after i learned about the accident and asked him if he needed any help and he said i want you at barksdale as soon as you can get there so i did and i ended up being there as the team informed and lifted connect entire experience my takeaways and my view is based on my memory which may not be completely consistent with everybody's memory -- it's been ten years -- by think it's pretty close. i'm going to use the charts most of which came from documents i did at the time but we will talk about some of the technical aspects and some of the things that stood out in my mind at another aspect of my perspective is that i ended up before i left nasa come as i did leave nasa i was there for exploration at the nasa headquarters which included

the programs that came out of the experience these are the programs that followed to be the next programs in human space flight, so as scott horowitz preceded me and a couple others but i was the last one and so i had the consolation program. i had then the space launch system and zero ryan had human research program and some technology efforts. that's my experience. all of this experience relates to where we are today and i'm going to talk a little about that. i'm going to play a short film. scott showed a little of it earlier. this is a miraculous -- there were a lot of things miraculous about this effort including all the people involved. but this piece of the film as i remember was found lying on the ground by itself, not in a

camera, but the film itself. let's see if it goes. >> don't forget about [inaudible] >> we are checking it out. >> sounds good. colombia houston will take another item 27, please. >> [inaudible] it might be. we see it out to the front.

>> it's kind of dull. >> will be obvious when the time comes. >> [inaudible] >> don't do that. go ahead and make sure you check your seat pressure. >> i will leave it there. so they are taking a picture and commenting on the plasma out the window. this was normal, that at this point in time, the plasma was actually burning into the wing as they were commenting on it. so the next memory of columbia is the debris from the accident and that's what we worked with during this period. it's a sobering even today to think about the human tragedy as

well as the aftermath. this really indicates what can happen when you have a technical problem. whenever there are things that you don't know about the vehicle that you are flying. when there can be a lapse in requirement where you don't absolutely have the requirements right and so you are not expecting problems. but it has to do some with trying to understand a vehicle that is very complicated. and in this case obviously the shuttle, we were learning about the shuttle to the days that it was decommissioned. so this is the hangar. you've seen pictures of it previously. as the pieces started to come from the incredible job done by the recovery crew dave and scott talked about, we saw a briefing

in the briefing room where the aerospace experts came in that had investigated basically every breakup of anything coming in the atmosphere ever since they started happening and they said you only get 10 percent back, as david diluted to earlier. and it probably broke up in 100,000, that's where everything breaks up. you only get 10% back because it is made of aluminum and aluminum will burn up. we ended up getting, as david said, between 40 to 45% of the vehicle by weight. a lot of that has to do with the fact it had title on the aluminum and survived because of that. we learned an incredible amount by studying this evidence. early in the process, all of the requests for data, all of the action was happening for the shuttle control board so we would go over and sit with these guys and you would see a picture or two of something they found

and everybody would try to solve the accident and not one piece. and actually everything was funneled through the shuttle program at that point in terms of the program office and this wasn't working and it became a choking point and they said we need a different model because we need answers faster and the data faster so that was arranged on an earlier charge john showed. frank had a team set up to interact and answer the requests and bring in new studies that were determined and so that began to work better. at about the same time we also realized that just seeing the piece and the picture on the screen wasn't going to get us where we needed to go in understanding the data. so tapped a good man is here

somewhere. he was my partner. we spent a lot of time at the cape going through all of the debris trying to understand not just the pieces that were in the area where the accident occurred, but looking at it in the context of every bit of debris on the vehicle and so that was very enlightening in terms of understanding how this all unfolded. the first day we got there we went and talked to the guys and they said we generally wait until we have all of the debris before we start trying to piece it together. we knew the shuttle program was wanting to get on with the program and we said well that's not going to work so we just decided we are going to spend a lot of time down there and look at the debris as it came in so we spent an incredible amount of time at the cape. use of the early versions of

this. was a miraculous effort to get this data. one important aspect is the new where each piece was found. that was incredibly important to understanding the accident itself. so they had accordance on every piece of debris so even though it is a broad footprint, that was incredible information. this is the stripe as you saw it and it was a piece of foam at the other picture shows the ramp that was a piece of foam at then end of the pot being the structure that attaches the shuttle in front of the orbiter to the tank cover the external tank and this is a piece of foam on the tank. an incredible amount of work was done to understand its

trajectory because that was important in the sense that we needed to know the relative velocity because of the time of the accident, it was felt that the film couldn't possibly do this kind of damage so because this light piece of foam yet when you calculate the velocity of it, it actually created quite a force that the leading edge. so as time went on, these models got better and in fact it did hit panel eight being on the leading edge of the weighing its reaction -- there is the rcc panel that basically looks like fiberglass. it looks tough, but basically we learned over time -- we didn't know early on but it did hit panel eight. i did something wrong.

here we go. another piece of the data was flight data and you are seeing a departure and the - line where what happened during the entry you are seeing the data with the ergonomics don't make sense of the orbiter starts departing and all of the checks are saturated and it keeps on going and it shouldn't do that. here is another trace of that should be kind of what it looks like and so we have the various data like this and everybody was interested in trying to make sense of all of the data and there's an incredible number of

studies that were done and tests and all kinds of analysis to understand what happened. one of things we realized early on as you have to integrate all of this you can't just look at the debris. you have to look at where things go on the ground and put that together and try to figure out what happened. this is interesting in the fact this was a ground track that shows where rcc, the pieces of the leading edge hit the ground and basically we had the left wing and right wing and the left and the red if you put it down into those areas basically a his the middle of the area where the left-wing hid. be is the area the tail hit the

ground and sea is the area of the right wing hit the ground. so, from this and looking at the debris, we determined -- we got a good idea how the vehicle came apart and it also helped us understand what was going on here. so this is a map to recall these charts pretty much or keep charts and i updated this on the latest small but we ended up with a final debris that shows where we have pieces of the bottom of the shovel and basically the shuttle of course is flying high in those first and as soon as the left wing who departs, we figured out if you look at the center of gravity and pressure where basically the outline of the shuttle it's going to change directions and we saw evidence of this on the

right wing of the ceramic titles going sideways which was consistent in the direction consistent with the shovel actually going sideways. the left side after the left wing who came off and then after the right wing cam off it was going to the after front so we were beginning to piece all of this together and we knew things were on the ground. that was a big point that people who fought the door came off and we also had a scenario where maybe they were caused by the left wheel coming down early and so we spend a lot of time at the

cade trying to eliminate these possibilities. this one just shows the pieces that we had in front of the well. we had enough of that in the circle and it shows some of that hardware on the floor basically we've ruled out anything coming out of the real well. there was a team looking at if the wheel came down and that is what caused it to depart. we got the year which go look at it and when we looked at i think that's pat holding it up. the front side if it had come down had chroma which wouldn't have happened. the under side was burned up pretty well which is the picture on the right and the would indicate that it was up during

entry. one of the scenarios. they're behind the leading edge that come off with title on them. the ones of those quarter inch bolts and the ones on the bottom half to of the quarter inch bolts holding them on and we thought maybe it wasn't the leading edge. maybe one of the panel's got hit and came off. as we gradually got all of the panels on the underside of the left wing. when it came and it was panel number eight and so right up to that point we thought it was a carrier panel that eliminated that possibility. we also had evidence where there were deposits this is on the inside of the panel eight pieces where this is molten material of

the plasma burning inside of the leading edge so we have deposits on the inside of the pieces. we also have the air and amex during computation dynamics and the cavity which is basically unheard of at that time reduce computational dynamics over its shape they are doing computational dynamics in the cavity with the plasma. and they basically showed the flow the would duplicate the damage that we are seeing here. so basically where we were at this point in time is we were pretty sure that there was panel eight but we couldn't prove it conclusively and then the recorder came in and i remember a picture that showed a sitting on the ground all by itself just sitting there and everybody was very concerned about the

condition of the tape when the open doubles pristine and so we got data from that and basically between the data itself that showed the changes and pressures and temperatures in the left wing and the fact that these measurements would go out offline because the lawyers were being burned through this is a close-up picture of the orbiter where the wires were routed and so we know exactly where the lawyers were and we knew they were separated by this much and you can see them going off one by one. and so, with that data, and here is a picture of the bottom of the leading edge with this information within the area of about that much. so the data actually was the data that cemented our

understanding of what happened. another thing that occurred is that we saw the data that showed the spikes. if the plasma is burning through the leading edge of the wing at some point it exit's and because the flow of the orbiter it is going to roll up and hit. so when you look at the ombs pod and had aluminum in the leading edge. early on we had a part of the vertical tail and was lying on one side of the floor and when we figured this part out, we realized on the slide that's on the floor we were walking by it everyday and so at this point when we thought about the scenario we said let's turn it over and see what it

basically because it happened and we never had a severe problem, we had demijohn titles out to cause the same problem. there was a schedule pressure during this point in the shuttle program. we had a space station to build and we already had deutsch other problems and the same time we

had other problems that grounded the fleet. the schedule dates didn't change. so this was a pressure that was on the program office. this was a chart out of the presentation. there were issues during the time of the flight. the engineers knew the shuttle was being hit and they had the data. they asked for ground pictures to be taken to see if there was damage of the shovels and those that were not done. there were people wanting to find out more during the flight about what damage there might be, but they talked about the crater model and about past experience and family and that sort

>> here this issuing panel 12, panel eight is the one that was hit. there was a study done by the mission ops folks to look at what they were possibly if it were pursued to go look uva, to a continuing see you get to go look at the win. so the studies done to look at this and said yeah, we could have done this. they also asked well, couldn't we, could the career have stopped it from burning in as quick as possible? that sort of thing. so there were a number of, a number of observations made. there were some issues with the independence of snm a safety and mission assurance that were called out.

and their independence. so there was some criticisms on the organization and that sort of thing. so, given all this and in memory of it, so what are the concerns? this is not a caib chart. is my. i pointed out the importance of requirements. do we have the ability to impose requirements? one of the concerns in question is under the approaches we are taking on programs now, do we have the ability to impose requirements? in particular, and to have the right level requirements? you can overdo requirements as well. we are good at that at nasa, over specifying things. so you have to find where the fine line is in all this. and when i was there, i put emphasis on common set of human rating requirements, because we have different kind of programs. with the commercial programs

that actually didn't mention earlier. those are mine, too. the commercial cargo and career when i was a a. we had those and we have as a less. so if, what is the fine line you draw? the same set of her garments to protect people should be as good for one program as they are for another. and so you can come up with a really good valid instead of requirements that are thorough enough but not overdoing it, then why would you have different types of requirements for different programs? a concern that i have with one, maybe that will be a follow-on session of some sort is with spacex agreements you can't impose requirements. the lawyers tell you can't compose requirements under space act. yet the safety prospect and the program goes on on a daily basis. if you look at shuttle program, if you look at any development program can make design decisions of some type every day that you go to work, and, and

the safety people and the independent center are there with you looking at the design decisions and weighing in on whether not it's safe enough. and so the question is, under space ask, are we able to impose requirements in a wave of confidence later that we are safe? because we're still going to fly people on them, and so, and the same thing goes for oversight. you can't impose oversight of the processes in the development and a space act. so now the programs have certification effort, or, or certification contract with the developers of, for the commercial career. but it comes late in the process but a lot of the design has already happened, so the question is, going back to admiral dyer's comments this morning, are we early enough in the process to have the

confidence in the systems that we are developing? so that's a question. the attention to detail is another aspect of that, and design detail, you can see small comedian see things in these accidents that you didn't expect. and there's incredible amount of detail that's gone into the shuttle program in terms of understanding it and try to get to safe situation. another thing that happens is, as you become successful, and they showed up in the shuttle program before each accident, you become successful, and you're not -- you're knocking flights at will, they can breed overconfidence which lead you to other problems that you see crop up in these failures. there are political influences in the shuttle program, the flight dates, the schedule pressure. there's obviously tension, political tension around our

programs today. so does have an effect and it can affect safety. so have a question is, not trying to answer it, i'm just asking questions or pointing out concerned. have we drifted away from that, the things that we learned during the columbia accident, its aftermath? i know for a fact that immediately following the columbia accident, and following the challenger accident, nasa became stronger. the people involved in the decisions, the people that were involved in it, from whatever standpoint, became stronger and worked from these lessons. but as time go by you can drift away from them. they become less relevant as you work your day-to-day job. so in the end, you have to ask yourself those questions, and whether not you are safe. because we know that the consequences for these missteps or misconceptions can be

catastrophic. and in the end, the people that fly on our missions, fly on our spacecraft, family members are they our friends, coworkers, and they depend, they can know everything that goes on in the vehicle they are flying. they depend on the operators. they depend on the engineers. they depend on the program manager. they depend on people in washington who are setting policy. they are dependent on all of us for getting them safely as we can. understanding it's still high risk proposition. so that's my comments, and probably don't have much time left. be happy to add to questions. >> -- be happy to answer questions.

>> we have time for a couple of questions. >> you talked a little bit about the extensive investigation but is there anything that you would change looking back as far as the breath or the depth of the investigation? or do you think that it's a good model if we ever need to do that again? >> from my viewpoint, there was a caib investigation. there was also all the nasa work and analysis, and he was combined effort indian. a caib have a job to do to review it, but there was a tremendous about of work done by nasa and supported this. and i think it was very thorough in the end, personally. >> again, i would add a caib brought in the best expertise we could find in the country to support our analysis. so we didn't think we were leaving anything unturned.

>> can you talk a little bit about the parallels between the and family issues relating to the foam and those related to the blow by on 51 l.? >> that's farther back in memory for it to go back that far, but there were, there were instances where there was the erosion of o rings prior to the challenger accident i do think it was understood that cold weather had such a factor. i don't think, until the investigation had i don't think people understood it that well. but there are, i mean, there was some of that going on then, too. that's a natural thing in human nature to say oh, we've seen that before. the problem is when that occurs, we might have seen it before,

but you don't know where you are with respect to the cliff. that's the way we usually call it. you studied things within a certain boundary. you done analysis. you've done testing. qualified of these conditions. and so something happens that goes outside that boundary and you say oh, we didn't have a problem, you know, but we never analyzed that. so the next time you allow ago a little bit farther. where do you fall off a cliff when you have catastrophic problems. that's the danger in that mindset, i believe. >> one of the people that we consulted with during the investigation, a woman named diane von, now with columbia university i think i'm agreed a very detailed look at the challenger accident and its causes. and it used some sociological literature, talk about being an

academic, she didn't say in family. she said normalization of deviance, which is basically the same thing. and i remember very, very vividly at one of our public hearings doctor ride, sally ride had been on the challenger commission and was listed as someone testified thing i hear echoes of challenger. there were some similarities in the process. >> i just wanted to make, i agree with that, there were similarities. there was one fundamental difference though between the engine aspects of the to action. so engineers are people, too, and the social logical things are worth pointing out. with challenger, the solid mechanics of the joint behavior as a structure were not understood until well after the action. it literally rotates -- it

rotated differently in a different direction than engineers thought when it was designed. so had the joint behaved as it was believed to have behaved, the design would've been adequate. in the case of the foam, i think everybody knew that it's possible if you hit the wing hard enough to break, a cognitive gap was whether not foam could do that. but there wasn't a fundamental misunderstanding of the physics. it's a question of the level involved, the forces involved. with the challenger solid rocket motor joint there was a fundamental understood me -- misunderstand how to work at all. those are different things. >> and my recollection on the phone, i'm not sure that, i do remember there being evidence that the work had been done to understand the velocities and the momentum of the foam before the accident. the foam had come off before by don't think, i never saw anything that i remember saying

that they looked at what it really took. they just thought this is a light piece of material, it couldn't possibly do that kind of damage. but even when you get to velocities at that mass you have issues. in fact, on that note, just quickly, i do have a film of the southwest research test where they actually -- >> nasa didn't want us to do this. >> so this is, naturally shows it right across the thing. so this was a test that was done about shall rcc at southwest research to see, at the right philosophies, see what the damage could be. from a piece of foam. >> kind of make one more comment? >> jim duffy at faa. at one time i was project manager of columbia. you mention an important of requirement, and another tool to

the system engineering is designed referenced missions. one of the things we learn on shuttle was that every time we analyzed the abort missions, they size everything. so if you design your system just to the nominations, or shuttle that was one, two, three and three b., you miss all the design levels. so we had to go back and redesign several things when we finally got around to analyzing the important missions. so that's another requirement that i don't know if it's employed well enough today. >> thank you. >> thank you very much. [applause] and if i get asked the next panel to come up. we have ken bowersox and ham.

and i believe in that order, you have to deal amongst yourselves. okay, we're going to put ken up first. [inaudible conversations]

>> [inaudible conversations] >> [inaudible conversations]

>> thank you, everybody. the reason i'm here, i was chief of the office we lost columbia. the interesting two characters at the table, there's a lot out in the room that work very closely throughout office. so although this panel is called technical lessons learned, what i really want to do is kind of offer up the perspective of the chief of the office what it was going through columbia and lessons more from achieve as you pointed to and from the astronaut that we learned. i want to pull together an outline, put physical cause, i think we just had a tremendous brief on that. doug cooke did phenomenal. in that information you spent

and that amount of time is very good, thank you. that was great. now, as chief passionate and show to my crib, my time has always been good. i can think this is one of the exceptions. columbia was the first mission i launched as chief of the astronaut office. the nation before, socks of the table launched on this on 113, i became the chief when he was on orbit and landed for the landing of the columbia 13 but i never launched a shuttle mission before. before i launched sts-107. so needless to say, fairly new as the chief perspective. so what i want to can cover today was the leadership opportunities that were presented, the fact communications is essential, particularly in something like this. and then given the columbia accident, what's next? what do we as the astronaut corps think is the right thing to do.

and interestingly, you know, the day we lost columbia i was in florida flying the shuttle training aircraft come expecting to see columbia show up. qaeda chase gormley down the file, gather the wind data. i guess if it didn't happen so it was a very sobering day. immediately lessons from the previous accident showed up as a positive and the fact that we now have a plan for the families, the spouses. so on columbia, i felt nasa did a really nice job from the administrator on down of treating the families right, giving them the right support and trying to make this come to everything that we could for those families. that include on the landing because the families now, it was a plan somewhat isolated them and gave them the privacy that they deserve. that wasn't something that nasa

had in order for challenger. so lessons from challenger were, in fact, used. when i got back to houston and got the office together, we had a meeting and it was a pretty, an emotional meeting. because the emotions are kind of all over the map. but what was very clear to me in that meeting was the astronauts were very experienced. as a matter-of-fact more experience, the more outspoken that day, the spatial, maybe it's too risky to ever fly again. now, keep in mind you're the minority. maggiore in the office had never flown the space shuttle, and they were, i won't use the word unfazed, but they didn't care that flying the shuttle was risky. they actually -- when am i going to get my chance to fly?

so the office was spread across both ends of the spectrum. we held meetings about every three weeks, because at this point i felt it was important, there were about 100 of us, to kind of understand and have a position on what we found. and it was really a recommendation. because the way it works in the chief of the astronaut corps, wasn't on the caib. in many cases don't officially have a say in what goes on. it's more of a providing a recommendation in some cases being a cheerleader. so for determining the passport i thought it was important as a core that we came to what we thought was the right thing to do. and what became apparent out of all this was to do the basic math, there was one of 13 space shuttle missions with foam, we lost two, two shuttles and the career. so the basic math says hey, your odds of a bad they are like one and 57 when you climb onto.

when i shot off in front of the nemesis in my trusty tomcat, my chance is not coming back, actually my chances of being killed were one in 20,000. so much safer to fly in combat in a fight on a mission than to climb on the space shuttle. when you climb on an airliner, it's an order of magnitude safer than it. so they start talking about risk versus reward. so obviously the best part is it's risky when you climb on the space shuttle. what's reward? and does reward justify the risk? and what we worked up to was absolutely. we have global demand on space station, complete building out space station. however, the other thing we recognized was when you lose a national that -- asset and seven astronauts, the whole country, the whole nation takes a hit in

prestige. and it appears that the further we go along, the less as americans were willing to accept the. so if you want a healthy spaceflight program, you've got to have one that doesn't lose the career. at least very often. so as we started looking through this we said well, do basic math, how many missions does it take to build out space station that answer was around 25. while, your odds of losing a shuttle 25 out of 157, basic math, you know, we have, the odds are we won't lose another study and we can build a space station. but also can we desperately wanted to be able to go forward and go to mars. is a debate on whether go to mars via the moon or just the right to mars. asked a nation that you get to opinions. i think we need is a test bed so i am bed so i outsold we believe in going back to the moon by taking more time, will be more expensive to go to mars? yes. i think that is the right way to

go. so we need a replacement. we need one that can go beyond earth orbit and it needs to be safer than the space shuttle. if you start looking at it, they can be easily an order of magnitude safer, particularly with the career system. so as an office which have generated a white paper, turned it in, kind of up the chain of command. and interestingly that, in fact, was the vision that was rolled out. i never had the dots connected to i don't know if it was quincy this or if it was input where the action had an influence or not. but the reality is, phenomenal machine as the spatial is, and i absolutely love the space shuttle, it was time to move on. so we could explore deep space. when you talk about leadership,

immediately realize communications was one of the most important tools in fact that i had. and so whether it was doc working with dave on the recovery and astronauts out in the field, whether it was talking to sox who is on station by the way, and by the way, i remember the first time i talked to the u.s. astronauts on station, you know, a day after we lost columbia, socks and don pettit, it was actually communal, their whole attitude was we're here for you. anything we can do we can do. leave is on orbit a year, you know, don't worry about us. take care of columbia. and that was pretty much the attitude i saw across the board, same with working with the shuttle program, same with working with recovery team. interestingly, we were part of the recovery team, recovering

our own, which, in fact, i won't go into anything other than to say it was not pleasant. after the fact, some of the medical folks had you know, that was not smart. if we could this again, we do not want the astronaut corps involved in the recovery, particularly when every recovering part of the astronaut corps themselves. i disagreed with that. we did go through psychological debriefing. for those in the field that did some of the -- unpleasant. but one of the most frustrating things is to be held off our be put on the bench, because somebody is worried about your emotions. so i was very thankful that we got to participate, do our part, do everything we can. in the end i think that all worked out very well. the other thing in working through this was working with the shuttle program on getting back to flight, returning to

flight. from the astronaut's point of view, the one thing that we really cared about was inspection. if we didn't have repair capability, fine. if we didn't have a safe haven, that's okay. the one really requirement we had was we just want to know if the space shuttle is damaged or not. because i don't think it was -- the fact is had we no space shuttle columbia was damaged, in fact, it was going to do it at a point of processing that nasa, we could have physically gotten into in the air in time to rendezvous with columbia. that opens up a whole question of a, what if you had done the? you just last last space shuttle to give don't really know why you lost it. would you really now launch another one? and being a military guy, you, i

can put it into, hey, when you fly in combat and you get shot down, we send helicopters can we send rescue missions in way more dangerous situations. and my guess is i believe that's what it would wound up with nasa as well, knowing that hey, there's a lot of risk, we probably would've gone forward to it would've been a risky mission, no doubt, but the point is if you know the risks, then that's where nasa has always excelled. so, in fact, from that point on when we flew the space shuttle, we always had 100% inspection so the career knew the integrity of the thermal protection system. if there was, in fact, a problem, the career could stay on the international space station, you would lose the orbiter that you would rescue the career.

-- the career. with akamai 50 minutes -- by the way, i'm kind give time for the others and questions at the end. here's a quote, which basically says, you know, it's in our dna to explore. as a matter fact a week ago when inspiration mars was rolled out, one of the questions at the end was a real tough one. what kind of inspiration with this mission be if you kill the crew? and i think the guy trying to answer it was having a little bit of trouble and somebody probably tapped the widow of the commander of the challenger mission, and she went to the mic and said hey, look, you know, it's risky, we know that it doesn't affect i'm not going to be june justice but basically she was an inspiration. so the inspiration that the

will, the need in our dna to continue to pour always be there. that's why this is so important. so that we make sure we learned those lessons because we absolutely learn much more from our failures than our successes. so with that, sox is up. >> as mentioned, i was on orbit when the sts-107 accident occurred. the experience there of course shaped my perception of what happened, and my ideas of what i think are important lessons from the experience. so i thought i would tell you a little bit about that.

when we launched in november with the crew of sts-1 13, paul lockhart, don pettit, mike and john harrington and me, i'd just like to put a shuttle sliding aa speech i gave because it's a cool thing. it's important estimate that. rommel talked about the risk and reward. their part of the reward of these glorious launches and what people do. and that's what carries us through. the difficult times. but we made it up to orbit, and one of the first things we have to do was make sure that our suits would fit and that our seat liners would work in the soviet spacecraft. that was already docked after. that was our backup plan. it's funny, this was the first of a new series of soyuz, and nasa and the space station program, shuttle program and the russians had sort of bent over

backwards to figure out how to keep my crew from flying on that vehicle. in the russian program history, they often have their problems early in their vehicle design cycles rather than later. and so they want an experienced russian crew to fly in the soyuz. although we're supposed to launch in a soyuz, the change that midway through our training flow, and we went to a prime shuttle launch a were supposed to fly home on the shuttle as well. we were supposed to fly home on sts-1 14. infect sts-1 14 is probably the most crew patch versions of any credits ever been in existence because of all the different changes that happened through the program. when we do this particular thing, we figured that the most important thing to come out of it would be with is pictured right there. this is a really good picture come you got to admit. they are good colors and if you

blue eyes to make them stand out. so this is one of my favorite pictures. but we really never thought that would ever get in the vehicle, and let alone come home any. in february, in the middle of the week before the accident we had a chance to talk with the sts-107 crew. they set that up for pr purposes, crew to crew. but is also a chance to just talk with her friends. we hadn't spent a lot of time with them because we're on opposite sides of the hallway when we were in training, but we would run into them in the bathroom occasionally between training sessions or something. and i remember talking with the folks, and we started off talking about the science parts of our mission. and then laurel came on and said hey, let's talk about our kids. enough of this science stuff. and our member talking with ramon, his kids and my kids were

going to the same school. and i said hey, if you see my kids, give them a hug because you're going to get back before me. as it turned out it was the other way around. the emotional part of that, of course, was difficult for us as a crew, but we had a couple more months that we needed to stay on orbit. we needed to do our work. the program spent that time regrouping to initially they told us you could be coming home on time. that's always the first answer for any program manager. don't give up the original plan. we don't want to switch to the backup plan yet. and i remember all of us kind of shaking our heads going, these kinds are -- these guys are really deluding themselves. this is going to take a long time. they really didn't have any other choice. they weren't ready to tell us what the back up plan was. during that period that actually did discuss leaving us on orbit a year, which as rommel mentioned him we were kind of all for that. it gave us more time on about

the environment. and at some point be analyzed logistics for station relies is going to be difficult to sustain more than two crew members at a time while they were there. while the shuttles were down because they needed the extra supplies. so at that point, a bunch of cruise after hours got reach out and they went from three-person crews, some flying on charter, some flying on soyuz to two-person crews. and around the may time frame, ed liu and yuri came up in this particular soyuz to relieve my crew. we're supposed to fly home in the soyuz that were never intended to fly on. for those of you don't know the soyuz we will, i like whenever have a picture of. no, explained globe about. it's a very simple physical. it's kind of a great example of

how spending more money doesn't always make vehicles safer. this vehicle is smaller, carries less crew. it's much less capable than what it can do performance wise than a space shuttle, but it's got a great amount of redundancy and a great amount of capability to handle failures. even though it's very, very simple and it doesn't land on a runway. the part of the soyuz that is close to the space station is the bulk chicken. that's a habit to chill module. there's a bathroom and a place to prepare food or so store food and a place to change clothes. the bell shaped in the middle is the descent module the crew comes home and when it separates from the rest of vehicle. the bottom and, the computers and the propulsion systems. well so, yuri and ed in the green suits came up to replace

nikolai, don and me. we got in the soyuz, and started on our way home. we're in the middle of backup option number one, thinking that okay, everything is going to go great from this point on. the intended entry profile, what is called a controlled descent, that's what a shuttle does. that's what jim and i in a polity where basically you roll your vehicle around to control just how fast you fall into the atmosphere so that you hit ethic as part of the atmosphere at just the right time to minimize the load on the vehicle but it also allows you to more closely control your landing touchdown plan. well, not too long before entering the atmosphere we had a change in the display on our screen and we were in backup mode number two. the control system had experienced a design feature and entered what was called a ballistic mode. autistic mode is a higher entry-level descent, a higher g.

level, and the vehicle get up to eight or nine she's on entry and you just go into a steady role, coming. it's not controlling the entry point anymore but it's just trying to control rolled rates. but it's one of the things that's leaders would welcome the downside for the group is now you than 300 miles away from where everybody is waiting for you. so our russian partners long ago had cut support at the ballistics an entry landing site out of the program to save mon money. and there was nobody there to meet us when we landed. not only that, when we're talking with the rescue forces were still a few hundred miles away when we're up by coming in on the ayrshire we were able to talk to them. they said hey, we see you guys. we've got u.s.a. as we are getting closer to defend, we said how close are we to the ground. they said we lost you, we get a. we thought maybe there was a force at the recovery site. even more entertaining than

that, because the talk to us we didn't tell them that we had a ballistic entry and were 300 miles short of the normal landing site. we also didn't know that they didn't have tracking. that begin have been cut out of the program a long time ago. so the russians did know we were 300 miles away. so we ended up landing, and because we thought they were probably close would probably inhibited our best communication antennas. so we couldn't talk to anybody and nobody knew where we were. it got quiet. we sat there for 20 or 30 minutes what you for somebody to come up and open the hatch, you know, nobody can. at this point i thought, you know, people have been wanted for ever, long duration members can really climb out of the capsule. let's give it a shot. [laughter] so we climbed out and as you can see a, it wasn't completely pretty but we did get out of the vehicle. then we saw to realize that this was starting to sound probably two people back there in the control center a lot like what happened on columbia. here's a group not too far from

touchdown, and with his radio contact and we don't know where they are. spent so, sucks, let me interrupt. -- so, sox, let me interrupt. i was in the control center standing next to a guy that turned whiter than this tablecloth you. his name is sean o'keefe, the nasa administrator. you could just see him go wow. two in a row for not getting the crew back. so it was extremely tense time. for everybody on the heels of columbia. >> so, nikolai and i were talking, and i said we really ought to try to contact somebody. we didn't have a cell phone. now crews have a satellite phone so they can call as soon as they get out of the vehicle. so i climbed back into the soyuz and started pulling out the rescue equipment. nikolai started to probably

shoot off all the flesh. went through most of the russian supply. had to go out with a new procurement to buy flares. we had to make contact with the searcher playing. the plane, one of their plans if they don't make contact with the crew is to start working to the shore part of the cycle, figuring a ballistic entry is the most likely scenario. they started working their way through it. about two hours after we landed we were able to talk with the rescue to point and tell them everything is fine, and shortly after that the kind of flew by and about three hours after the the helicopters came and grabbed us and brought us back to star city. in a period of three or four months, we've got a backup crew member, don better replaced a prime crew member. we'vwe have done a backup optioe exercise our backup crew mode and we've used a backup option

on the entry control mode. so you can see one of my lessons learned is backups are good. backup options help reduce risk. when you're not sure what's going on or even we think you are sure, and the consequences are high, you made an error, having a backup option can help you through the system. that process, that problem. well, after i got back, six-month rehab, came to headquarters for about four months which is a lot harder than a ballistic entry working in washington with was a lot tougher than any of that training, anything. i really, i empathize with the folks that work here and the difficulties that they go through every day. and it was good for me to experience that it was also great to get back with the shuttle team, who by this time were about a year into getting

with sts-107. my job was as the director of flight crew operations. in the position of the responsible for all the ash not office and my job is to represent the crew in some of the program and senior leadership forum's, represent the crew point of view. and i think i remember from those early meetings coming back at your leader was just how much emotion was in the meetings. i me, people you could see people were full of -- they were having personal problems dealing with the issues, and they're also having personal problems with other people who they might have held accountable, and it would come out in arguments and hot tempered exchanges. and every meeting would take twice or three times as long as it need to do. i'm not faulting anybody. i have plenty of emotional

baggage coming back from space station. that's what humans have to deal with, but it's in times like these that leaders will need to understand that it's more than just a technical things they're working with. they also need to understand how to work with their people. and i thought that bill parsons did a great job of that, and rommel shepherding the astronaut office at this time, getting them into a mental state where they're ready to get back onto the shuttle. did fantastic work. one of the things that we have to do as part of one of the boards i was on, mission management team, was to go through some special training. wayne gets blame for thi this am country until it was really his idea but he's the one whose name usually gets used to at the time i was a very happy about it, okay? i was a very busy guy. i was just trying to cope every day, and i wasn't sure that this training was really helping me much.

and i was thumbing through one of the books one and i got to this diagram. i don't know if you guys can see really well, basically it's from a book by guide named james and, one of these experts that studies how organizations fail. and he's done really great work. it's a really simple picture of how an organization oscillates back and forth and it's vulnerable to accidents, from protecting itself to producing or getting things out the door. and the thing that caught me first was there were three little explosion things here, and i kept thinking apollo one, challenger, columbia. it's got nothing to do with a diagram that that's what hit me first, there were three them. and i thought, this is part of the oscillation has been going on inside our organization. so even though i didn't like the training, and wasn't happy about it, it's probably really good for me, all right? all right? one thing when you're close to one of those big flashy explosion things, the

catastrophic side, you are usually pretty busy and you don't have time for turn. you don't want to learn about this kind of stuff. but if you want any of this knowledge to benefit you, if you want to benefit from the experiences of people before you, you have to be exposed to it. and sometimes you have to make people go off and get that training. so i found it to be really useful, and i thought it was a great representation of some of what was going on at nasa. so the real usefulness of the tool is how can you use it going forward? well, if you're aware of this, you might look for some of the signs at which side of the oscillation your own. one thing is important to realize is when you get one of these inflection points where you have gone to where you build up your defenses quite a bit and then you turn around and you start tearing down your defenses, you will usually have an organization, we are to say,

too conservative, we're not doing enough. you are going to hear that when you're making that corner. when you get down there to the bottom, people are often kind of numb. they've been cutting for a long time. you may see some bitterness. you may even get complacency because people just get tired of getting beat on. so your normal safety bold outcome you tell them no, i'm not going to do that. instead of fighting it he says okay, cut me again. then the accident happens. so less than three, training, good. but another topic today is lessons unlearned. we've talked quite a bit about cycles, and other people's speeches and how organizations learn lessons and then unlearned them. i think this particular model

can be applied a little bit of what we've got going on today, and that is our prime way to get to and from the international space station is the russian soyuz, okay? you have to realize that this cycle is going on in russia, too. there have been signs that they've been on the nasty side of that cycle for a while. they did a lot of cuts and then suddenly they've got to put a lot more soyuz into space. and they don't have a backup option. you know, when shuttle needed to be grounded while we were working through the technical part of sts-107, we have so used to take chris back and forth to station. if we have to shut down soyuz, when the going to have anything for quite coming in, another five years or so, for five years. and that puts extra pressure on them to keep their vehicles falling. maybe even when they shouldn't. self u if something for us the real sensitive to. we need to work i think as hard

as we can to get our alternative vehicle going as quickly as we can, to be good partners with the russians. >> and that's all i've got. it's time for pam to talk about the crew survivability study. >> okay. imacs are going to talk about a couple different things as they progress. and as they say, where you stand on something is usually where you said. so where did i sit? at the time of the mishap, i was also in florida. i was the keeper city, the lead caped crusader that recovers the crew members. so i was waiting for my training to unstrap the sts-107 crowd on the runway. because of that position, when

the reconstruction was decided to be set up at kfc, i was asked to take over as the crew module lead so throughout the reconstruction i was in charge of pulling all the debris, organizing it, and securing it. as a follow-on to that job i stick with the reconstruction team as it morphed into the columbia preservation team that determined what was going to be done long-term with colombia, with the debris and the issues going forward from there. and, finally, culminated with in the deputy project manager for the columbia crew survival investigation. and all of these things really that into each other. i could easily spend 15 minutes talking about each one, especially the technical lessons learned are instead, i decided to try to keep it at the top level and focus on some really big thing that i consider the foot stoppers for all of them. for the reconstruction, we have

seen this picture several times today. one of the things that you never see when you see pictures of the hangar is crew module debris. that is because it is off the picture on the right hand corner behind a wall. so i didn't know what the plan was. i was just told, hey, ksa want you to get there and figure what to do with the crew module debris. and as is not surprising aggressive went back to the challenger model and said, what do we do for that? in a separate room was set aside for all of the crew module debris that was recovered from challenge. and so that's what ksc decide to do, too. so i showed up and there was a small room set to the site. it had a computer and a telephone. and there were a bunch of people who said, what do you want to do? and, of course, i didn't anything what i want to do. so it was a very interesting experience. but i think it turned out to be

extremely important. and one of the things that i found was that when crew module debris as it worked it's way off the truck through quality and finally was brought inside the crew module, that it was exceptionally stressful for the reconstruction team to see it. anything that looked like he was vaguely recognizable and that anything to do with the crew. we got to the point where typically things would get bypassed or fasttrack and brought it very quickly into the crew module and the and processing them separately there, because it was so stressful for the folks on the floor to see that debris. and i think that's really, really important. it's something worth considering. there's just no comparison. the rest of the debris was also very sensitive, but it's nothing like seeing the equipment of the crew. one of the very hard parts about doing reconstruction was trying to figure out what things were

and where they originally were stored on the orbiter. so the photo documentation turned out to be absolutely critical, and it was surprising to me, as an astronaut, i thought everybody knew exactly where nation it was just like magic. if you're on or within the did know where something was, you have a computer program to you could call them, everybody seemed in osaka where everything was. in fact, that wasn't actually the case at all and so that was a very difficult part of that job. and, finally, one of the focuses that i put for the team was to be able to find things quicker but if you want to know where all of one type of material was stored, or where it was kept, how much was recovered, we had a database that at all the debris loaded into it. and so we eventually morphed to a system where we used keywords so that you could do searches and find a certain class of debris, or find a location to everything that was recovered

from a specific location in the vehicle. that turned out to be extremely important for the crew survival investigation and i was glad that we did that. for the preservation team, that was a little bit of a different experience. obvious at the time deeply invested in the debris, and very committed to preserving it. but what was really interesting was the discussion about whether we should bury it in a whole, like was done for challenger. so challenge is at the bottom of the missile silo with a big concrete plug over it. and when pieces of challenger wash up on the beach, there is basically a shaft next to the missile silo, the doors opened up and the piece is thrown in. of course after working on columbia, this just totally horrified me. in fact, there were a lot of us who felt that we should dig of challenger and bring it out and preserve it just like we did with colombia. one of the things that we tripped up against the was the

families of the challenger mishap really identified with challenger been buried. they identified like a crew member. and so the idea of taking challenger out and doing something with it was just as horrifying if we said we think we should dig up the crew members. i mean, they really reacted that strongly. the columbia families, on the other hand, built the concept of preservation into their grieving process. so the columbia family believed that the stories, the things that were being told from the debris, the learning that was happening, the fact that the debris went out on loan for students to study, obviously that the sensitive debris pieces like ohio and so forth, that was a part of their grieving process to say we continue to learn everything that we can. so what i found is the most important messages would end up doing with the debris it's kind

of become a part of the family story. so it's important that you think that through from the beginning. public display was a discussion that went on for a long time. there are a few pieces at a couple of the centers. that decision was made actually several years after columbia occurred, but it is, i think as you heard over and over again, how incredibly emotional people are in the aftermath of the mishap. and people had very, very differing opinions on what the right thing to do with the debris was. and the fact we had opportunity to give some of the debris to the smithsonian who was interested in it, and we could never converge on that. i'm not sure that turned out to be the right thing to do. colombia is now in the dav, and gateway funding cuts and so forth, it's been greatly scaled back, the preservation program. and i think sometimes need to be

better off if you're at the smithsonian. but it is a very difficult decision i think we did the best that we could. and, finally, the most important lesson to me was how much you can learn from the debris. said doug gave a great talk about all the things that the analysis that nasa did look at the debris in trying to understand the mishap in support of and at the same time at caib. but for me i was inside the crew module and it was talking to me. i knew there were more stories to tell. so the caib was interested in crew survival. they formed a crew survival working group to look at the survival gap. i consider this a major advance over what happened during challenger. they were actually several pages devoted in one of the appendices to the crew survival investigation in the caib report, and that was a positive thing. i think it was a lot more

information there than you saw post-challenger, and that was headed in the right direction. the biggest challenge that the caib had come and as mentioned earlier, is because of the nature the commission, everything needed to be made public. and so working through the privacy issues of the family and the sensitivity of the crew died, that was really just a road too far for the caib. they had enough on the plate as it was. so they left that part knesset. they did make -- not a recommendation, and occupation that there should be information and incorporated in the future. the spacecraf space craft crew l integrated investigation team did actually form nearly a year and a half after the accident. and it was more than a year after the final section of the caib report was published. i think the comment that i want to make about that is that

really, that's reflection of the fact that it was really not desirable to talk about. there was good information published in the caib report. of a lot more to go want, but it was extremely difficult to get -- and with all the other things going on with return to flight, to get any focus on it. it was just too traumatizing. and so that really hindered the opportunity to go forward with that investigation. it just was not considered a priority, too many people shied away from it. the other the difficult part about this is that in the military when there is an aviation mishap, for example, which is my background and my assumption, both on i was like really? we are supposed to be doing this. but the flipside to that is you get an outside organization, another squadron, somebody from a different unit comes in does the investigation. because let's face it, it isn't very pleasant to relive the last minute of a crews life over and over again.

so we've been fortunate did not have that option. and i think it is really small community, that's an important thing to remember. one of the things that we did when we started with the crews survival investigation was to start with a challenger data. and what we found is it was incredibly hard to find. it was actually i was taken to a warehouse at jnc and shown boxes upon boxes of papers that were stacked together. and the spreadsheet, of course the time they were catalogued, there was nothing like excel but it didn't really -- that was later added in. but to try to figure out which one of those boxes -- boxes had some nugget and i could possibly want was very difficult. in fact, the most amount of data that would not relative to the crew from challenger was talking to people in jnc engineering.

what kind of look both ways and they would go okay, we go to the office. they would unlock the bottom tour of the filing cabinet, pulled out and handed me pure gold. the best i could get. and it was very, very hard to find but it was incredibly hard. and that kind of in gendered a really strong belief in me that the overweight that we're going to get, to preserve the information that we were eyeing out was to get it published. our investigation uncovered five potentially lethal events that occurred, including some very serious shortcomings in the true equipment. it was very, very hard to get the report published. basically it was moral courage of three people, two of whom are in the room, wayne who was the program manager, mike coates was the center director, and mike griffin. and yes, after all these years

-- thank you. it was really important. [applause] >> one of the great difficulties was that this, that discussing how the crew died was indeed going to be very difficult for the families. we had to consult them. we worked very closely with them. it was extended difficult for them. they showed great moral courage as well. and the great thing for me now is that i know that precedents has been said and it doesn't matter, the next time there's an accident, people will go back just like we did to challenger and find this data and know that they have to do this same kind of investigation. so there's a lot more information in the report. i just wanted to kind of give the top level lessons learned, and that's why he was with important for me to be here

today. [applause] >> so we have time for a couple of questions for our panel. opened the floor for questions. >> i just want to make a comment. pam complemented mike coates and wayne and myself. what people who were not on inside don't understand is the incredible degree of persistence that pam showed in pursuing the crew survival issues to the best conclusion we could get. and if you don't do that, what you are doing is you're putting the next crew at risk. and that was how pam sought. and she pushed through. i want to you personally, at

threats to her career in order to see that you conclusion. so i think pam, the crew that needs what you did well maybe not know what you did, but they will owe something to you, so thanks. [applause] >> okay, and with that, we can go on to the next panel and really just one more round of applause. [applause] >> my mistake. i just love doing benefit what i should have a nice break and we will reconvene at 3:30. [inaudible conversations]

>> [inaudible conversations]

>> [inaudible conversations] >> the space shuttle columbia was lost on february 1, 2003. this discussion is in all the discussion at george washington university. we will continue in about half an hour. with a couple more panel discussions on the lessons learned from the columbia explosion and disaster. we will continue with our live coverage here on c-span2 when they resume but we did want to show you some of the art of discussion, the introductory marks including comments by the head of human spaceflight operations at lockheed martin from earlier today here on c-span2. >> workshop on the columbia plus 10 lessons learned. we are really pleased everyone could be here today. we thank scott pace at george washington university for hosting us here in this facili facility. and right off the bat would also like to thank our sponsors without him we wouldn't be able

to do this. to lockheed martin for their premier support. today we have mike and mary in the audience. boeing, pat am a jeff and bill are here. and dianetics. .. the was created out of our life

experiences. for example, i was at the college of the mainland and some of my colleagues and i were getting ready to run a 5k in the parking lot getting ready to start the race and of course with it being a small community in houston there were other people in the crowd also getting ready to run the and one of them came up to one of my colleagues and said you need to call in to work and the shuttle is lost. my immediate thought was how can you lose a shuttle? it's really big. i had come from a background of working at the aircraft where our job was to work on reducing the rate of the cross section of aircraft from radar and so through that experience that lens of my life experience the was the first thought in my head this is a large vehicle with a radar cross-section how can you lose it? so my colleagues called in to work and we found out that lost was lost in the really bad sense

of the word es and of course we went into the office. so for each and every one of us here as soon as we heard the words the shuttle has been lost from our first impressions our e notions were different from our personal experiences and drove the view of and reaction to the accident. this result in a kaleidoscope impression formed a collective lens for which we as a community synthesized and dealt with such a tragedy. we had to put aside our demotion, focus on the facts, try to sit through and get to the kernel of truth about what happened and why. this was hard. it still is hard. i mean i don't know about you, but i still have some very strong emotions attached to this set of experiences when i think about the memorials that we went through, the constant public discussions that occurred dealing with the dissection of what happened, the meetings involved and trying to move forward and figuring out the right solutions. is still has a strong emotional pull and i expect to experience

that here today. ten years later with our perception is more diffuse with the colors and a jagged edges of our individual lens a word of it, we come together again to view the accident and its aftermath through our collective kaleidoscope. we are here to review what we have learned and make sure we have not forgotten anything if we need to refreshing those lessons and once again examine the tragedy and pull the last ounce of meaning out of it. we do this to honor the crew and others that bring us this hard-earned knowledge. pos accident, there was a lot of activity, and one of the things i was involved in for the astronaut office was leading the return to flight office. i was representing our office opinions out to the technical world and then the head of the office at the time was in the loop about what was going on. the end of the work that was done after the accident from the recovering and ... to the search of the solution this was an incredibly complex process involving a huge team of people,

spanning the country engaged at an incredible level of detail of what happened, but went wrong, let's examine the tanks in great detail, look at aerodynamics, what it takes to break and rcc panel, how to do things operationally better. i think one of the good things that came out of it or one of the things i learned this the fact that when we all pull together in a highly focused motivated fashion, we can achieve anything. it was incredible to me the amount of energy, focus, dedication and hard work that went into getting us return to flight and we did that really when you think about it a short period of time of the whole community pulling together is a very powerful force and i think that is a lesson none of us should forget. so today we are here to review what happened, how we reacted and why we did the things we

did. we are once again going to reexamine the methodology process that we've used to identify and fix the problems we discovered. we are here to make sure we haven't gotten anything important and if we pick up a few lessons with the benefit of the tenure hindsight that's a great thing. we are not here to discuss how they apply to the current times. that is another discussion but that is for a different day. today we have a program lined up to walk us through the accident and the recovery focused on the lessons learned in both the technical and the non-technical sense. most of the panel members and people speaking today were in the middle of the efforts that we went through during the recovery and return to flight and the dissection of what happened. and so we are going to hear firsthand from their experience in some of the lessons they've learned. thank you for coming on behalf of george washington university and the aiaa. i hope you learn something and i hope to learn some new things today. i think it's a fitting way to honor the crew and their

sacrifice that began to these lessons and make sure we don't forget them. what i would like to do right now was introduced mr. w. michael hawes. his biography is in the book. one thing i do want to comment on, he's currently at lockheed martin. i asked him how i should introduce him and he sent me a note back that said some old guy that used to work for nasa and now works for lockheed martin. i said i can introduce you like that but i would like to add mike has had an illustrative career at nasa. several positions starting at johnson space center and then several different positions at nasa, and i won't go into detail because that is in his bio. currently he's the director of the human space flight program out of washington for lockheed martin. mike if you would like to come up and make some comments, that would be great. [applause]

>> we agreed to exchange bullets ahead of time but she went first and got all of the good ones any way. so, you get to listen to me because of the sponsorship thing which is one of the things the we do in washington, and it's always a privilege to the alta sponsor these kind of things but for me it's also something that's intensely personal kind of as sandy said about where we all were at that time. i was actually probably one of only a handful of folks in the room that actually was sitting in mcc for the first launch of columbia, so columbia herself had a large part in my life and really it touched me in a lot of different ways. the accident was an odd time for me personally. we all have a more personal. i was on sabbatical i was supposed to be working on a doctorate that i finally drug out of the university years ago.

i had been following the flight only superficially. i wasn't at home watching tv. i was out and about with my son and i didn't know until i got a call at the headquarters that said the you think i need to go into work? i don't know why. he said you need to turn on the tv. the shuttle has been lost. so by that morning i had already talked to bill and was in the headquarters with the primary debate of the morning. we were going to put david on a plane but we were not sure where he was going to land, and that was the we know we started today and i served as bill's assistant through the return to flight time frame so i have seen all of these different pieces. there were lessons to be learned and this whole spectrum of activity not just the accident and the proximate cause itself, but as sandy said, the response

we had, the interactions of all the agencies that came together to make that happen. i also would make a note and ask you as we are going to think back and reminisce and think about our friends and the crew but please also remember buzz and charlie that lost their lives in a helicopter accident during the search. our tribute is how we learn from this and so the topic of today is critical. and i think that as sandy said we will have another session to look towards the future. most of us are still working on the systems headed towards the future. so we have to take these lessons learned from these discussions and apply it to what we are going to do. in one of the memorial services, like remember very distinctly john clark the question of whether we were going to be a space very nation or space

fearing nation. i like to think that we are still all headed to be a space bearing a nation. i fervently personally hope that we are to be a but it's only by continuing to focus on these things and learning the lessons that we can and apply them in the future that we will be able to maintain that. so, with that, i would like to then just take a minute and introduce joe dyer, our keynote speaker. and in the sake of brevity, we all know that joe has had a very long and distinguished career both in the navy and in the industry most recently serving with ai robot but we must know him for his very long serving dedication to the advisory panel. a really critical function. when you've been through the wringer and you've had the pleasure of going through a

number of times to see your career you do respect what the team does and how much they bring to the debate particularly in the cases like this. so it is my great privilege to bring forward the vice admiral joe dyer. [applause] >> thank you very much. i have good news and bad news. the good news is that i have 20 minutes if information to share with you. the bad news is i'm from the mountains of western north carolina and it takes me 40 minutes to do it. if you grow up in the flight test business and have years of western desert this is a picture you have seen played out many

times. it influences your world view and your life. it builds an innate conservatism born of pain. columbia ten years ago, hard in some ways to believe, but i am confronted with it almost every week. if you come in date to of the river and turn right towards the exchange, you will see what is a growth of saplings planted in memory of the lost crew. today, 12, 14 feet high. it's a visual representation of how quickly time goes by. and also if you serve on the asap it's a reminder of how important it is for the passed to influence the future. colombia was an amazing machine.

the shuttle's all amazing machines. first left in 1972 to north american aviation. few of us remember north american aviation. many of us don't remember 1972. it was the first shuttle that was the. 27 successful missions, lost on the 28. looking around this crowd and visiting with you this morning, i have no doubt that many of you knew we had had personal friends that served on the crude. in their lives they've made nasa better. like the first speaker, i always have to struggle a little bit with my own memories and my own

emotion. william was one of my guys and he was a joy that would illuminate the room. i miss him terrible to this day and similar stories could be told about all of these wonderful folks. this may surprise some of you coming from the safety guy, but i want to pose the question this morning how safe is safe enough? but i don't want to come at this from the perspective of just more safety. as my congressional so points out, those wonderful exporters that expanded our world and the 1400's and 1500's wouldn't have left home in an environment of overarching safety. indeed, great reward and the pursuit of the great reward

comes hand-in-hand with rest and it's this relationship between risk and reward that has to be appreciated. are we today at nasa and as a nation too risk adverse? i think so. but it is a topic that the panel has focused on with great energy. we are supportive of accepting more risk if we recognize it as what it is and the risks that are associated with that, very acceptable. unacceptable is to accept more risk and pretend you're not. certainly the asap as constituted today recognizes that it is not line management and it's certainly not to answer this question of how much safety

is safe enough. but it is our focus and intent to point out areas where we believe the state requirements may not produce the requisite safety, the required safety. a bit of an eye chart let's talk about the important parts. this is great work done by bill and his crew taking aboard the wisdom of 20/20 hindsight and posing the research question knowing what we know today what were the risks statistics, how probable work you to lose a shuttle as you look retrospectively? i will not drag you through the detail but i will tell you retrospectively the first watch had not lost potential of more

than 1,000 but probably a loss potential in the neighborhood of one in ten to lead to orders of magnitude below the design are. tool orders of magnitude. that has to send a message to all of us about margin, design, what is built into a system to take care of those things that we don't yet know and we don't yet understand. a bit of an off line for my remarks this morning we've already talked a little bit about risk and reward. we will do that some more as we go through. i want to tell you a little about the asap and may be introduced some of the people that are a part of it because it is a good group that works hard in support to nasa and support of the congress. as a keynote in just not to be a

spoiler but we will touch a little bit on the return to flight. then i want to look at issue is going forward, the nexus between how history needs to inform our future. we will have to talk about budgets. please try to restrain your enthusiasm. last, let's talk a little about the directions and alignment. if you look at the charter, it's all about speaking to hazards, operations, safety standards, culture, other duties as assigned by the administrator and an annual report submitted to both the administrator and to

the congress. coming out of the accident was a congressional the mandated director that said the asap shall in its annual report examined the compliance with the recommendations of the columbia accident investigation board. the panel members include myself as the chair, dr. jim as many of you know, a mechanical engineer, medical doctor come astronauts, air force 06 special operations officer. i tell jim if ritalin had been available in his youth he wouldn't be as accomplish as he is today. the air force retired to start army acquisition executive one that really understands the interplay between risk and

acquisition strategy retired captain bald conaway who's now interestingly enough get to know him he's the director of safety for disney and i secretly suspect the path to those tickets. john frost, long, long time safety expert at the arsenal probably knows more about bad things that happened to rockets than any human on the face of the earth. she is a fellow at l3com, former chief engineer for naval aviation, dr. george monreal who is with us today. as a matter of fact stand up so these folks can see you. george is an active duty i would say civil servant, long

experience in both air force and nasa. we are blessed to have him. many of the future questions, future opportunities to the answer route commercial space are in george's portfolio. and in the finest tradition she is both the regulator and advocate. where are you, stand up, sir. former astronaut, will never a former marine, director of safety mission insurance brings deep, deep knowledge to the asap and a very happy to have him. dr. sanders, patricia comes to us from ballistic missile defense after a long tour the

department of defense she has as an expert in the test evaluation in her tour as an executive ballistic missile defense office could probably includes the rest of us in the roundup of the budget. this has been an interesting week preparing for this pitch because it's not often you have the occasion to stop and look back ten years and to try to integrate the cost of those things that you have experienced but may not remember. those are the covers of some of the asap reports. if you look at the white one in the upper left-hand corner, the focus then it was all about a return to flight, and this handover turnover between their return to flight test group and the asap. the next, which was 2008 upper right-hand corner will all about consolation, it was all about

aeires in the classic sense. the bottom left-hand corner, 2010 the topics were do we extend the shuttle, could you extend the shuttle, could you extended the international space station? and then the latest report available online by the way has much to do about commercial space and the exit strategy and safety. just a bit of the timeline you will see the accident very quickly followed by the admiral who is with us this morning. good to see you, sir. the return to flight group that had a return of something

approaching 18 months much longer than they had ever expected. you will see that the consolation had a long life that goes from the middle of 2004 to the end of 2010 you see evo watches as they were dispersed following the return to flight, and then you will see the beginning of the commercial space program. if you look at this chart if you look at those covers, you have to be struck by how many different space programs we have had over the last ten years and i will tell you one of my overarching features is another administration irrespective of who they are or where they come from another space program. if that is the cases would be a

long way to mars. the recommendations 15 of those were designated as critical, my word, to return to flight. the return to flight test group that was led by tom stafford and dick covey looked at those recommendations and said in a country boy terms we can beat 12 of them and three of them are just too hard or impractical at this point in time and at this point in the design sequence. those three had to do as you well know with the external tank debris and the leading edges and of the thermal protection system inspection and on orbit repair.

while these were too hard, we did elect to return to flight. there is no doubt that the system was significantly safer after rather than before, and one could say the things that really needed work or at least made a lot of us sleep better was the concept of rescue and the ability to launch on need if that were to be the case. planned initially for the first flight or to but carried throughout the program until the very last launched to the switching gears less looked today and look forward. these are the issues color coded as in your stoplights green

being good and ready to go, yellow being some work left to do and read being a serious concern that could adversely impact the success of the program. we will talk about commercial space as we go deeper. i have put out of scope the expiration system development program except to say that in our opinion it is progressing well. it is relatively well funded. these are the other opportunities. it's still going to be a long way to mars but it's progressing. we will talk about uncertainty but it is the one flaming red. real progress in the international space station i will mention that briefly. technical authority, technical authority is a topic we are not going to dwell on today. it was a topic dear to the heart and the return to flight group.

i hope it gets a good discussion today. i will say only it is an area that we worry where the asap worries there maybe some regression and the need to intent authority. we will talk a little bit more about risk-management. so what is the purpose of commercial space clacks is it transportation to lower orbit, is it to return the united states ability to launch folks to and from the international space station and other undertakings or is it to develop an industry to do economic development? well it started out on the left, took a tour to the right and found itself in the mill that says let's to both. pragmatically, one could have predicted this perhaps but i

will point out that safety of various as you might read from left to right. if all you are doing in the economic development, feel free to be capricious. stretch, innovate, try new things. if you have to haul humans to space and you have a different responsibility. i want to talk about the question of how safe is safe enough and put just a few numbers on it. in aggregate for the mission the system will have to be as safe as the shuttle on a 12 day mission. at least as safe as the shuttle. so we pose a question is the bar high enough?

you could propose another question that says if we suffered a loss rate it can to the shuttle, will we have the fortitude to press on without extended hiatus? for those of you that are right branders, let me give you the same chart in graphical form. looking in round numbers at a threshold has in it has to be of one in 300 for the loss of crew and a migration with the intent of making it better to work towards a one in 1500 loss potential. this is for the second infantry. here is the same chart for our degette mission on the station included and you see here that the loss rate now targeted

design in the neighborhood of one in 150. this is a program that came out in mcdonnell douglas over 25 years ago. it's a chart that talks to lifecycle cost but for the purposes of our discussion today, i submit to you that there is a relationship between cost and safety that one mirrors the other. how many of you are familiar with this charge? a few. let me spend just a few minutes to talk about it. the bottom curve looking out -- it looks at the way that we spend our money. it says that look, when you start to do the design of a new system or if you want to think

about it in home terms if you are going to build a house with an architect with some design and, not a cookie cutter house, you spend about 10% of your money in the design phase, you go into construction or production and spend money like a big dog and then it goes into this long really far extended operation and support of the care and feeding and maintenance of the system. we all know that curve and we are pretty comfortable with that but let's look for a minute at the curved above. this speaks to the importance of early design. it says that by the time you have spent 10 percent of your money, you have preordained 85% of the cost and i would say safety of the result system.

what we used joint strike fighter for you for a minute. in their earliest days of the joint strike fighter there was the determination of the single pilot to become a single engine, it would have a specific set of radar cross sections and a logistics scheme and avionics architecture. when you have done that, you have preordained the cost of the system and god help you if you think the you are going to fix it later on. this experience makes me suspect of the concept that says when one in 300 early on the we are going to make it better leader. that is an incredibly hard job. this is the current acquisition strategy.

it's something that we can spend a lot of time on degette we won't do it today. i will tell you that it represents both real progress and some residual worries. real progress is that the asap has been saying for years nasa, you have not clearly communicated requirements to industry. you haven't communicated what it is you want, you haven't defined the success criteria, and in doing so, you are at risk of committing the program and let me tell you what i mean by complicated. one could see a future where nasa working under a space act agreement is at arm's length from industry. so as to support innovation and the concept, so as to support speed and to leave the industry unencumbered. but nasa said we are going to

certified system for the man flight could i will pause here for a second so you have tremendous freedom to experiment with cargo but man flight is a different kettle of fish. so we worry about nasa getting in the box where the industry says here is our system that we have designed, and nasa says that's not its and the industry says it would have been nice for you to tell me earlier. so we pressed hard on how do you certify and communicate certification? the progress is that there is now in the plan of the ability to do phase one which develops a plan and then phase to get down to the details of the certification.

the asap has the hair on the back of their neck standing up with regard to what's called the optional milestones. the pressure grows there would be in tent for industry for the under space agreement to demonstrate test supplies. i don't know about you, but i don't think there is enough money in the world to take a statistical approach to flight safety. i think you have to have intimate understanding of the design. so we are a little suspect of what might go on in this optional milestone as we look at the budget pressures coming downstream. there is a decision and we think

a very positive one to say that a formal federal acquisition contract shall be part of the approach to strategy. there is still a debate about the contract type. i will tell you at the end of the day i don't think it matters >> we have a problem with our live coverage -- >> the name is the original contract -- >> from earlier today discussion on the tenth anniversary of the columbia disaster we are going to take you back live to george washington university where the discussion will continue and wrap up this afternoon among the expected speakers the former admiral who was the chairman of the accident investigation board and later, the former director

and space shuttle program manager live coverage here on c-span2. >> again ncnb said at the beginning, all of us live the accident through our own personal lens. i will share my and. we have bryan o'connor, who is the former chief of the safety assurance organization. bill parsons, the space shuttle program manager for the return to flight who had to incorporate a lot of these lessons, and dr. miller of george mason university who write to the corroded dhaka and columbia who shamelessly stole the title symposium from. i figured i would start with just discussing some of the personal observations and then a little bit of a synthesis. as i said, you know, these are sort of my personal impressions of the accident.

frankly i was asleep and when the phone rang and i got a call from a friend of mine at equine evac representative kennedy and waiting for columbia to return and said the vehicle was late. as sandy said you don't lose shuttle's. something is bad. so i had to get into the office fairly quickly. and on the way they're of course we had the reports of the breakup occurring in the sky over texas. we set up a room on the sixth floor. a deputy administrator gregory had come in and a lot of folks of course had come in that weekend and we had a crisis response plan set up for the notifications and coordination said several of the earlier speakers have mentioned to read this was created after the challenger accident and this was something we were grateful for in the organization for the first few weeks after the event.

the plan only took you so far but in terms of providing a structured organization, initially it was very important to have those contingency plans in place. the next few days and weeks one of the immediate issues that came to my attention, those in the headquarters was the damage on the ground, the effect of a global community is we have schools the proposed and roads that were blocked and local police forces guarding the debris sitting in the middle of the streets in the middle of the school yards. i worked with the chief medical officer rich williams who was invaluable in working with some of the state and local authorities. rich was a flight surgeon but he

also had a role not only taking care of the crew were reading about the loral but it formed an interdisciplinary function where you have people better manager really responsible working with first responders, working through david king toward meeting in d.c. so you would have these amazing telecom's everybody was pulling together to do the right thing. we were concerned with civilians removing of the debris. we didn't want any more harm occurring to people on the ground, and of course we were worried about the potential data being compromised as the debris was removed. it was quickly apparent the major breakup had occurred sooner or later in the more densely populated areas had come so on the many miracles was the fact that no one on the ground

was killed or injured although we did lose to folks leader in a helicopter accident. we intend occasions with the white house of course and we were tied in very closely and dave mentioned how the was helpful when clarifying the chain of command early on. we also needed to organize. it was going to be an in-depth investigation. admiral gates was announced as the chairman. again i think that the predilections for the navy were certainly known but was also not an accident that we had a lot of military aviation people on the board because we were looking for the best people in the country that we could get for what we need to be an interdisciplinary sort of analysis so other names were added later as the doctor described but we were trying to pull on with the rest of the

country had to offer. in the course of the activities from my particular novel i had the unusual duty of chairing the final approval for before releasing their responses to the increase. it's quite a process and it worked i think quite well. the tone was set early on that we would be as forward leaning as possible and release all material prior to the accident itself. materials released with country conference room and binders and they would get one last look for personal identifying information, people's home phone numbers, social security numbers, that sort of thing and ensure the public affairs people, congressional relations, technical and the specialists themselves new law was about to be released and in the approval the material would be uploaded to the nasa website for access by the press. it was sobering to see material on your desk one day and the front page of "the new york times" the next morning. legally it could be argued we didn't have to release as much

material as we did. we had a number of exemptions that could have been used but it was a top-level decision that i think was a wide one in setting the tone inside of the agency it would be as bluntly honest and frankly we knew the people would see things and connect the dots probably faster than we would initially and that was to the good, to find out what happened and why. a week or so after the accident as we were in this process, the nasa general counsel was there and he was a lawyer from louisiana, longtime friend of the administrator, smart guy. i remember saying to him before this is over you are going to hear a phrase called normalization of deviants, the turn from the ibm's masterful look. i won't attempt to imitate his

accent as he looks back at me and says the normalization of what? but rather of sociology term certainly became clear what we meant. when the theories about the phone shedding a came out you can see the skepticism and its first hand i think on one hand there is a reluctance to increase the theory because as one of the other charts said first information is often wrong and there is danger in organization finding a convenient answer and locking in to it soon because everybody's looking for the manning to impose patterns may be where they don't exist and so the point the was made about chasing down all other possible avenues is important to be intellectually honest. i personally got the analogy that i was told by a number of people that couldn't possibly be like a styrofoam ice chest thrown off a pickup truck and a hits you and it shatters and

it's scary in the impressive but it's what can i do, it's been a maintenance issue for years. of course was more than maintenance. michael greenfield, the chief engineer showed the historical charts on the underside of the shuttle and i was trying to figure out what this was in the context. it's been awhile since i was a real technical rye. do they exceed tolerance levels or margins of some kind? and he looked at me and she said that's the problem. there isn't supposed to be any strike on the bottom of the vehicle. it's supposed to be zero. the standard is zero and then you have the moment of my god. in short it was performing as expected. but the system as a whole was exiting a dangerous behavior that we didn't recognize. it was a complex highly complex system and we failed to listen to the vehicle and what it was

telling us. failure in such occurs not just in the parts of the components but crucially at the interface between the major system elements. that is the first place to go look. i was reminded when i was a young engineer and rockwell when the challenger accident occurred my first thoughts experienced several explosions in the testing in the 1970's and of course i remember being surprised and seeing the design that was quite different in the propulsion systems that i had seen earlier. it's not just ignorance, it is why you believe to be true that is not that can be even more deadly. so a lesson i would take away from the columbia accident and other space and aviation accidents is to have a degree of humility in front of the hardware that is listen to others who are paying attention. by definition the failures are going to occur in alladi and

unusual ways when the reality trump's our assumptions about what we think is reality. with the thought i would like to pass it over. >> can you hear this in the back? my name is bryan o'connor to be the safety mission insurance officer for the agency when this accident happened. i had been in that job for about six months. when i think that on the lessons learned put them in about four different categories. and they tend to be in the root cause, the organizational, the cultural part of what came out of the columbia accident investigation board. i was struck by a quote that i saw that says do not look where you fell, but where he slept. certainly the cave looked at where we fell and also where we

slipped to read some of the things the we looked at affected my organization and me personally quite a bit for example, today when i am watching a movie of lee roy and all the other folks in mission control, several times he asks the question well, is their something, and? the failures that you have just seen in the three parameters several times to ask the question do they have anything in common and the answer came back know there's nothing in common. in retrospect of course they all have something in common. we all know that now but it reminded me of the importance of doing analysis of your design not just from the failure themselves which tend to drive the flight rules and so on you look at the different failures

and figure out which ones can happen, what kind of things lead to those and that gets to the question of what do these things have in common. you also have to do top-down analysis and that is the purpose of what we call hazard analysis. top down and the bottoms up you're not done. you want to do sideways analysis which tends to be things like reliability or the risk analysis or accident analysis. these are things the safety community i was in charge of our responsible for helping programs deal with their contractors and engineers especially with safety or systems engineering integration people because that's where a lot of these end up being important. anything that can lose the vehicle tennis to be in my opinion and integration issue. again, and bigot section on

integration. as someone earlier mentioned, it had dwindled quite a bit over the years. after all the 87 successful flights. scni, which shortly after the challenger accident was beefed up to significant levels including those that fell over the safety insurance, the integration safety efforts. i guess a was a different but there was an awful lot of work done after challenger to baseline all of the analyses so one of the things i learned from this is the you cannot just depends on certain analyses. you need to have the team and that is credible and capable to ask why are you sure this is going to work the way it does. the design engineers like to do things the work you do need to have somebody there the last

white and what if it doesn't. that is the purpose of the safety michigan team as they ask those tough questions. over time that had dwindled as well, the seeking mission insurance. part of it goes into the second day and i put these in which i call flight tests. after a period of time we got into the mode where we considered ourselves operational and therefore we didn't need to think like flight test people as much as we did before. we could start phasing out of that mentality. we can tell the engineers to put on their maytag repairman uniforms and we will give them a call if we need them. we sure don't want them hovering around everything because that is expensive and they will find

something to work on and there is attention. you don't want to do too much. you have to keep flying. you can't afford to do too much better over time we got to where we were feeling like we were operational and we were no longer in a flight test mode and i am not talking about the columbia accident. i'm talking about the challenger accident. we were in that mode before. we got into that mode again. sally ride said she had a reminiscence of challenger things when she was looking at the accident, the columbia accident. we had flown of more than 86 flights since the challenger pt 86 or 87. they were very successful by and large. we handle the technical matters of them but by the time of the challenger accident, we couldn't say that we were in the same flight test mentality that we were by the time we got to the

columbia accident that we were after the challenger accident when we said we will never do this again. it's not really operational. this is tough because after a while you get to that bit the says aren't we finished learning yet? the anomalies are fewer. we aren't having these big problems showing up. at some point can we back down and get the cost down on this so we can do other things in the agency? it's always going to be a pressure. but one thing we ought to watch out for is something that i saw in augustine's laws and number 25 is interesting. he drew up what that showed how much flight test was done for a whole bunch of missile and rocket systems. he did this back in the 80's so he missed out on some recent ones. he had rockets and missiles down

from a dollar 98 hand-held rocket all the way through the tight end missiles and there were about 20 or 30 different rocket systems and missile systems and he lead goes down into his chart. cost of a unit versus number of flight tests that were done the number of flight tests was inversely proportional to the cost of the net so there were fewer flights done on the tight end for example than there were on a very small relatively simple hand-held rocket where they did a thousand test flights before they declared it operational. this isn't the way your plans work when you go and look at the defense department and the faa, they are looking at many test flights before they approve something for operational use

before it is allowed to actually conduct operational missions. in human space flight we were not following and we never will follow the mode when it comes to this matter. what we have to deal with is we cannot wait to do the requisite number of test flights for complicated systems to then declare them operational. what we are going to have to do is fly our missions relatively early in the flight test phase and admit to ourselves and be honest with ourselves we are doing flight tests for the life cycle of the system, the system meaning the space shuttle for example. we were learning stuff and then the flight test mode until flight 29 and that's a good. it's expensive but it's the right mind set and i thought that when we wound up this program we had a very good mind set on this.

they had enough money to deal with problems that came up. the engineering community was very involved. they were not driving the cost as much as they probably would like but they were not to the maytag repair guys either sitting in the room waiting to be called up. the reserves the program manager had were sufficient to do the kind of detailed analysis they needed when things came up in flight right up until the end of the mission. when you go back to just before columbia, there was a record low amount of reserves and what they called apa in the program the data for the columbia accident. it had never been that low. there was very little money for the manager to use to achieve some problems. i'm not saying it's the money people, saying it's a big problem across-the-board. everybody shares in that. i touched a little bit on something called check and balance. technical authority is the fer bucket of lessons learned.

again, the columbia accident investigation board compared with a saw with the shuttle program with three or four other programs and institutions that were doing a lot of engineering and a lot of programs, and they didn't like what they saw when it came to the authority of the technical personnel with respect to the program manager. the balance have gotten out of whack even within nasa. i think when we look back on it and we realize that balance had gone to where the program manager had a lot more authority than maybe he did earlier in the shuttle program in engineering was much more involved. and much more authoritative on the matters dealing with technical issues. so, it's very good to get to where your engineers have the authority to deal with things but that's not enough and i can tell you why spend the whole rest of my career in that job as the safety mission trying to

make sure that we have the capability that went with it. there is nothing worse than having people with authority that don't know what they are talking about. it's very disruptive. if we are going to get this and have technical authority, we have to get good people in there and i have to get into the center lectors, the really turned and got us some good people in human space flight safety mission insurance organizations and that was very helpful to the program managers. you can ask them, but if they had people who knew a little bit about what they're talking about it was easier to ansar. the last bucket is complacency. some of the of the things i talked about, not doing enough analysis, not having the right amount of technical authority and checks and balances, not having anything in your research plot to go and shes problems with to get all these things are

signs of institutional complacency. it's hard to say that there was anybody in the shuttle program operations or program office itself that you could point to and say that person as complacent. but there was an institutional complacency that said again we are not in flight test mode we are operational. we don't need to have all these people helping us do the job and that is a form of complacency you have to deal with for the life cycle of a system like this those were the general lessons i got out of this business of looking where you slip and not where you felt. thanks. >> im bill parsons, the program manager for the return flight. on that particular day i was the director of the space center probably had been there just less than a year, maybe less than six months even. it was a saturday morning and i was driving over to pensacola.

a high school friend of mine was in such bad shape he was in a nursing home soil is going to go visit with him over the weekend. and of course a call on the be there because i left pretty early in the morning. i was able to go in, tell him hello, watch it on cnn and then go back to the center. of course i was in touch with some of the folks back at the center. they were already looking down information and starting the process that was in place for us to be ready for an investigation like we were fixing to go through. i didn't have a clue what i should do it and i picked up the phone and i called the j and he was a former director of the program and i said what do i do? she said get back to the center and wait for the call because things are going to start happening really quick after this. our relationship -- i didn't know the crew very well but i

did know mike anderson. we were on the selection committee together. we did a lot of the interviews he sat next to me and i got to know him well. it's just it becomes personal when you have somebody you are that close to so i can understand how some of the other folks felt that new the crew better than i did. >> i want to go back a little bit and tell you that i was the deputy director of the johnson space center when we were the lead for the human space flight. some of the things ryan just touched on i observed as the international space station was in the cost overruns and things were happening we were making decisions, and again, i was part of that along with a lot of other people. we were making decisions on how we could cut the cost and the shuttle program became -- it looked more operational to most people come and we were cutting things that we didn't use very often. some of that was the analysis that we would do. we weren't doing that kind of work and so we could let those

people go because apparently, we didn't need to do that kind of work and we were losing that expertise in the shuttle program. i don't think we realized it, but that's what was happening. another thing that i would tell you is she's done more as the shuttle program manager. before columbia he announced he was going to retire in the bill was in search of a new program. as the center director, i was consulted a number of times and i'm sure some of the other directors were as well on the names on the list. i can tell you at that point in time if it was on the list it was away at the bottom of the list somewhere. so there were a number of people that were being considered to be the shuttle program manager after ron retired and a few months. he was supposed to retire i believe just a little laughter columbia happened. he stayed on a few more months after that happened but then eventually left but i will tell you that as i said i didn't think that i was on that list

and i will tell you a story about that in a minute. i do want to tell you one thing that bob once said to me and maybe it is a lesson learned, maybe it's not to be that's for you to decide. .. it's something to think about. during recovery, dave talked about that. i will tell you everyone pitched

in and did many things. one of the things i thought of and thought that we should do is the stress on the people doing recovery was great. it was huge. dpaif didn't touch on it too much. it was huge on dave having to set up that whole organization getting it kicked off. and in the meantime, he also got very sick, and, you know, he did -- i think he left with a pair of socks and didn't have a lot of stuff when he got there. so we offered up alan flynn and they are rotated in and out and i'm sure other people did too. it was long days and a lot of hard decisions. you needed to be as fresh as you could and you needed to have people that was aware going on. it was one of the things i felt was an important thing during recovery. you have to backup and people willing to get out there and, yes, you have to pointmen,

pointwomen, whatever. you have to have people willing to go in and make sure they can jump in and help you out and give you a break. let you get refreshed a little bit and get back in there. yeah. so how i got assigned to be the program manager, we were doing a lot of recovery and somehow i guess had gone down the list and asked off names asking people they said no. and one day we were having a little bit of a appreciation ceremony the community had done hard work. we were giving out awards and patting them on the back. i was telling dave king you have to -- it's a tough job. and he was smiling. he didn't let on to me that maybe he knew something, but that evening we were sitting over there was a can dough there

was a few sitting around talking. everybody went on the porch, and i walked in to get a beverage of some sort, probably a beer, and i heard the door open behind me, i felt a presence on me. i turned around and there was sean 0 kef. it was me and him in the kitchen and he said, bill, are you red destroy step up? i said, sure, i was in the marines. i fell in to marine mode. yes, sir, whatever you need me to do. >> and i gave him a beer. >> he said what would you family think? i don't know what we're talking about. maybe i'm going to marshall center director because he's going to be the center program districter. that's how naive i'll be. i said, sir, they'll adapt. i'll figure it out. he said good and walked off. i was like, well, -- [laughter]

so about two days later acalled bill reid and said what was the conversation about? basically he said you're the man. that's when i found out i was shuttle project manager for return flight. i will tell you, i didn't feel qualified. i didn't feel like there was any way in the world i can do this. let me tell you that it was not -- i know it was not my skills in the shuttle program. i worked operations at kennedy space center much like dave king. i didn't have a strong technical background. i had been a marine. i had done some -- i had been a leader in a number of places. i was thinking that's why i was allowed to do the job or asked to do the job. but one of the first things that they did for me is they said you can pick your team, and that pretty, you know, when you find yourself in a crisis like this

and you put somebody in a leadership position, to give them the full range of you can pick your team, you can have anybody in the agency and they are available to you. and i sat down with then senate director of the johnson space center and a few other people in the agency, and we came up with the names. of course, that's where wayne was picked, john shannon was picked, john was picked, steve. i can go down the names. nancy curry, john casper, i mean, i got the best, and i will tell you that was an extremely important lesson for me. when you find yourself in a position like this, you get to pick your team. i will say there's a bad side of this. everybody in the positions before that had been a part of the shuttle program that had the history that had the -- they

just learned a lesson probably in the hardest way that anyone could ever learn it. many of the people moved on to go to other things. we were a brand new team coming in with a new perspective. we were trying to get our legs under us. luckily some of the people that were in the positions before stuck around with us and talked to us and told us what was going on. they did not back down. they did not fade away. they really did help us. we needed their help. that was really important during that time period. there a lot of other names i can go through and a lot are in the room today. and touched on something that was probably the thing -- as i went to the job, i had to change the culture. that was my, you know, it was changing the culture, and get in there and be transparent, open up, have the conversations, have the tough conversations. but as you heard earlier, there was still a lot of raw emotion,

and there was a lot of disagreements. we had to work our way through those every day. i can tell you it pushed my patience to the limit. to a limit i didn't realize i even had. yet it was required. we had to work our way through that. a lot of difficult decisions were made. i remember people in tears in our requirements. i remember people walking out just, you know, they're done. they're not going -- they can't stay anymore. so we had to go through that process. i'm not sure that it was -- there's nothing to be learned from that other than the fact that you put that team together, you put them in a situation like that that they have to go through that process. it just takes time. you're not going get back -- we were saying we're going go back to flight in december. we're going go back -- yet it was not going happen. and the team knew that, but that

was our messaging we were doing at the time trying to continue to keep funding, i guess or support from congress. i'm not really sure. whatever it was, it was the message. we were having to work through a lot of issues during that time. one thingly add is even though we were changing the culture and there were a lot of support for changing that culture, i was still around great deal of pressure to get return to flight. the message to me is if you don't get it done soon, we're going shut the program down. i got the message a number of different ways from different people. i don't know i don't -- that wasn't a threat. they really believed it. and i think that, you know, that created a tension in me that said, i've got to, you know, i don't want to be the guy that is shutting the shuttle program down. if that's the way it works, that's the way it works. i had great support from the team. and i named off many others that

i didn't name but just unbelievable people. people that just stepped up and did their best, and again, and as you look at the names who became shuttle program managers later on wayne hale, shannon did a great job. as many things as we got right, we got a few things wrong. that's a lesson learned as well. we rolled out to the pad and we did a debris summit and found out that the feeline needed a heater and it could build ice. it could do the modification. it was ready to be installed and all the numbers said you have to install it. we rolled back from the pad to do that modification. there was another long and hard discussion and it was about the power. the ramp that runs down the side of the external tank.

it's a lot of -- it's a lot of foam that could have come off. if you recall, it ask come off. i can go through how and why it came off. it doesn't matter. i can tell you that we as a team talked long and hard about taking it off, yet, it would have been a great -- big delay to the program. and we had a lot of evidence that said it didn't shed foam. we weren't going to have a problem. we decided to fly. you look back and think it was a tough lesson right there. we knew the answer. we knew what we should have done. yet we talked others ourselves in to it because we needed to fly. we thought it was low risk. turns out it was a higher risk than we anticipated. i guess i don't really need to smiers. there's lessons in there you can take them, you can think about them. they may apply. they may not apply.

they were things i took away from the experience. and it was the hardest thing i have ever done. i can honest let me tell you that i grew old it felt like during that part-time -- time period. at the end, it was a tough thing to bear. i'm just telling that from a personal standpoint. i needed -- after it was over, i needed to get away from it it. it had taken that much out of me. somebody else needed to carry the burden. when you are in jobs like that and they are that stressful and you have to think about making sure you have somebody ready to take over for the next guy. that's an important thing. that's all i have. >> this is really unusual where the academiaic has the view --

[inaudible] all right [laughter] it doesn't happen. >> i think it was sally ride in the cave who said there was eerie similarities especially between the organizational elements preceding the challenger in the loss of the columbia. it got me thinking about organization learning. i'm an political scientist. it seemed to me that there was a lesson to be learned here. how could be that the organization like nasa was not a learning organization? that doesn't compute. it doesn't make any sense. so i started to investigate how this could have happened. what i'd like to talk about today is the relationship between -- how unlearning and forgetting two down sides of

organization learning might be connected with normal accidents. most of you probably know that a normal accident is the inevitable consequence of having technical systems that are complexly interactive and tightly coupled. that's is there are unexpected and unpredicted interactions among parts of the system, and that once the interactions begin the system cascades to failure. there's no buffering, there's no way to stop the process from continuing on. his solution is well, back away from some of those kinds of technologies. and we see that kind of normal accident happening in a lot of settings. katrina is often characterizing that way. i'm suggesting here that normal accidents don't just happen in technical systems. they happen in organizational systems as with el.

-- well. that's sort of the kind of situation i want to talk about. you can have organizational systems that are complexly interact i have tightly coupled and have devastating consequences. there's too much on this, i know. bear with me. when we talk about connell mention interactions in an organizational system, we are talking about policies and standards, procedures that can become complexly interactive either because they were poorly designed to begin with, or because bit of them get lost over time. so with ocean unlearned parts of responses they had in the past when they forget parts of responses that they had, solutions they had in the past, suddenly systems that did work

suddenly become complexly interactive. when you have highly centralized and highly formalized procedures within an organization, this can lead also to the notion of tight coupling. there's no way to stop the process once it gets started. no one has the authority to stop it, nobody has the information to stop it. the systems are formalized and centralizes. there are analogies to these notions that the about antecedent of normal -- these organizational systems and that what we think of is unlearning and forgetting are major contributors to the sort of normal accidents. how do organizations forget? by forgetting, i mean, the intentional loss of lessons we

had. it creates gaps what we think we know how to get them down. it generates the unrecognized complexity. the cause of forgetting, you all know if your own lives, how do you forget elements of procedures that you used to know? you reorganize. that's the best way to do. it any time you reorganize you essentially paralyze it for some period of time. because most of what organizations do, and most of what they know is not written down or if it is written 0 down. it's not what they really do. what organizations actually do is what is embedded in the sort of knowledge, the passive, unthe almost unwritten knowledge that people have of how to get things done in an organization. if you followed all of the rules all the time, that's the way to get around contracts and paralyzing the organization. so reorganization, resign it.

simply lack of time and lack of time -- practice over time. several of you have mentioned today that the shuttle had a long period in which things seemed to go relatively well. so that procedures for investigating what may be near misses or potential accidents were not exercised any longer tbhais -- because they hadn't been exercised for so long. past lessons can be forgotten simply as a result of missing pieces of the understood routines that kept things going. over the period between the challenger and the columbia, there were a lot of instances of reorganization, con -- downsizing, and a lot of time. unlearning is the intentional -- of lessons previously learned. they appear to be not

progressive, they appear unattractive. this generates, then, again, gaps in knowledge about how to cope with complexity. some causes of this are the good things that management reforms. when we talk about faster, better, cheaper. there's something so many different versions of that. [laughter] those kinds of progressive reforms are good why ideas. they were necessary ideas. their effect was to undue the structure changes that happened after the loss of the challenger. new budgets and new priorities come in that erase, make impossible to do the redun end organizational structures, the repeat -- that return to flight after challenger. a new agency administrations

coming with new ideas, and they cheerfully and with great enthusiasm undo the things that the previous administration brought in. of course new contract relationships to complicate it. after challengers we see a lot of shifts in funding levels, contracting relationships, the priority for the space station program, government management reforms, national performance review is a corollary on the national stage of better, faster, cheaper reform and the agency stage. so all together these well intention changes lead to losing information. losing knowledge, losing lessons that we had had from previous rounds of analyzing what was going wrong. and i guess i leave you with the issue of what did we learn about

forgetting and unlearning? what are we forgetting now? as a result of the sort of post shuttle era, what are we forgetting? what lessons that we were putting in to practice that we practiced in that era, are we not practicing anymore? what are we forgetting as a result. what lessons are we cheerfully unlearning. nasa did after the changer learned to cope better with debilitating budget cuts, slowing down, declaring more firmly what could be done and couldn't be done. the adaptations were lost. the lessons were lost as schedule pressures and budget pressures ramped up again. we are not out of those woods now, obviously. we have other kinds of management reforms. the reforms that are examining nasa output all the time.

so i guess i would just leave you with that. whenever you want to show -- i wish i had the good shuttle launch image that i think it was ken bowersox showed awhile ago. if you want to show an image of culture you show -- if you want to show an image of american strength and innovation and scientific capacity, and the human spirit you show an image of a shuttle launch. you see it all the time. whatever we can learn about the program that we can apply to the next kinds of programs are important. thank you. [applause] >> thank you. and we would like to invite questions or comments from the audience.

[inaudible conversations] >> you really brought out something i had on my mind. i was going ask about anyway. i'm going ask it. we have talked about the past, and i'm very concerned about where we're going go in the future, and we were at the point in cons -- that we were learning a lot from the old quote, unquote, apollo people, and how they did things and lessons learned and that we were getting pretty good about networking with those folks that were still around or that we could connect with. and then my consolation closed down. even though a lot of that was knowledge capture. i guess my concern is that we

don't know exactly when the next human mission is going to be. how -- what is your assessment of how well theseless song -- lessons from challenger and columbia and apollo and the shuttle program in general will be able to be transferred to whatever human exploration program comes next? >> in some ways that's a difficult question to answer because that's prospective. it's a guess where things might go. any answer is going to be inherently speculative. i think what was good about dr. mahler's piece to say that the problem of -- again using the jargon the complex interactive systems. these problems are not specific just to shuttle. i can talk about other cases of say nuclear weapons design.

we're not doing any testing anymore or what is the role of active design say in 787 aircraft, or nuclear power plant certification? there's a lot of things that we do as a society that are complex that are prone to accidents. one answer as some academics will say, don't do it. and others go, no, we need to go out there and do these dangerous things. so part of the question is how well do we set up organizations to capture these sort of lessons learned? and there's a number of different ways, i think, that can be captured. one of the things i said was have humility in front of the hardware. i think the other think is the importance of flight test not simply as a means of getting statistics. again, i agree with bryan. i don't think we will get the statistics with launch vehicles that we get with aircraft. it's a way of training organizations. it's a way of driving out, who is the engineer who may be does

the annual budget report. who is the person you want fixing the vehicle late at night and making the hardware work? you can't tell who the people are from the resumes. that is there's a skill mix, there's a background and training they have, and their own intellectual capacity and as a manager overseeing the group. when you select for people, you have to have the operational experiences whether it's flight test, experiment, hardware, getting your hands dirty. whatever. that allow you to mary real world analysises to operation. it's an organization. not just technology or handbook gs that fly. -- capture the intellectual capacity. so my answer to your question is are we going give our organizations public sector or private sector the opportunities to learn and gain that

intellectual competence to fly? and one of my many disagreements i promise not to go political or anything today, but about, you know, saying we have been to the moon already, and to say, well, maybe my grandfather went to the moon. we haven't been to the moon. it's a younger generation. whether moon omars -- or mars. how do we give opportunities to capture the lesson. not only learn them academically but give them knowledge so we fly and safely. that's the long answer. >> can i add a little bit to that. that is the importance of -- is this on? the importance of passing on knowledge from one organizational generation to the next. especially when you have the reorganizations of downsizing. you lose so much information.

there needs to be a way of overlapping people within organizations to make it more possible for them to pass on lessons that will never make it in to a rule book or a standard operating procedure or a policy. and so i think some kind of journey system or something that makes it possible for people tow learn the fixes that people have found in the past. and carry those on. >> another question? okay. i can tell we are getting late in the day. with that, we're on time. doing well. i would like to call up our ultimate panel. admiral al geeman and wayne

hale. [applause] [inaudible conversations] [inaudible conversations] [inaudible conversations] so -- actually draw straws -- so

he gets the final word. i will give you my remarks. i think there's a third dimension to the conference that needs to be mentioned. it's not just what did you learn or unlearned? but what are the wrong things, perhaps that you learned? that you need to know. and i say that from a perspective as a -- i should have found my on here. there it is. as a young first line supervisor at nasa, i lived through the challenger incident. and i can tell you right now, i came away with the wrong conclusions from challenger. pardon me for having my company lo gone on. it comes up because i did it on my company computer. i can't get rid of it. pardon me.

when i was a first line supervisor in 19 86, and in the astronaut office, we took a break to go to the library and watch the challenger launch and thought, okay, we have eight and a half minutes, ten minutes and we'll be back in here and finish up the checklist review. it didn't turn out that way. for the next couple of months, i watched the rougers commission do their work as we all did, and they came out with a report, which we all read, and it was not necessarily what i lender. -- learned. whey learned from the challenger launch accident was that a manager made an improper decision and browbeat the troops in to that wrong decision and didn't inform the superiors.

and for seventeen years, i believed that was caused the challenger accident. after the columbia accident, i heard them talking about the book challenger launch decision from a boston university psychologist. i kept talking about it. i said i better go read that book. so i wept downtown -- down to barnes & noble and found one copy left on the shelf. i got -- i read it and i never understood what it meant. i hold two degrees in engineering. i never understood what it meant to be a engineer until i read the book. i didn't understand how nasa worked after having been in the organization, you know, more than a dozen years at that point. i never understood how nasa

worked. i think about the courses they taught me in engineering school, i had to take most them -- solution to series to nonlineal -- that's a skill i never needed in my career every since. if i can take that three hours back and study of sociologist or have a study of psychologist, or study organizational i did ma'amic. it would have been a better use of my time. she said it can be clearly said that the challenger launch decision was a rule-based decision. it was not a morally calculating manager violating rules that were responsible for the tragedy. it was conformity. and i never knew that. i had come to a wrong con conclusion. one of the things that came out of the wrong conclusion we're

basically on the right track. we know what we are doing. we had to watch out for the one lone wolf guy off getting carried away by scheduled pressure. it was not that. it was the whole culture. it wasn't necessarily schedule pressure or money things that contributed, it was this feeling we had a mature system. we understood how it was going to operate. we understood the boundaries in which it was going to operate. we investigated a lot of failures and inflight anomaly they were relatively trivial. but we thought we basically understood what was going on. and therefore that allowed us to be trapped to not paying attention to a four-letter f word. foam. before we get in to that, i want to spend a few minute. we heard about -- a term i had

never heard before. it comes as a shock to you. what did you study in school? what did you study at work? you know, in my line of work as a flight districter going to seminars was frowned upon. it was thought to be a distractions from the real operational work. so we didn't study on these things. let's review what the organization does. if you read the litedture there's six principles. higher liability organization is preoccupied with what i would add the possibility of failure. it's a learning organization. it's a reluctant to simplifying interpretation. it's intense -- sensitive to operations, and -- now i have a nice poster which has got erg that is happened in human space

flight since 1960 all in one page. all the close calls, all the accident, all there on one page with blanks that lead you to more indepth discussions. out of 300 flights, ish, that's a lot of stuff. and the question is are we learning from it? we have a great lessons learned d.a. bases at nasa that nobody ever reads. but now that we can talk about how things put in context. i think we are becoming a close call -- a close higher liability organizational study from the past mistakes. i want to take a second because we tend to talk about spice flight as if it was easy. i know, some of you will say it's one of those discussions about how space flight is hard. but i think it's important to frame the situation that we're working in, you know, captain

young who i have a great deal of respect for summed up space flight, shuttle but applies to other space flight. we put seven people on top of six million pounds of explosive. send them 100 miles in the sky, what part of that sounds safe to you? and so i want to spend second talking about this we're going make space flight like aviation at least in the near term. i flew out here on the most popular airlining 737. you have probably flown it yourself. the reason i picked it. it's roughly the same size as the space shuttle orbiter. i have a few statistics here. if you load up a 737 full, it's about a third of the fuel, about third of it is a crew, baggage,

which you had to pay money for, and about 0% of is structure of the vehicle. the faa requires, last time i checked, 45 minute fuel reserve for lower and enough fueling to avert to a alternative landing place. the total shuttle wayed four and a half million pounds. about 14% is vehicle structure, and about one and a half percent is payload. it's typical or. it was.06% of the total load or about a third of the second run time in the main engine. that's a standard way to get to orbit. that's the reserve you've got work with. as a matter of fact, as the graph shows most of the flights we cut off when the amount level is standby mode from the tank.

the tank being -- well,let do a little high school physics. remember high school physics? potential and kinetic engineer, potential energy times mass altitude. i had one of my nasa intern do the math for me. if your airline is 30,000 feet above -- 500 miles per hour, compare that to a spacecraft to 100 knot call miles high. it's 20 times with, just to get to the altitude. the potential energy therefore is twenty times more. but the kinetic energy goes as vilosty scared, the kin kinetic

energy is a thousand times squared. or about a little graph here that is looks at kilogram and metric. there you are if you're flying on a commercial aircraft. the world's fastist military aircraft, spaceship one, and the highest and fastest mercury red stone. how about any vehicle in earth or bit? it's a tough problem. it's low margins. reentry i'm not going go through the math here in the interest of time. 99% of the energy that comes out of the vehicle during return to earth is taken out by air friction. here we are "columbia" and half your day is piloted over houston. we can see them doing mac --

mach15. 14,000 degrees fahrenheit. hotter than the visible surface of the sun for about a half hour. that's the problem. just to come back from orbit. if you want to come back from mars or the moon some place it's a different problem. that's the problem ahead of us. as i said, i learned a lot from dr. vaughn's book. one of the thing she is said in chapter 6 there, the interior engineering practice, which after the accident always looks like an accident waiting to happen. it's nothing more than normal technology. normal technology is unruly which is to say not unrulely like a 5-year-old child but comes without a set of rules. the rulebook has to be put together. practices don't follow rules. rules follow evolving practices. the implementation -- technology logical systems new rules and relationships are continually

being re vebted. that's how engineering works. well, when bill parsons and i -- thank you, bill, for picking me. i really don't know why -- you know, he we had a tough job on our hands. and the first thing that we knew we had to do to get the space shuttle back to fly was to keep the foam on the vehicle. there's 4,000 pounds of foam on the surface of the external tank primarily to keep it from blowing up. everybody can tell if you had a vice in the tank. that would be bad. we started talking with people and said what is the smallest or the largest piece we can have come off the tank. we did a bunch of analysis. if we came out with the answer of coming off the worst location at the worst time striking the most vulnerable area of the shuttle. you can bring a shuttle down a

piece of foam weighing a half an ounce about the size of a deck of playing cards. we can't guarantee to keep the foam on the tank. we set up and said the primary hazard control is going to be elimination of critical debris. we're going call it the -- they came up with the next four layers that allowed us to have a to fly. one of which was to be able to detect had a problem through boths a set means and cameras and -- rotation pitch maneuver on orbit. then we had tile and rcc repair. when we started the journey, we had numerous exports come us that tile repair is impossible. and in fact in the 1970s, they

did try very hard to develop it. it failed. we flew for 22 years with no way to repair the heat shield that was damaged in orbit. it was an accepted risk. i remember dr. kraft came to us and said, there's two things you should never do. never attempt to do tile repair. it's impossible. never change the outer mold lineup of the vehicle. we did both of them. and it cost a lot of money to do both. it took a lot of time. we came up with repair and finally if the repair was unsuccessful. we had a wild crew rescue scheme that might have worked if we ever had to call it in. that was the. we staid said how with we doing reducing the risk. 20% of the total risk flying shuttle was due to damaged to

the thermal protection system andlet see how we did. well, in a qualitative way we have e eliminated a lot of asset debris. we added orbit inspection repair, we added rescue capability. what we haven't done anything about the rest of the blue, the other 80% of the risk. so what are we going do about that? we're going to enhance systems engineering and integration. we're going add enhancement to admission insurance. we're going do a culture change. we'll talk about that a little bit. we're going do some fixes to the maintenance that was neglected at the end of the day we are left with significant risk to fly the shuttle. and so you to kind of decide is the the reward versus the risk? maybe 60% of the risk made after

everything we did after two and a half years. don't say a word about the csh culture change business. i went school of flight control. dr. kraft was known to remarking on more than one occasion. it's not arrogance if you are that good. i studied under jay greene, some of you know jay greene. i love him. those were my men stories. i being a hard nosed no nonsense, hard to get along with, stickler of a guy. all a sudden bill par sops comes to me and said we have to change the culture. this is what we need to co. you there in charge of helping to change the mission management team culture. i thought it's -- [inaudible]

we were ultimately successful. it flew out successfully and we completed the international space station a year late but it was done. and one of my favorite pictures is tracy looking out at the international space station. a great resource in lower orbit, our toe hold in the stars and the space shuttle program threw out safely. what can we learn? i think it's kind of like the sign. george is laughing. you read the sign, it says look out, the sign has sharp edges. be careful. in fine print it says the bridge is out ahead. [laughter] i think sometimes we everybody size what is easy over what is really important. not that we want somebody to get the fingers cut on the sign. we need to know that the bridge is out ahead.

we need to be careful. i a made a list of ten things. i hope i'm not using too much time. i list of ten lessons i learned and offered to take away from this. number one, it can happen to you. it can happen to you. just because you're younger, or smarter or history lessons, don't think you won't make mistakes. events can't get away from you. they will. nobody is smart enough to avoid all the problems that sliver of fear, the knowledge that the universe is out there waiting if for the last lapse in your attention is ready. that motivation might just help you avoid catastrophe or perhaps not. let hope he is wrong and that accidents in complex systems are not inevitable. let's work to prove him wrong.

the first principle is being preoccupied with failure. be very preoccupied. second, focus. that's very simple. and the mission controller credo, be aware that suddenly and unexpectly we may find others in a row where the performance is ultimate consequences or even better. be sure you slide -- quotation from an raf captain 1930 aviation is not herntly dangerous but even degree it's terribly forgiving of any carelessness in capacity or neglect. life happens. strax distractions happen the probability is one of those critical decisions will come to you in your career. one will come to you in your career. i will guarantee you it will come on a day when you least expect it.

when you are most distracted. focus. the organization is going to fill your time with busy work and bureaucracy. it will. it will. keep focused on the important issues. number three, speak up. speak up. we had a lot of people saying, well, i was worried about columbia, i was afraid to speak up. because i would get fired. better to ask a foolish question than allow a mistake to be made. what is the worst that could happen to you if you speak up? lose your job? it's not the end of the world. lose the respect of your peers maybe? that cuts close are on the matter. misout on a promotion? letting a mistake go unchallenged may have other consequences. funerals, programs shut down, lifelong regret. weigh the costs, make your choice wisely.

speak up rather than remain silent. if the organization can't stand you speaking up, it's not your problem it's the organization's problem. four, it goes along with the first one. you're not nearly as smart as you think you are. no matter how you lower that bar, i can say that's probably still true. remember what your mother taught you. god gave you one mouth and two ears. you should listen twice as much as you talk. you should listen. deferred expertise rather than to leaders. check your' koa egoat the door. too many people are so busy passing out their point of view. they will fail to hear the warnings coming at them. comprehending and acting are also required or to quote captain young, if you're not a little bit scared, you don't understand what is about to happen.

charles wilson, who was the ceo of general motors in 1950 had a great quotation attributed to him. if we are in agreement on the matter and decision, i propose we will post payson further discussion of the matter until the next meeting and give ours time to develop did disagreement and gain understanding what the decision is about. you you haven't examined the problem closely enough. if there's not a natural problem maker in the group. appoint a devil's advocate. make sure that the devil is articulate just like the name sake. draw people out. make them participate. don't let them walk away in silence. six, question the conventional wisdom. people in groups tend to agree on courses of action which as individuals they know are stupid. it's called ab lean paradox.

flying in the space shuttle was as safe as flying in a commercial airliner. we denied the pressure suits. we wanted to believe that even though the proof was in front of our eyes. it is and will be a risky business. challenge conventional wisdom at every turn and look past for the truth. do good work. had used to go it factories and say do good work. i think the opposite. i think we got the crux. the nub, and he said it perfectly and simply. do good work. there is no room in this field of endeavor for halfhearted effort or second best. do it well or don't do it at

all. don't cut corners, and don't let your so-called leaders bully in to doing less than your best. don't accept excuses from others. eight, engineering is done with numbers right here at university they teach spacecraft design. he has a set of laws on the internet. i think he's right. engineering is done with the numbers. without numbers is only an opinion. not having all the information you need is never a -- space is an uncorp. fiving environment. if you screw up the engineering someone will die and there's no partial credit. most of the analysis was right. remember the motto of the mission. in god we trust all -- don't be per sueded be arm waiving lacking foundation and principle.

number nine, use your imagination. after the apollo fire, frank said that the fire was the result of a failure of imagination. they just couldn't imagine a ground test could be hazardous. keep your eyes fresh and imagination active so you can see the possibility, good and bad and work according belie yingly. nothing worthwhile is accomplished without -- [inaudible] james wrote a great book about the korea began war. made a movie out of it. at the end of the movie after several flight crew off the carrier task force had been lost, the admiral in the scene pondering where do we get such men who will go put their lives on the line to do this job? people who will do everything

and put their whole being on a line for a cause. we're fortunate to be in their presence. make the risk as small as you possibly can. then go forward. it seems that someone far wiser than i said once upon a time that safety is very important. we should always keep safety in front of us. but in exploration, safety is not the most important thing. in exploration, the most important thing is to actually go. as i get older, the world seems more -- from the past. people, places, things that are no longer with us. i guess that's just a natural consequence of getting older. after the age of 50, the process is seemed to accelerate with

frightening speed. without passing the lessons and memories on important lessons will be lost and the mistakes will be repeated. tonight let it happen. be sure to take a solid pride in your accomplishment and don't forget the accomplishments. i'll leave you with that. >> i know that you're all going to be disappointed they don't have any powerpoint presentations. [laughter] as a matter of fact, as a matter of fact, the views i'm going express are my own. every comment i made, every view is based upon the report. i'm like the old southern preacher that says, in the book. everything i'm going say is in here. including the page on powerpoint. how powerpoint is going to be the death of western

civilization. i want to thank gw university for sponsoring this. i frequently get asked whether or not i feel like the report made any difference whether on nasa's were better or worse or different. whether or not we have learned anything. if we get asked the questions. and it's forums like this which will keep the principle alive. we had a question over here about how does the next generation learn from the previous generation? forums like this are useful. as i sat out there today, it occurred to me that what we're doing here is we are reopening a painful wound. very tragic event. in our quest to understand what happened and keep the lessons in front of everybody, we are

willing to reopen this tragic event in order that we can continue the process of learning. i'm happy to see so many familiar faces. it's like a it's like a reunion here. it's a pleasure to be here. what i want to do in a few minutes, i want to take a step back from all of the very interesting presentations that have been made today about the tragedy, the flight, the investigation, and all of the aspect of the investigation. and see if i can convey to you the overaging philosophy in each one of these areas we talk abouted that the board had in its mind when it report -- wrote the report. when we wrote the return to

flight rights, that is a tactical detail level repletions that we came out with that is part of an overarching theme that we had. to address the bigger issues. that's what i want to do in my few minutes here. step back and tell you or relate to you what we were thinking. it is a tragedy the loss of the "columbia" and the seven members. it was a personal tragedy, it was a human tragedy, it was also a national tragedy. these are national programs. they represent the will, wealth of knowledge, the skill of the united states of america. and when something bad happens, it is appropriate that the nation take stock of whether or not we want do this