the classmates won second prize for their video on gender wage inequity in the workplace. then our bus stopped in los angeles for a ceremony for the third prize winner before heading to elk grove and rockland to present winners in those areas with awards. c-span extends a special thanks to cox, time warner cable and c comcast in helping in coordinating the visits in the community. watch one of the top 21 winning entries at 6:50 a.m. eastern before washington journal. the former google executive enlisted by the white house to fix the problems in government information systems. from the computer history museum in california, this is an hour

and a half. [ applause ] >> now for tonight's program. in october 2013, the obama administration faced a very large domestic crisis. healthcare.gov, the portal they were supposed to use for health insurance was in shambles. it was several months late and more than 300% over budget. it looked and worked or didn't work like the internet of 1996. the consultants and experts who built it were warning it would take millions more and more weeks if not months to fix. then something quite amazing happened. largely due to the creativity of the president himself, a entrepreneurial team of engineers and project managers came to the rescue working day and night from a hacked together headquarters in maryland, they

rebuilt healthcare.gov. the total cost was the single digit millions. the new site was a complete success. 18 million americans have signed up for healthcare because of it. from that startup-like environment was emerged two digital efforts within government. what the president calls the 21st century equivalent of the peace corps. one is the u.s. digital service. the other is an organization called 18-f. they are calling young digital professionals to washington to lift the 20th century bureaucracy and its technology into the digital century. the effort is quietly taking the government by storm. the agencies who have experienced it now say they will never go back to the old way of doing things. tonight we have three founders along with the u.s. digital services officer assigned to the department of education. we're going to explore with them this revolution from its roots in the dark days of

healthcare.gov 1.0 to the victories it is winning in washington today. join me in welcoming mikey, hillary and lisa. [ applause ] >> hi. hi. hi, guys. >> hello. >> as you can see, the president referred to guys in t-shirts. they're not all guys. >> some are in skirts. >> in fact, this is the most unique panel that we have had tonight for a number of reasons. we're glad to have you here. i tell you what. rather than my trying to introduce what each of you does and what you are doing now, why don't you introduce yourselves briefly. i would like you to add what it

was you were doing just before you started doing what you are doing now. mikey, let's start with you. >> sure. i am mikey dicker son. right now i'm the administrator of the u.s. digital service that is a government word that means manager. and right before i got there to do this -- if you count this as the healthcare.gov story, right before that i was here working just across the street in cl-4 at google. i was a site reliability engineering manager there. >> hillary? >> sure. i'm hillary heartily. i came to this through a fellowship. we launched 18-f. there's a more dramatic story there. i have actually been working with government for -- gosh, about 20 years.

which is crazy to me. but it was always sort of from the outside. so when i found out about the presidential innovation fellowship, it felt like the right time in my career and where i saw government going and what i wanted to do and how i wanted to have an affect on the services that all of you use in your daily lives. the fellowship felt like the right fit for me. i came to government for a short stint and have been there ever since. >> a short stint? >> it was supposed to be six months. didn't go home after six months. >> lisa? >> i am lisa. good evening. i am the chief digital services officer at the u.s. department of education. i have been in government eight months now. my first day was april 6. before that i have done zero civic stuff. before this i was running digital for bet, black entertainment television network. >> thanks.

haley? >> i am the deputy administrator similar to mikey at usds. before usds i was breaking a few bones trying to get it set up and created. >> that's great. thanks. what is the problem? >> who is going to take that one? >> i'm hoping somebody else will jump on that one? >> the problem is so multifaceted it's actually hard to start describing. the biggest issue is that the private sector over the last two decades has spent an incredible amount of time and energy inproving itself and getting really, really great at building digital services that delight citizens and people across the world. all sorts of innovations that we don't think of as innovations anymore have taken place in the last few decades. user center design, cloud, all sorts of things.

the biggest problem is that all of those gigantic transitions have skipped over government. government is still sitting back in the early '90s back with all of the fallacies where cloud sill legis illegal. and that is the biggest issue. there's another side of the problem inside government which is the government is really good at persisting and it has gotten good at developing processes that maintain the status quo. to the incredible detriment of government. there's such a pervasive concern and kind of failure or kind of concern of failure, essentially, that it has created this downward spiral where the status quo has become the riskiest option. and government has a really hard time breaking itself out of those molds. what we're trying to do is come in and break that cycle and

actually bring in the private sector best practices that are very, very normal to every single person in this room and transplant those into government where they are less normal and very radical to be honest. and actually try and shift -- raise the bar and shift what the status quo is today. >> it would be really easy for us looking from the outside in to say, well, that's just so typical. i mean, this is the federal government. dug in, kind of ossified, status quo. it would be a very simplistic way of looking at this. but what you guys have found in digging in and working in the agencies, it's a lot more subtle and complex than the simplistic view we have is. can you talk a little bit more about both of those things? >> well, i would like to talk a little bit about 2012, the group of people that started the presidential innovation fellowship. because it really was the mvp of this movement in terms of saying

we know that there are amazing, brilliant people inside the agencies who live and breathe their job and live and breathe this idea that all they want do is do their job that is helping government be better. for whatever reason, many of them are stuck against walls or they don't have the in-house talent to get it done. they have tried to procure things. maybe that has gone awry. todd park and other folks that saw this need and said, let's see if we can entice people from the tech sector, people from industry to come in for short tours of duty. six to 12 months. let's start there and see if we can attract this new type of technologist that has sort of -- as todd likes to say, hearts the size of jupiter. get them into the government. get them partnering with people who have amazing ideas, who are just stuck for one reason or another. get them partners and see what they can do. and it worked.

i think that's really what we're all building on today is, yes, you can take a survey and you can say, that is blowing up and that is blowing up and we need to help out here. at the end of the day, there are people on the ground trying their hardest to get that done and are stuck for whatever reason. >> i would add on to that. as someone who is embedded within an agency, that is 100% true. and they are welcoming this change. they are embracing the change. they are asking for more change. i had somebody say to me when i came, we thought would you be wearing a cape. they're really -- they are inviting us in to do more than software but to talk about how to change, how to make culture change, how to make culture shift, how to start thinking about using the methodology for software development, like usingusing human center design and put the customer first and how do you extend it beyond tech? how do you extend it into policy

making? and into what we call user centered government. they have been really -- the agencies are -- any tool in their arsenal that they can add to actually fix stuff and make stuff better day in and day out, they are all about it. >> the name todd park has come up. i want to say for those of you who don't know, todd was one of the leaders who created the presidential innovation fellowship and is a former cto of the government, one of the co-founders and real drivers behind the u.s. digital service. he is sitting right over there. hi, todd. [ applause ] >> you can stand up. >> i shoutout for todd. there's a whole team of people sitting over there just waiting to recruit at the end of the night. if you are interested, make a beeline to those two rows over there and talk to them. you have talked about what the vision is and you have defined the problem a bit. lisa, you talked about how it's

taking shape. mikey i will roll the clock back to healthcare.gov. these things weren't -- clearly, they weren't happening, the vision for today. in fact, maybe the things that are most representative of the problem were really on display. talk a little bit about what you encountered when you guys came in to try to clean this up. >> sure. this is a little bit of reliving trauma of the past. so my answer to that has changed over the last year and a half. so i know some of you have heard me talk about this before. there are many layers to what was going on there. at the surface, if you just walk in as an engineer and walk in and look at other engineers and see what they're doing and how they're doing it and it was just totalinsanity when you first looked. there were 55 different

companies contracted to work on different parts of healthcare.gov. which is a fairly complicated operation. but it ain't that complicated. and there were -- i don't know -- conservatively -- this is another piece of insanity. nobody knows how many people were engineers, developers, whatever you want to call them, technical rolls on the project. it was at least hundreds. in dozens of buildings. not only did they not have any habit or custom of working together, they were in most cases explicitly forbidden from communicating because of the way contracts are managed through the government. this was set up. there was nobody -- the government reserved for itself the job of coordinating how this was going to go. the problem is the government doesn't have that skill set. will leave that where that is. i could elaborate on that. the government just doesn't -- it wasn't really equipped to do

that job. what was going on just made zero sense. and so a lot of the things that we did that helped -- a lot of the stuff we did that was the battlefield medicine that helped a lot in a very short period of time was stuff that seems slil silly to explain such as having those people meet together sometimes. in a place. you mentioned in your introduction makeshift headquarters up in maryland. there's an operation center that's still go. it's going full blast around the clock. we have people there in fact. i'm not sure they are exactly there tonight because it's not a high volume night on thursday. it's still there. that was the thing. installing monitoring is a thing that had not been done. without that, these hundreds of people who are responsible for little tiny pieces that have to collaborate together to make the system work, literally didn't know whether the site was up or

not other than by cnn. that was exactly what monitoring was through the first three or four weeks or so. >> you mentioned your view had changed. it used to be one thing and it's now another. what's changed? >> so all that stuff i said is still true. you will still find behavior like that when you walk in and look at a project not going super well. it's just -- if you want to know why it got to that point, ask those questions, peel back the layers, there's a lot more to it. it goes to what you said a second ago, a simplistic two sentence explanation would be the government is stuck in the past. we use that as shorthand. we say that. it's true enough. it's harder to figure out that government is not actually stuck. it does change. it moves forward for whatever forward happens to mean at the time. it just does that at a pace which is a lot slower.

right. and it was designed to be this way. that's what's so hard about it. a lot of -- how many people -- forget people in a job like ours. politicians run for office and say that -- what their vision is to make government run like a business, more efficient, make decisions faster. we don't want the government to do that for the most part. we don't want social security to radically change behavior between this year and next year. that would be not popular. that would be not good. people are depending on those benefits to live out the rest of their lives on. that's not a business that you want disrupted. nobody wants that. the government was designed with value. it's written into the constitution that everybody will get equal protection under the law. that's another constraint that businesses don't operate with. nobody is building -- take whatever tech company you want. look at any restaurant over there. pincknck any of them. every one of them has put into

thought who is the target market. it's not the entire united states of america. everything from the decorations on walls to the opening hours to the prices have all been chosen targeting a customer. not the whole entire country at once. government is trying to serve the entire country at once. >> one of you -- it may have been you. one of you used the term planet scale websites. that's what are now required. you referred, haley, in your opening to we use websites that are accustom to doing these things. and yet the government has not been ready do that and is only just now getting outfitted to do it. why has it taken so long? does anybody want to talk about -- we're now -- we're 20 years into the worldwide web. and probably way longer -- not as long but 15 good solid years into planet scale e-commerce. is there -- does it have to do with the way it's structured?

there's no one decision maker? has no one been the quarterback until the president? what would be your theory about that? >> there's a lot of factors that go into that. i think one large one is how we actually buy and procure services. and unfortunately, as i kind of alluded to earlier, government is so simply a waterfall shop through and through. everything we do is through very, very long, slow processes, particularly in procurement. that's where you see the starkest difference between how the private sector operates and how government does today and what we're trying to shift and change. we still buy software the same way that we buy battleships which is through five-year long requirements, gathering phases, before we build any sort of -- start coding anything. usually it's another five years after that before we push anything to production and let

real users touch it. that doesn't work anymore. as everybody here in this room knows. so i think that's at the very -- but those processes are propped up through a very intricate, well intended set of rules and regulations, which mikey was alluded to and well intended but are clashing as we look at modernizing digital services in a way that are able to move quickly and adjust to planetary size services that quite frankly even for the private sector weren't a thing 15 years ago. we kind of forget that some of these innovations have happened rather fast. and government isn't designed to move that quickly. so how do we combine these two? which is actually the procurement space is something that 18f has been doing fantastic work in. i don't know if hillary wants to talk about the new innovative way at looking buying services from the private sector.

>> go for it. >> absolutely. one of the things that came out of the innovation fellowship was something called rfp-ez. research for proposaproposal. trying to disrupt how those get done. they can be 300 pages, maybe 1,000 pages long. as haley alluded to. it , we will gather requirements and you will bid on this and you will come back in two years and deliver the thing. so from build into our dna was trying to think of new ways to get procurement, the way government buys things and specifically in our case software done. so one of the experiments -- we're running two big experiments right now. one is the notion of marketpl e marketplaces. just to back up, very quickly, 18f resides in this ecosystem, the digital coalition as you referred to in the picture that john showed. so 18f is a consul ttant in the

government. we're consultant that work with agencies to make essentially a build or a buy decision. is 18f going to build a product or are we going to help you figure out what it is you need, how you are going to get it, who is going to build it for you? kind of with that context. one of our lines of business is sort of around acquisition services. and our two big experiments are one, a concept of marketplaces. one thing that we built is the agile bps, blanket purchase agreement, which is a government vehicle for vendors and companies to get into a pool that has been sort of precleared and can do business with the government very easily. the concept of the bpa in and of itself wasn't -- isn't terribly radical. but how we got vendors into it was -- i will say it shook things up a little bit. what we did was instead of having you give us a 300 page

rfp, we said, to compete for this and get in this pool, here is an api, an application programming interface, a set of data, here is a set of data, here is an api, here is the problem statement. build us an app. we want to see how you do it. we want to understand that you understand what we mean by agile, by user center design, doing user research. and all of your code is going to be open source. we want to be able to see it and judge it. so we gave them two weeks to build something. at the end of that, we picked i believe it was about 17 companies that are now in this pool of pre-cleared vendors that operate like all of us do. operate like our teal tems do t have this laser focus on the user. >> does anyone say, two weeks? are you kidding me? >> yes. that was the point.

there's this concept in our world, in software development of an mvp, minimum viable product. it could be a propoe typtotyppr. we wanted to see you could build something, connect data and have output and that you understand all the way we work. some of the results surprised us in terms of there was one story -- >> in which way? >> go ahead and tell the story. >> there's one story. i will not say the name of the company. but they actually sent us an e-mail during the question phase -- we did all the typical phases of this procurement. during the question and answer phase they sent us an e-mail asking for the data on a cd, on a disc, because they didn't understand the requirement. they didn't understand that there is this open data, on the internet, there's an api, work with that. >> what's an api? >> yeah.

so there were some surprising moments. by far -- we now have 17 really awesome teams that we can -- our in boxes, our intake is exploding. we only say yes to about 10% of the projects that come to us for various reasons. the main reason is because we don't have the people to do it. so this is going to allow us to scale in a way and to partner with the business community in a way that is -- it's going to be revolutionary for how software gets built and for -- like hailry sahail haley said, having this business community come with us to really spread that out through all of the government contracts. >> when we were there, i think i heard that phrase more than any other. it stuck with me. which was, modern software techniques. i asked people again and again, what are you doing? that phrase was, we're using modern software techniques. is that essentially what you are talking about? >> essentially.

>> it's a business practice and it's the way you develop the software. one more process question. then i want to dive into some real stories about things that are -- you have been doing and are working. >> things that are opportunities. >> explain -- yes. you are reading my notes? i have to hide them. let me -- someone explain the difference in the relationship between usds and 18f. >> yeah. so to start, i think when we look at government overall, it's really important to -- from the outside, it can feel like this monolithic single institution. it's important to recognize it's much less like a single company and more like an entire industry that needs disruption. in order to do that, we came up with a very interesting kind of three layer technique of how we can actually insert disrupter agents at each level to help cat

lies that change. the three layers quickly are the first is the united states digital service that mikey and i work at. and we deploy teams into agencies. the most talented people we can find to go work on the most mission critical important services at agencies across government. the second layer is also part of the united states digital service. but we feel one of the most important things for institutionalizing this is to disguise the teams, put them in camouflage and embed them into the agency so they become part of the host environment. and they can actually start working on that transformation and culture change from the inside. those are united states digital service teams at agencies. lisa ones that at the department of education. the third layer is 18f, which is -- has an incredible superpower that the first two layers don't have, which is they can operate and function like a business. they do -- they have this incredible model where they are fee for service.

their compliants are plients ar them. they can scale in a way we can't. these two work really, really well together. in terms of kind of top down being able to drop down into th highest need is, whether we're invited or not, candidly. and 18f can work from the bottom up and help scale a lot of the common services and, you know, function as a consultancy for the agencies. >> the business models are different. >> yeah. >> and like haley explains better than i do, you need both of them. they both are going to provide -- they're both a critical piece of the solution. being cost reimbursed in 18f and this is the way the general services administration is the home agency for 18f and it is the agency, its purpose to provide cost reimbursement services for the rest of the government. that means they're not bound by a congressle appropriation in

how big they can scale up to. they're in a better position to build up things that are going to -- to build a product, say, as a shared service and maintain it for a long time. that's the thing that doesn't really make any sense to do out of to white house the way we do. and it has one disadvantage, which is that kind of like vampire rules where you have to be invited and you have to do an agreement before you can go into a place that might need your help. and we have, from the white house, we are limited to the size we can grow to how much money congress wants to appropriate for us which means we'll only ever do a fairly small number of things. we're not set up in a way to build and sustain something for decades. we can certainly take on stuff on the scale of months or a few years. we're well positioned for that and we have kind of kool-aid man rules where you can just go through the wall if it's just -- >> and hillary, why is it called 18f? >> our little homage to "30

rock." the services at d.c. are at the corner of 18 and f. i don't know if anybody here has seen the silicon valley hbo show where they're brainstorming all the names for their different products. we totally did that. i had a white board full of things. we narrowed them down. we came up with about four that we sent off to the lawyers. and 18f was the only one that really didn't pose any problems. but it stuck. and actually i really like it. >> i love that story so much. and i was really hoping it was like area 51, there was some kind of, i don't know, little surreptitious dark about itp. and there you are at the corner of 18 and f street. that's really very cool. let's talk about really digging in and getting some serious things done to develop or revise services. lisa, let's start with you. because you're in an agency that touches everybody in the country, education, in some way. and you guys have done some very innovative things. talk a little bit about what you've been up to.

>> absolutely. so my first project was something called college scorecard. >> college scorecard. >> college scorecard. it released september 12th. and has been really successful. but the idea behind it -- this is actually a presidential initiative in partnership with the department of education. and the premise was college education is the surest path to the middle class, right? unemployment rates for people with high school diplomas only is something around 12%. if you have a college degree, a bachelor's, it is 3%. there are studies that show that over the course of your lifetime, if you have about a degree from a four-year institution it's worth a million more dollars to you over the course of your lifetime. getting college degree is superimportant, yet the people who are kind of most in sneed of it, low income folks, first generation college goers, people who are learning english, like they don't have access to great

support systems advisers. the question becomes how do you get this information about what makes for a good school, what's going to to give you the best value, how do you actually get that into their hands? that was the kind of tool that we were charged with building. i showed up. there was a meeting with the president the week before i joined. and they were like lisa will fix that. and so that's what i ended up on the ground with. >> you pulled on that cape and there you went. >> exactly. but it was a fascinating experience. again, having come straight from private sector and loonding on the ground and understanding kind of the magnitude and how important this was and how this could really affect and shape the future of our country, because you talk about if every person in the country had a bachelor's degree, just imagine what it actually will do, what kind of impact it will have on our economy and our jobs. so it was -- the magnitude of the problems really awesome. it was an interesting experience. so i got on the ground, and i

had to actually understand one, how government worked, but more importantly, what this looks like and what we needed to build because nobody had a clear definition of what it was going to look like. so we went around, we talked to all the stakeholders, people who were creating policy and the people looking at data and the white house a and all that stuff. but with four days into the project, i was like, how do students search for colleges. i started getting like highs, i must talk to some students, i have to talk to some students. and somebody on my team -- and i'd been in d.c. for four days. i don't know anybody here with college-aged stoopts. somebody on my team, so you go to the mall. these genius. the mall where the kids hang out, where there's the store. except it turns out she was like or the mall outside the building, the washington mall? it happens when you don't know d.c. and it worked out -- >> you have to define your terms. >> it was also a school field trips season, so that helped. >> it was spring break.

it was genius. we started accosting people as they came out of the air & space museum. we got people from wisconsin, nebraska, minnesota. it was actually a brilliant idea. and from that we also went to talk to a high school in anacostia, a public school in d.c., we talked to people who had written letters in to the president. we talked to charter school folks in south bronx. we talked to high school in iowa. so we gather all this information and kind of figured out what it was we needed to build and how we could get this information into the hands of students. if you're a low income student what you will pay to go to college is not the sticker price. that, in fact, you might be better off going to a private school from a financial perspective than to a public school. to understand kind of how well a school is actually servicing the needs of its students from you can actually tell if a school's got a 15% graduation rate versus a 0% graduation rate. looking at how much debt you actually might leave school with and then also looking at the earnings that you're going to

have after you've attended the schools so you can actually know that you'll be able to pay off any loans that you may have to take on. these are the new data metrics that we try to get out there into the hands and really change the conversation, not just -- we wanted consumers to be able to get that information then vote with their feet, right? we wanted to change the discourse, change the conversations systemically. so the first thing that we built was something -- was college scorecard, which was a summer facing tool. it was mobile first. and it is -- i mean, it does exactly what we set out to do. an mvp and minimally viable product. and it works really well and i'm really proud of it. more importantly from my perspective is what we did was we opened up the data. our password at one point was set the data free. so we actually -- >> that's probably still our password. >> no, now it's all completely reopened. >> no, i change that. >> all completely in the open. and so we built an api, an

application programming interface that would enable other tools, other, other organizations that might be creating stuff for niche audiences. not everybody is going to come to an ed.gov website. now that we have that become the standard. right now schools are prized for being exclusive. so not letting more people in. and is that the right metric that we actually want to value in our schools? so actually getting the data that we think is important to look at in a school out to -- to the students wherever they might be. you say you want to get your content out to your audience wherever they are on whatever device at whatever time they're looking for it. that's the idea behind creating an api and getting the data out. we actually built a consumer facing tool on top of the api which in technology is called dog fooding, the reference implementation on it. and i think we were one of the

first organizations in government to actually use our own service to power our front-end website. >> how is that going? >> it launched supersuccessfully. we got a million users, i think, within the first week. and it is -- so when we actually stood it up, it wasn't actually the consumer tool, but when we actually stood it up, we had also there were seven people that stood up with us at launch and incorporated the data into their own tools. >> a million and change unique users in the first week and that was against the year before. there was a previous version of college scorecard, this is not the first time this has been tried. in the previous year, the old version did 168,000 unique users, which is a number i'll never forget because i got asked that question in that meeting with the president by the president in that meeting that lisa was talking about a second ago and i didn't know. [ laughter ] so i went and looked it up. if i -- >> so how did the president

react when you say i'm not sure? >> he moves on. i think it was better than lying. if i could add a little behind the scenes commentary to lisa's story there, she talked about how we did talk to users, did user centric design and even policy decisions what data we were going to release, which was incredibly touchy, by the way. this was the end of a, what was it, three years earlier that the administration had announced its intention to do something like college scorecard? and the very idea was incredibly controversial. the higher education community has feelings about this. their revenue depends on federal student -- it was a tough -- it was a messy policy issue by the time we got involved. and in that meeting that lisa's talking about is april 1st, was that when i remember, we heard from the president, we heard from the vice president, we heard from secretary duncan who

was the ed secretary at the time. all of them had impassioned views on it and the president made clear what he wanted to get done. and i was there, and after that and put in my two cents except for the part where i didn't know the answer. after that was over, we had some really frank conversations with the policy decisionmakers and said, we can help you do this, we can make a website thing, whatever it is that you want, we can do that on the timetable you're talking about. but you're going to have to be -- we're going to have to have a lot of say over the product decisions because it's not going to be possible to do your typical government christmas tree waterfall plan with everybody hanging their pet project as an ornament on it somewhere. that can't happen in six months. and given that we were up against the clock, people agreed to a lot of terms and conditions

that they wouldn't oftentimes agree to. after that, we were out on a limb and had to actually deliver it. and that was -- we made that promise, as lisa says, about a week before lisa started. and she came on her first day, here's what you got. you got about this much time to get it done. so we had a bunch of time -- we sweated it to the end and had some -- it was a stressful launch thing, but since it was -- once it got out -- once that thing was actually released, it got very positive reviews. >> that's great. >> and the agency is not accustomed to that happening. [ laughter ] >> that's true. >> it's true. >> that's fun. >> so i just want to -- so part of it is, it was really successful and it got a lot of great reviews and the data got out there. all kinds of tools. spanish language tools now, tools specifically directed towards low-income students that do side by side comparisons that put values on school, it's really tremendous and there's still much more work to be done, right, part of what we're doing in imbuing this culture of

continuous iteration. so that continues toive willen op about one of the incredible things and why the partnership works so well and why government can be effective is i started april, we launched september 12th but development started on the project in the middle of june. so the bailment of this project, apis, front end, all that stuff went from the middle of june to the middle of september. in three months and not for nothing, even private sector that would be pretty fantastic. and this is what i want to talk about a little bit which this is the partnership of u.s. dij attal service in coordination. so we partnered on this project together to bring it all to fruition and we would be able to do it if it weren't for the services and the platforms that 18f hadn't already started building out. and we were able to plug into. i don't know where you were going but that's how we got it done. >> that's exactly what i was going to say. this is a great project to talk about just for a couple of reasons. one is that partnership. the department of education needed to get this done, 18f was

able to essentially do an agreement with them to put a development team together. and to leverage the services, the infrastructure that we had already built. so 18f playing that support structure for the rest of the other two layers comes into play. the other one is to just talk a little bit about how we work and to really talk about what's different and why it's different now. >> good. >> and 18f has done this a couple times but it happened to a certain extent with this project which is our stakeholders have never been as important as the president, but you know, essentially, as mikey said, this is something that's been on the -- kind of on the docket for about three years and the original vision was that the president wanted a ranking. wanted to, whatever, a hundred. and that kind of iterated and changed and it was due to this team saying, that's great, mr. president, and we should talk to high school kids and their parents and guidance counselors and figure out what they need. do they want a ranking or do they want to be able to search

and compare and contrast and what data are they really searching for? >> and was he cool with that? was he open to that? >> i was not in those meetings. >> this goes back to what miker brokered before i joined is we were going to have unprecedented kind of say. yeah, the thing that was important that we focused on was doing the right thing for the audience. we took into account what was important for policy, what was important for data, what was important for -- we took all that into account then put the user first. >> there are two other not universally needed necessarily projects you've been working on, but big gnarly areas with big policy implications behind them. one is the veterans administration and the second is immigration and the whole green card system. and you guys have taken both of those on somebody described those as really big hair balls while we were in washington meeting with you guys. can you talk a little bit about those? because those are even more in

some ways significant. >> i wish it was still only those two, by the way. that was when we talked last july or so. you can add a few other agencies to the list now. yeah, so the v.a. and hs -- you want me to do it? >> go for it. >> so v.a. problem statement in a nutshell, we are creating disabled veterans at a pace that we can't absorb in the rest of the system. we've been doing that for 10, 12 years or so, owing to a certain set of policy decisions. you can probably guess what they are. it's been overwhelming the rest of the system since then. a backlog of disability claims at one point was 600,000 or so. that high water mark was hit around march 2014, if i remember correctly, about just a few months before we came on the

scene. and the delayed disability claims in the case of the v.a. very often mean a lapse in treatment which could be physical therapy for a recent -- for a new amputee because we create a large number of people with that problem. hearing loss is incredibly common and mental health issues are incredibly common. and untreated post-traumatic stress and untreated depression is not only a life-threatening situation for the veteran, it is for a lot of people around the veteran as well. so this is -- if you follow this part of the news, this was a lot of it for those years 2014, 2015 or so. our job there was to try to make the processing of those disability claims more efficient, and also the appeals to those disability claims more efficient. we also need to get better at exchanging -- doing a handoff between the department of defense when you separate from

active duty and you're handed over to the v.a. as a customer, there's a seam where your medical records are to leave the d of e and head to the v.a. system. that sounds like an operable medical record. that was largely done and is still largely done. that mandate, the mandate for interoperable electronic medical records is several years old and is met so far by the dod taking their approximately one file box average paper records per veteran, hiring a contractor to scan them and sending that as a pdf to the v.a. as an image pdf. it's still mostly that way, by the way. we've made it somewhat better, but there's still a long way to go on interoperable medical records. the nutshell version of what there is to do at the v.a. and we're in the middle of it. >> someone else from google working on that, right?

matthew weaver? >> yes. >> is matthew here tonight? >> i think he had to go to san francisco. >> they get to pick their own job titles. kind of like silicon valley tradition. >> i don't know if he picked it or not, but anyway matthews' handle within the v.a. is robe leader. >> yeah. >> if you're a "star wars" fan, you'll get the implication. >> he gets to pick what we wrote on our business cards. >> sorry, go ahead. >> he's here. so he's been a big part of that v.a. effort. you brought up immigration. the short version of that is the process by which you enter the united states on either what's called an immigrant visa or nonimmigrant visa, either way of a particularly for those trying to immigrate and become american citizens, process is incredibly difficult. it involves interacting with a lot of different bureaus of the u.s. government, all of whom act

as if they've never heard of each other before when you interact with them. you get something from one place and you physically carry it yourself to the next government office where you give it to them. sometimes you have to carry a package of paper in a sealed bag that you're not allowed to open. and you take it with you to like the border crossing, stuff like that. you have to pay fees at a half dozen different places throughout this process. each of those fees will be a different amount that's hard to figure out in advance. you'll interact with a different payment system to do it. so you might be able to use a credit card or you might not. and you'll have to create a new log-in password each time. all these things are true about the immigration process. now, we came along and attached ourself to the side of an existing initiative at the agency called u.s. citizenship and immigration service which is a part of the department of homeland security. it has been there for years and been working on this problem and we kind of picked up and accelerated some pieces of it. we did successfully put online in a new forum, in a new actual

human design forum what's called the i-90 process which is the way you get a replacement green card if you're a permanent resident from 'n the united states and you lose yours. this is your piece of paper, this is your piece of documentation that gives you the right to work in the united states as well as not be deported from the united states. so losing it and being without it is a stressful condition to be in. it used to be 6 to 9 months if you were lucky after you mailed in the paper form when you got your replacement back. it's on the order of -- i don't want to promise precisely because i forget. but it's on the order of a few weeks now after the launch of the new thing. we started with that because it is a relatively simple process as these things go. its a ps only a couple of forums and a couple of fees but a high volume, one that affects a lot of people. there's high hundreds of thousands, 700,000, 800,000 transactions of that a year. quickly that's the v.a. and the -- >> those are two. >> haley?

>> and the immigration one is a really fascinating example for a couple of reasons. one is that -- let me say there's no idea as a new idea in government. because if you thought of it, someone's already thought of it way before then. lo and behold the idea of digitizing the immigration system was not a new idea. it had been tried a decade before. and had been under way for a very, very long time. we won't go through the number of billions, yes, i said billions with a "b" that had been spent on trying to digitize the immigration system before then. >> about 2. >> yeah. as a taxpayer, it hurts my soul. but the point is this process had been under way for over a decade. and it was less than a few weeks, about tlhree months from when we were able to drop in a very, very tiny team, less than five people and from when they showed up to when we were able to get the first form out the

door. what's fascinating about that piece is what that means is the folks who are on the ground already working on this inside the agency were also the people that fixed it which shows the big strategy we're working with here. you can actually change the context and the environment and have dramatically different results without a huge amount of lift. you have to poke in the strategically place all the right pressures on the system and it can actually start shifting faster than you think. which is why i love the immigration example so much. >> you know, for decades we've been hearing presidential candidates and other politicians use this phrase, waste, fraud and abuse. and for the life of me, i wondered for the longest time what that would actually look like if it ever got uncovered. i don't want to be one to suggest that fraud or abuse is going on here at all. but the stories you've been talking about for the last -- i'll let that go unremarked. but the stories you've been talking about for the last hour or so are certainly stories in this day and age in 2016 of

waste, not for any intentional purposes or anything else. it's just simply not -- it's not functioning. >> not efficient. sure. haley got at it a second ago in the beginning there. we have -- and back to your very first question, i had to think about it. what is the problem. well, many, but a big part of it is the normalization of failure, by which i mean we have arrived at a state where the status quo way of doing something, which is put out from an rfp, wait for three or four months, look at the bids that come back, go through the procurement process, do your source selection, do all that stuff, hire a huge government -- hire a business who does almost all of its business with the government to do the waterfall plan that

you're used to, spend seven years and a couple billion dollars. the fact about the government is it's easier to spend a couple billion dollars than hire a few people. that's absolutely true. all of us have dealt with it. all of these things, this is almost guaranteed not to succeed. there is a standish group study that we cite a lot that says that 94% of government i.t. efforts come significantly overbudget or behind schedule or deficient in functionality or all three of those things. that's the outcome 94% of the time. >> 40% of them never see the light of day. >> never seen by a user ever. totally unsurprising outcome. so this happens all the time, but when i say the normalization of that, what i mean is nothing bad will happen to any of the people that were involved in the contracting decisions that made

that, which makes it the safest thing for them to do. so like when we say that it is -- like haley said the riskiest way to do a project, it is in the sense that the project is not going to succeed, but the least risk free the perspective of the people in the government who are responding to a different set of incentives because nothing -- like if a project doesn't work, nothing will happen to you. what will get you in trouble is if you try something new and dangerous, if that goes anything less than smashingly well, then there will be a lot of attention on you. then all of the oversight and all of the accountability mechanisms of government, which is congress to conduct investigations, gao, which is the account accountability office that congress' investigative arm will conduct investigations, your internal inspector general, it will just be all bad. so if we just -- if we only did one thing we mostly talked about how -- this is again a thing that's shifted over the last year. it seemed a year ago that the

really special sauce we were bringing into the government was these new ways of approaching technology problems and new practices and new this and new that and we're all superfancy and shiny, and that's true, but probably and even more important ingredient in the special recipe is we are a way to outsource that risk from the perspective of the agency. we are term limited appointments. we are not -- neither of -- nobody on the stage has in mind my own career security as forefront. and it's hopeless, by the way, like it is a misunderstanding and it's a failure to understand, to blame the government employees for putting their career stability first. like everybody puts their career stability first everywhere in the whole -- or their own, you know taking care of themselves and their families is the most

important thing to anybody working any job anywhere. the ability for to us be there as somebody who won't be here five years from now and honestly having somebody else to blame, that's a big -- >> but i want to say, it's more than kind of career stability, i would say. i think you touched on it with like the incentivizations for what success looks like is just different. and so i don't necessarily know that it's about, oh, my god, i need to make sure i have a job in a year or ten years. i think that the definition of what win is and what i've done any job well is is maybe not well aligned with taking some of what you would -- what they perceive as risk. but again we know that risk is actually couldn't edcontinued t status quos this statistic that mike mentioned earlier about if the status quo is 94% of all projects come in late, overbudget, behind schedule, 40%

never see the light of day, that's a norm, right? everyone is just working to the norm. >> yep. >> and part of what you guys are trying to change. i want to talk for a minute -- we've got such great questions from the audience. two other subjects i want to talk about quickly before i do. you're doing this with a relatively small team. there are 113 members of the u.s. digital service. they fan out across almost every agency in government. there are only 185 people within 18f, which in washington terms is you can fit them on the head of a pin. i want to talk about the kind of person that it takes to be in this -- on this team and to do what you do. it surely can't be a good thing to walk in and say, i'm from silicon valley, get out of the way, i'm going to show you how to do things around here. so what kind of person is successful on your teams? >> you raise a fantastic point, which is we are looking for --

actually we're not even looking for. they tend to find us. an incredible combination of skill sets. which is not that anyone on our team can be the most technically competent in the room at any given moment but also have such deep eq and ability to communicate well that they can also win over the hearts and minds of the room. we actually just had our first full year about a day or two ago where we had our online application available on the internet. we launched at city union last year. we have our first full year of data. in that year we've had over 4500 people apply. and to give you a little bit of context, it turns out our acceptance rate is actually way more competitive than harvard's. because we are really looking for the best of the best. and i cannot tell you how incredibly thrilled i am to work with such a talented team because the people at the united

states digital service are phenomenal. we have this really interesting collection of leaders from across the industry, everything from people like mikey who helped set up the engineering class in google, to founding members of amazon, and those who took the structure from its days till now. and the collection of the smartest, most genuinely incredible people i could have ever imagined working with. and they're inside the government. which is the mind blowing part. and i think that's the differentiating factor between the folks on our team and any other team is that they're coming in to do this not because they get to put the white house on their business card or because they make tons of money because everyone knows they're usually taking a pay cut to come into government but they're not too shabby of salaries, if you look across the united states. they're coming because they actually want to make a difference in the lives of americans. and they actually want to work on things that matter.

and so those are the type of people we're looking for. but like i said in the beginning, they tend to find us because if you are looking to have an impact, if you're an impact junkie, there's no better place to be working that right now in the united states federal government. >> haley mentioned a term eq. and we're really looking for the people that can be -- that have empathy with the people that they're building the services for, they can have empathy with the people across the table from them, to win the hearts and minds, as haley said. and 18f and i'm sure u.s. hds is the name. that information driven, impact driven person that absolutely we're screaming for a certain skill set and certain level of skill, but really almost more than anything is that eq side of it. is are you here for the quote,

unquote, right reason? because if you're not, once you hit that first bureaucratic wall, you're not going to want to hop over it, you're not going to know what to do. you have to be willing to bash your head around that wall and figure out your way around it and keep going. >> i would just add on that as again somebody who is coming straight from private sector, just as an example of this, which is in terms of the scale of the impact and what it means so i worked on something called shock wave which was a little while ago but was basically the advent of animation on the web, right, before shock wave the web was static text and graphics that didn't move. and so -- and then i helped launch hulu, worked at bt digital. my products have shipped to literally hundreds of millions of people. i've had an opportunity and been really fortunate to affect the way people consume media in different ways. nothing compares to what i'm doing now. the ability to -- like if i

could even just affect one person's life in trying to help them go to college and make a better life for themselves, this would be worth it. and i'm getting the opportunity to actually affect millions of people's lives and have a significant impact in meaningful ways. the scale is enormous and completely different from anything i've done in private sector. >> i'll do it and i'll be quick. i would say you got to be pretty good at whatever it is your particular field of expertise. you got to be reasonably confident in yourself in that field of expertise because you'll be in a lot of rooms where you'll be the only one who has done an adul sprint before and you need to explain how it is done and you need to be convincing when you do that. a lot of time you'll need to be very patient and very resilient and ready to let -- and ready to be okay with it when you let other people take credit for the

stuff that you have done. if you're all that stuff, then you're ideal for us. also so haley mentioned we're very selective. that is true. i would remind everybody who is listening, though, that there is no correlation between how good people are at what they do and how good they think they are at what they do. [ laughter ] so please don't hear that and think, oh, this is not for me. this is for like some elite engineer somewhere else. you may be exactly that person whether you think you are or not. >> that's fantastic. one other question, then we're going to go to the audience questions. some ideas that come from this president are not always the most popular in washington, especially with people on the other side of the aisle. so the natural question is especially since he's had such a high profile in leading the effort, how is this being received in washington?

is it bipartisan? what sort of support is it starting to get now? haley, why don't we start with you? >> this is one of my favorite parts about the united states digital service is that in the midst of all the political rhetoric we hear about, d.c. being so at each other's throats and the two parties not working together, they've managed to thread a straight through both sides of the aisle and is one of the most faskt examples of a bipartisan agenda that we can see today in washington. it's in incredible thing that's happening where in the process of delivering digital services that work for people and making it easier to get the care and access to people who need it, we are also, conveniently, saving tons of taxpayer dollars and making government more efficient and effective at the same time which is, quite honestly, both parts of what can be stereo tip cancally generalized as different sides of the aisle although i don't agree with that

assumption. but we as a result are getting tons of support all over d.c. whether the hill or the agencies or the white house. and one of the recent developments that we're very excited about was just a couple weeks ago congress just came out with their fy '16 budget which is lot of insider baseball but the point is we received almost our entire funding request from a predominantly republican hill which we see a very strong bipartisan endorsement of the work we're doing. if you actually line up how we select our projects, we have a criteria that's mostly focused on how many users are going to benefit from the service and is it truly providing a life changing impactful service but also what is the failure? what does the opportunity costs of not doing things differently. if you line up our list and the president's and the gao, which is the government accountability ufs which are independent and work closely with the hill, they're almost exactly the same. it turns out the biggest problems that need to be solved,

everyone agrees with. because no one thinks that veterans should be waiting longer to get access to their benefits. no one thinks that government should be wasting taxpayer dollars on things that could be much less expensive and work more simply. a really coming together of both sides of d.c. >> the budget thing is supersurprising, by the way. >> very surprising. >> it is inside baseball so we won't tell you the half hour version of it, i promise. but people who do follow these things very closely were dumbfounded because not only were we funded like -- not only are we a project that the president's very personally invested in that was asking congress for money, asking this congress for money, but like our appropriation lands right inside the white house. and if there's one thing that this congress doesn't want to add more money to, well, there's a few of them, but the white house is high on the list anyway. so this was a very surprising outcome, and haley and some

other people helped -- obviously, i was new and didn't know what i was doing and still don't. and we went to -- we go and talk to the appropriations committees and we make our -- we bring our power point slides and make our sales pitch and say this is what we're here to do and please give us money. and they're very kind of stone faced and don't give you much of a reaction, and then we find out when the spending bill comes out just like everybody else does. we don't get any special insider access to that legislative process. a very surprising outcome. >> this is a good segue into the first audience question which is given that this is the president's last year, what are the goals for 2016? can you talk about those? hillary, you want to go for it? or -- >> well, from 18f which is separate -- the administration is separate from the white house. but for us two big goals this year are to expand some of our consulting services, so that we can help with the agency embedded teams that the usds is

getting into the agencies and then we can help like sort of go in and be a start-up team for certain agencies. so we're expanding on that business line. we're also running more experiments around the acquisition space, so i mentioned the marketplaces and the blanket purchase agreement that we set up. one of the big things we're experimenting right now are micro purchases. essentially very easy to purchase something on a credit card in the government. but you have a spending limit of up to $3500. so what we're doing is we're actually built a platform -- it's an auction platform sort of where we're holding our own bug bounties, if you're a developer and ever participated in something like that we're sort of doing that. we've got all of our code is open source. we developed completely in the open, it's all on get hub. then we pick tasks. we pick issues that have been, you know, written for certain projects, and we say, okay, we're going to put this one up for auction.

and it's interesting because we've done it twice now. we've done two different task batches. the first time that the task got bid down to $1. and it was really -- the person was trying to make the statement. i'm an open source developer. i would have done this for free, but here pay me a dollar. the second time we sort of tried to solve for that a little bit. because obviously we want to be spurring business. we want to be participating in that community. and good for the community. so we made a few tweaks. we didn't say you couldn't bid a dollar. but it turns out that didn't happen this time. but another interesting plot twist was that i think all of the final bids were between 250 and $350 at the end of the day. for -- some was -- one task ended up being about 18 line of code which is guy shipped within 24 hours from the time the auction closed. we're really able to kind of change the game of how services can be delivered and how we can, you know, we can scale our

efforts. but the plot twist was actually that somebody -- again, it wasn't a dollar, but somebody went in and actually fixed the thing that we had put out to auction before the auction was over. >> what about usda? what were you trying to get done this year? >> we -- so it wasn't quite the way you phrased the question, but it's obvious enough. there turns out there's a presidential election this year. it is almost a guarantee -- i know some of you heard me say this at lunch. i know it is almost a guarantee na that the next president and the next person in my job will not spend as much time together. unless and until the next administration has their own healthcar healthcare.gov and this becomes front and center on the agenda again. it could happen, but let's assume it doesn't. so we've got to figure out how to -- if we assume for the moment that something we're

doing is worth preserving, we have to figure out how to create enough of our own reputation and relationships with the rest of the -- with agencies and all the layers of people that are still going to be there in 2017 that we continue on into the next term. and this is the institutionization conversation which started months ago with us and will go all the way through the end. that word can mean a lot of different things to people in washington. the place where we are -- what we are betting on is just flat-out delivering enough stuff that's of enough value that we are worth continuing. like, i don't actually mind. this is frequently criticized as a big weakness of our model. we depend on an appropriation every year from congress. if they didn't want to fund it next year, they don't have to. and if they don't want to fund it the year after that, they don't have to. likewise we depend on the

executive power from the top of the agencies, it's a top-down support. and that means if we don't continually earn the right to exist, we won't continue to exist and i'm at peace with that, but it's the kind of thing that causes people some kind of stress. ma we're doing in 2016 to keep us worth don'ting, we need to ship more stuff at the v.a. i talked a little bit about the appeals process that we're working on right now. we are on the hook to produce a new system for doing social security disability claims adjudication not unlike the process at the v.a. the immigration stuff that you talked about, that's ongoing. we have to improve the refugee administration process and a new one i mentioned -- i wish it was still only two. a new one, one of the newest things we're on to is if you follow really closely the white house talk about the executive actions on improving criminal

background checks before you can buy a gun, we have done work with the fbi to modify that system as well. those are things that we need to show real results on this year. >> one additional thing on top of what mikey said, yes, there are a lot of unknowns in 2017 and in the near future, but one thing that is exceedingly clear and that everybody in this room knows is technology is not going to become any less important to government than it is today. it's only going to become more and more important. and i think one of the incredible transformations that's happening right now is it's exceedingly clear that, as president or ceo of this country, yes, you need to have a background in policy, ideally it's great if you have a background in law and economics but there's another leg of the stool now and that's tech. just like you can't run a company in the private sector without understanding tech yourself or at least empowering a cto, you can do the same thing with a country anymore, whether it's the united states or anywhere else in the world and that trend is not going to

change. that's for sure. >> i have a question for you. >> okay. >> not about 2016. but it's an important question. someone wrote tonight this is the first time they've heard of college scorecard. so what are you doing to get the word out there to everybody who wants to hear? clearly if a million people have created user accounts, that's something but there are a lot more people who could potentially benefit. >> absolutely. i will also say the target audience is, again, kind of underserved communities. we talked a lot about high school students but actually 50% of the students in college today in this country are over 24 years old. and so it is really, really important. we built this minimally viable product and i'm glad but this actually falls into what we're doing in 2016. just actually one little piece of wa we're doing in 2016 which is actually again kind of building and iterating and building this culture of continuous improvement and part of that includes outreach and part of that is actually figuring out how to get the message out there, outreach through guidance counselors who

are actually reaching those students. how do we continue partnering with third party organizations who are in those communities and how do we actually get them entice them to use the data? so there's a lot more. we did a what i would call a one out but we'll continue to iterate both from a future perspective but more importantly from an outreach perspective. that will continue on. >> part of the beauty of the college scorecard you don't actually need to have heard of it or gone to a dot-gov website to benefit from the data that lisa and other team did. that's the whole point. this data is getting pushed into users are which is on different education website. honestly not every person thinks of going to a dot-gov to get help for things like this. we want to make that okay and push this information and data where people currently are instead of forcing them to come to us. >> two more questions from the audience, then i've got one more final question. this is -- what about -- i'll just read this.

what about push back from the hundreds of thousands of contractors engaged in developing software the old way? you are so few in number and they are making a lot of money. >> those are facts. >> as i mentioned, we are very, very much trying to work with the vendor community and enable and empower businesses and small businesses to do work with the government. that's part of what the general services administration does. so we're taking on that mantel on the technology side. all those things are really in service of that. all the acquisition stuff, the marketplaces, trying to get new businesses in and doing business with the government for the first time are the bug bounties, you have to be registered in this thing called sam.gov. you have to be a registered business owner, but these people

are doing business with the government for first time and it's giving them a lens and window into this new experience. so yes there are absolutely companies all over the country and specifically in d.c. trying to do business with the government and making a lot of money at it, but i think what we're doing and you heard us all say it in various ways, we're shipping code and software, but at the end of the day, we're shipping culture. we are trying to fundamentally change the way that these things can get done. you know, so that it can be faster, it can abouty better, it can be cheaper. you can have all three, you don't have to pick two, which is the old adage. and when you show that that can be done with a lean, mean group of people inside the government, and you also show good faith that you want -- we understand there's fewer than 300 people doing this, it's an 80 billion -- at least an $80 billion industry to do business -- to do i. dchli.t. b

with the government. you can't even see the dot that we are in that number. so we know we can't do it ourselves. we have to be able to rely on the business community, you have to be able to rely on the other vendors out there. so what we can do is to give the people we're working with a new experience. so that they go through a design studio for first time. they go through an agile iter attive cycle development for first time and they come to expect that from the people they work with and hopefully look for those qualities in the next people they work with so the whole system starts to gradually change. >> that's a great answer. go ahead, mikey. >> we have talked this one through a few times. and all of us have slightly different take on what strategy actually is, and that's fine because everybody has their own ideas, but first point i would make is it's not -- it is not the -- we don't have to change every project in the whole entire government. we don't have to hit all of that

$84 billion a year in i.t. spending for this to be a thing worth doing. the five or ten important projects of which i described a couple for different questions tonight, if we only ever affected those, i would feel like what we did is worth doing. that's point one. point two is, yes, there is this ginormous industry which is going to continue, which is going to be mostly contractor driven because to bring all that work inside the federal government would be a staggering expansion, hundreds of thousands or more federal employees. it would take an agency bigger than the v.a. which is the biggest agency in term of employees to take this all inside the federal government and i don't -- we don't intend for that to happen. we're not making any effort to make that happen. i feel like i say this every event like this and it never gets printed in the press, it never gets picked up, never gets amplified, but we're not here to kill that industry. i don't care.

that industry can continue to exist. that's great. that's fine. we're going to have to -- i don't care if the same exact companies exist. we just need to fix the ecosystem a little bit so that competition is a little bit more healthy so that we spend -- we can still spend -- the government -- this is one thing i'll predict with 100% confidence. the government will continue to spend a -- load of money. >> that's probably why it never gets quoted. >> damn it. i knew there was a reason. so that's going to continue to happen and people will still make a lot of money and making a lot of money is not on the face of it a bad thing. but we do need to -- it would be an improvement if either spend the money but the things actually worked when we're done. like if we didn't have that 40%

that never saw the light of day, if everything we bought actually worked that would be an improvement if we didn't spend a dime. and if we didn't get anything to work better but we saved a lot of money, that would also be an improvement. >> we don't want to villainize contractors. we couldn't do this without them. but here's the hypothesis. they don't want to build -- digital services either. they don't want to work this way either. and that's kind of the hypothesis we're working off of. if they can change the environment, they would actually love though work with us the way they're used to working with their private sector clients. our government is propping up a failing industry. i don't know who else is waterfall except for us now and i'm sure the large contracting companies of the world don't appreciate having a waterfall style operating system. >> they do what they are hired to do. they submitted bids to the -- like hillary's talking about some of the new kind of experimental stuff we're doing with making the competition more competitive, and they will

adapt. but it's up to us to set the direction because we write the paychecks and we write the rfps and the contracts. last audience question is about whether there is a role for nondevelopers in all of this. are you looking for people who run businesses or who do biz dev or have entrepreneurial business skills and not necessarily tech skills? i see you shaking your head, haley? >> yes, absolutely. we need an incredibly diverse range of skill sets because this work is incredibly hard. we need the best engineers in the country as well as the best project managers and designers and researchers and policy experts and people who are generally good at getting -- done in hard environments because i guarantee you there's probably no less hospitable environment than the government than to getting stuff done. if your still stuff also happens to be bureaucracy hacking, please, please apply. >> that's a real thing. >> i'm going to ask the final

question, and make it just a little bit personal. you talked about your shipping culture not just simply services. and shipping culture and having a culture affect you is kind of a two-way street. i want to ask you kind of as a final thought working with you haley and working this way, how has this experience changed you and what do you think you'll take away from this when you look back on it? >> i think, to be honest, the reason this has changed me is for the rest of my life and i probably still haven't learned all the ways yet, my biggest fear is that i will never be as satisfied working on a project again because the impact of the federal government is so large and so meaningful in people's lives that i will probably -- don't know it yet, but will probably spend the rest of my life on a quest to find something that impactful once i

leave here. >> for me -- and again i've only been there eight months -- and having worked with a large corporation and several large corporations before, the way government works and the bureaucracy how to actually get stuff done is not completely foreign to me. the thing that i have learned that has kind of struck me the most is i knew when i was joining that my team members live in the digital coalition at u.s. digital service were going to be superpassionate and engaged and excited about what we were doing. what i didn't know as an outsider of the government when i showed up at the agency i was blown away by the folks who i'm partnered with there who show up every day, get out of bed, put one foot in front of the other because all they want to do is make the world a better place and make it better for students and make education easier to get and more equal across this

country, and i kind of expected all this resistance and this -- you hear about right from the outside you hear about how government is and how hard it is to get anything done and nobody wants to and they're a bunch of bureaucrats and you're on the ground and all anybody who is there wants to do is actually fix things. that's a great learning experience for me. >> plus one, plus two, and i think for me you sort of centered the question around culture. and to me that has been what has resonated most. when you -- so i was a designer really building web services in the private sector and then mostly around and for government as i mentioned from the outside and always coming from the design side, doing marketing and sales at the end of my stint until i got to the government. but had kind of been preaching

user centered design for a very long time. focus on the user. think about user needs before stakeholder needs. and the thing that has sort of stuck with me because my job really almost from the beginning has not necessarily been delivering product but has been building this team, has been delivering the team. and it is incredible to me what you can do when you put your focus on people and on culture. and the number of people that when you ask them what's the best thing about 18f, across the board, the first thing out of their mouth is the people. and like i said, when you screen for that, when you put a focus on it, when you put a focus on people, on diversity, on really wanting to hear from everybody, it is really incredible what that team can do. and that's kind of changed how i approach everything from now on. >> mikey? >> i had the most time to

prepare. >> and you just took a big sip of water. >> and yet i feel like i'm to l totally unready to answer this question. i will acknowledge first, because i feel like it's important, on behalf of all the people that work for us, to acknowledge that, yes, none of our lives are ever going to be the same. a good number of people are going to come and have come and done a limited time tour of duty just like we sold them in like the video that was up there. and have gone back to something resembling normalcy for what they were used to. i most likely am not. i can see that. i have no idea what's going to become of any of us. we've talked about this. it's not just because we're on stage in front of 800 people, it was the same when we were at lunch. we have no idea what will become of us in a year or so. the human cost of this is high. the sacrifice is high.

you go into that in some depth, too, but i won't to spare you. i've become very, very aware that because particularly this is a specific job that i do which involves being a backstop to a lot of those people who come to work in the government and turns out to be >> he has a speech he loves to do about there is no possible informed consent here because it doesn't matter. this is what happens when i talk to people one-on-one just before they come and say i can't tell you how hard it's going to be and you're not going to believe me and you're going to imagine the hardest thing you can imagine and it's still going to be harder than that. so a couple of weeks after you get here you'll have hard moments and we'll work through those and that's what i doful since your question was how has this changed you, my frame of reference changed to where a year ago if you said how are you going to fix the government i would talk about design process

and whatever else we're going to use. it's going to be great. technology focus stuff. i don't think about that hardly at all anymore but what i think about is the problems of managing enormous organizations of people where communication becomes really difficult and coordination becomes really difficult and it's not sufficient that they want to get something really good done. they do all believe in their way they want to make service better for veterans so that's one of the things you have to come to terms with really fast. you meet all of these people and you realize they all want the same things you do. we're building this organization ourselves and leave it to be difficult to run a group and

approach consensus. those apply to us too. it's likely that i don't have any idea what it is and they'll probably be thinking about those kinds of problems, mostly. >> can i add one thing? >> sure. >> the other thing that strikes me is this has also been a tremendous opportunity to have a platform. to have a platform to talk about building diverse teens and i they we have taken on ownership to prove that it's possible and be representative of the people we're trying to serve but more importantly we understand and know that we develop better products if we have diversity of opinion and voices and that's something we tried to show case and it's something that you talk about in the tech industry and the elite folks that we're trying to recruit and the more

we can actually show case if this is possible, that it's possible like this and, in fact it is a better team because of its diversity that i think that's an important take away as well. [ applause ] >> i feel like we discovered something rare and wonderful tonight in the work that you're doing in you as individuals. so thank you so much.

book tv has 48 hours of books every weekend. here's some of the programs. starting saturday at 1:30 p.m. eastern book tv is live at the los angeles times festival of books at the university of southern california. then at 10:00 p.m. afterward with former oklahoma congressman j.c. watts. he talks about dig deep. 7 truths to finding the strength within which outlines the principles in his personal and professional life. he is interviewed by espn's editor and chief. >> you have to overcome diversity. you have to have humility. so i learned to try to run my race and maybe i was just a focus, really wasn't, i told that reporter i said if my skin

color was an issue, that was everybody else's issue, that wasn't mine. >> on sunday at 10:00 p.m. eastern, author jillian thomas talks about her book because of sex which examines how title 7 of the 1964 civil rights act which made it illegal to discriminate based on sex effected working women. go to book tv.org for the complete weekend schedule. >> i am a history buff. i do enjoy it. the pab rick of our country and how things, just how they work and how they're made. >> i love american history tv. american artifacts, they're fantastic shows. >> that's probably something that i really enjoy. >> with american history tv it gives you at a perspective. >> i'm a