another finalist for this year's national book award. she is the author of weapons of math destruction. the denial of services loans and more. [inaudible]

>> thank you so much for coming out tonight. it is our honor to welcome you. weapons of mass destruction. were really excited to get into this. i want to let you know for you get started. or any noisemaking devices. and also the format is different. we will have a conversation for about 20 to 30 minutes. i would really appreciate it.

we will be doing a book signing after. at the mention of that. as a partnership. if you look at the calendar you can find where there things are happening to the state.

increasingly. they pay for health insurance. the same rule. it's urgent and necessary. who has been involved in this by wall street. recently started an auditing company.

and against the notion that because of it was implemented and not to perpetuate a buyout or adjustment. [inaudible] the metrics of that. on the hedge funds. she started working at it science company. she's been doing this since 2013.

please join me in welcoming her. kathy o'donnell. imac. >> maybe we can jump right in and get some definitions and ideas out there and talk about one of the examples in your book because we were chatting in the back one of the products we've done they use that. so it turns out that a lot of businesses want you to take those and you don't need to take them anymore because we have them that can look at all of the data left behind.

we are gonna talk about what weapon of mass destruction is. and then we could draw out the characteristics that make it that. there are some ugly resumes that i carry about and when i care about our really a destructive tear characterized by that. to make it real. there is a guy name kyle name. when he wanted to get a part-time job at kroger's grocery store and his friend was leaving. you just head to felt the paperwork online. and so he started to fill out the paperwork and 60% of job applications in this country

he was required to take a personality test. online before he got the interview. if he fails this which he did the new would it get to be there. he was lucky. most people don't find out they just never get called out. it got read by it. he has the lawyer. they are applying to minimum wage jobs. he talks to his father and his father ask him what kind of questions were there on this test have a job. he was going to a competitive college. he got straight a's in high school.

they were a lot like the ones we got in the hospital. it was a mental health. they measure that. how emotionally stable are you. or super mellow and nothing bothers you. it's easy to find that out but it turns out they show up in all kinds of other data we leave behind you. >> what happened was he talked to his father actually what happened was that it is illegal. you can't make them take a health exam.

including a mental health exam. that is the american disability act. they are trying to prevent that. his father is filing a lawsuit. on anyone that ever took this test. he went ahead and try to get a job in six other large companies and he got the same personality test for all of them. he was prevented from getting the job in his area. as a first example i want to talk about. >> in fact a lot of these kind of insights about people come from these models that are built on ugly resumes.

the fact that they can harm people or had bias that is the nature maybe you can define that. it's a widespread and impactful. algorisms no one cares about this. they start caring when they affect a lot of people. this is good to be a good example. it was widely used it was actually located in boston and all sorts of companies to use instead of their hr people. the second is that it is secret. kyle did not understand how he was being scored in most

people who took this personality test did not even understand that they were being scored. and finally it is destructive. they destroyed the chances of getting the job but it was destructive in a larger sense. with the "weapons of math destruction" they create and reinforce feedback which is destructive to society as a whole. systematically refusing to employ people with certain disorders. one of the examples that i like most in the book. it's actually a local one. highlights the really secretive glass box nature including the computer science. and that was the use in the dc school system for getting them fired.

she was the school chancellor for some time and she instituted this policy by some people would get fired if they have bad teacher assessments. the assessments are complicated but the very short version is that most of the ways that they are assessed they don't have a lot of spread. they get acceptable or very good. and the people who really want to discriminate against us are frustrated with that. they want more people who are terrible very few of the ways that we have. they instituted this assessment called the growth score for the value added

score of the teacher. the very broad way of thinking about this is the idea that the teacher is on the hook for the difference between what their students should have gotten versus what they actually got. there is an underlying model that estimates what each student in the class to get. you are all fifth-graders right now. let's say you got a 75 out of 100. you got more than was expected. it's not as clear how this expected score was actually

determined per student. there is another source of uncertainty which is what you actually got at the end of fifth grade after all it varies every year. and kids get different scores. .. are being said based on error terms, which is -- if you're not a stat do statistician, we would

call this stuff meaningless and this is the way the teachers -- the scores came out. almost meaningless. and i say that not because i have hard data on that, because it's actually a secret score. the only way i got my hands on any real data besides story is -- i talked to various teacher. talked to one teacher who got six out of 100 in 2010 ask then 96 out of a hundred in 2011, from new york city. in general, it's a secret system. we can't get our hands on. so another source of opassty. so going back to washington, dc and dish interviewed a william who got fired because her overall teacher assessment score was too low. it wasn't entirely do to the score. it was 50%.

the other 50% were these things that don't have any -- most of the information in her score was they value added score. one more thing to tell you about this. sarah's score, it that a lot -- she was a fifth grade teacher. already fourth graders have gotten really good scores at the end of fourth grade. the kids coming into their class in fifth grade had gotten excellent fourth grade scores but they couldn't read and they couldn't write. and so she was suspicious of their really very good scores and in fact she has every reason to believe the teachers of the kid cheated on their tests at the end of the year in order to get better evaluations for themselves. then their kids got better than expected so they got good scores. so they set her up to get worse than expect it. does that make season?

she has good rope to believe unusual erasures. i don't know if you remember the scandal but unusual erasures which never real got investigated. they got hush-hushed basically. in any case she got fired for this test. so let's go back over the three characteristics of the weapon's of a mass destruction. it's widespread. value-added scores are being used in more than half the states. mostly in urban school districts. it's secret. so, sarah didn't understand her scores and she was suspicious of it again but when he appealed it they said, you know, no, it's fair because it's mathematical. and finally, destructive. not only did it destroy sarah's life because she got fired. i should say she got rehired the following week in sort of affluent suburb where they don't

use the scoring system. and that's the final thing which is what destructive in average larry sense because you'll be surprised if you know about michelle reed and why she was brought in, which is there's a sort of nationwide kind of war on teachers going on, which is find the bad teachers, get rid of. the and then we'll fix education, and a bunch of presidents want to be the president that fixed education. we can talk about whether that makes sense but the idea was to get rid of bad teachers. right? and in a larger destructive feedback loop enengine gird the value-add model, get can rid of good teachers because good teachers have been quitting, retiring early, and most of all they've been moving to affluent suburbs that don't have the scoring system. and right now we have a nationwide shortage of teacher is.

so i would argue that this regime of this opaque value-added model hat been very bad. something that is especially insidious about that. talk about algorithms and that can sound techy and foreign, things we don't interact with. but a lot of the algorithms we're talking about here, that we both work on, have tons of similarities, sometimes are identity underneath, like algorithms on netflix that recommends films and amazon that recommends books. amazon is moët of the time super boring and not insightful. you but a a steven king back and it's how about these other steven king backs. you're right but not helpful. sometimes it's totally wrong and you go, where did this come from? and then sometimes you get these beautiful glimmers of insight. so i bought this survival guide and then i buy this 18-inch knife that folds out. never would have thought of

that. a great insight. but you don't just buy everything amazon recommends, don't watch every are every show that knelt flex tells you to watch but the algorithms people don't treat them us a one point in the decisionmaking process. they trees it as oracle of truth, even when the people using them don't understand that and that makes them more restruck disbecause you overright our great human intuition with this little bit of math. >> great point. it's actually worse than what you just said. you know how -- give you two examples. how the facebook trending stories thing has been fucking up lately with the weird, fake stories. we noticed that and we give feedback to facebook saying, that's not a real story. right? that's something that doesn't happen for the teacher value-add model. there was no ground truce for the teacherred no secondary

assessment of the teachers to compare their scores to. they were just told, this is your score. the guy who got six out of 100 was shamed. he next 'er he got 96 out of 100, and the said what struck me we front am terrible teacher to great teach center there's no feed back mechanism so teachers can't say, i'm actually a really good teacher so please update your model. but that would be the equivalent of us saying knelt flex i don't want to see that movie because it's awful. >> and then you have a button, don't recommend anything lick this ever again. we have a way to tell netflix it's messing up. teacher does not have access to that way. parents or administrators knew they've were good, they couldn't go in and override it. >> exactly. you're in the other thing i want to agree with is that people really did just trust these scores because they were

mathematical. the way i came across a algorithm. my friend is a principal of a high school in new york city and started complaining to me about her teachers getting scores, and i just said, well, can you tell me how they're being escorted? what is the algorithm. she said, well i, i asked my contact at the department of education and they told me, don't -- you wouldn't understand it, it's math. and i was like, that's not good enough. let me just say that mathematics are supposed to clarify, not confuse. that's not a thing you do with math. that's weaponizing math. right? not okay. so i asked her to keep pushing, and the got through three different layers, elayer saying to her you wouldn't understand. it's math. three times. she finally got this white paper which was unreadable to me, and i'd been -- she had been doing for seven years and i cannot understand what these white paper is saying.

so what i did was i filed for a freedom of information act request to get this source code. >> wow. >> for the algorithm. and it was denied. and by the way should i say that the reason -- the "new york post" filed a freedom of information act request and successfully gotten all the teachers names and scores and published them as an act of public shame can for the teachers. off. it was like if they can get the scores shy be able to get the way the scores are made. i was denied. then i contacted someone at the institute that -- the data institute that built the score through a friend of a friend, and they explained to me i would never get that source code because they had a contract with the city of new york which stipulated that no one in new york city would get the code. would ever see how the stuff was made. it was secret from them. which is the say secret from the

officials in the department of education so they can't explain to their teachers how they're being scored. >> so, part of this book -- really the book as whole maintenance fest stow of your argument how math its being weaponnized here. thought maybe you could give us an overview how you came to that place. you were building these for finance for a while and then you kind of became the enemy of finance. my words, not yours. but joining "occupy wall street" is probably not something your hedged funds would have smiled upon. right? >> definitely. >> many years of your life. >> yeah, yeah, so i joined "occupy" in 2000. back up. joint my hedge fund in the early 2007, and right straight into the crisis. walked in and then, boom, crisis. and i was, like, really disillusioned quickly because

the people i thought were experts really didn't seem to know much about what was going on, didn't understand the market, which is not to say i understood it perfectly but they didn't understand itself that minute better than i did. at the heart of it -- by the way the person introduced -- i'll talk about -- that's why i fell in love with math. how beautiful -- i had this idea mathematicsed being this pure, clean, beautiful, almost artistic endeavor and then at the foam crisis, the heart of the financial crisis was this mathematical lie, the opposite over the beauty of a rubik's cube. the triple-a ratings on mortgage-backed secures. they were promises that people who were really good at math, with ph.ds were in the back crunching the numbers and promising that the mortgages weren't going to good bad.

well, they weren't doing that. they were just basically creating a lie and selling it for money in the triple-a ratings? a. no did people believe it but the people believed so it much that the scale of the market for marked backed securities was very large um one reason it was such a big deal. that's one thing i realeye arized it was a weaponization of people's trust in mathematics. they trust it math, they trust it mat me -- math mathematician. people were corrupt and shielding are their corruption by calling it math. don't look here, you wouldn't understand this, it's math, and it's right. you have to trust it. so that's kind of the big thing that i realizeed. this isn't the right way to deal

with math. became a data scientist and was, i'm doing almost the same thing as in finance so predicting markets i'm predicting people. that's what data scientists do. they use historical data to build algorithms to predict people. and that didn't seem so bad, but until i came across sort of the teacher i mentioned, which was that's not good, and then i think the moment that i decided to quit my job and write this book was caused by this one action is had in my store where this venture capitalist came to visit, and he -- i was thinking of -- he was thinking of investing in our company, series b funding. and we all sad and listened to him talk about the future of tear i tailored advertisings ump was looking on advertising in the space of travel, like he can

speed ya. relatively benign. you eave offer some people hotel rooms and some people you don't offer them hotel rooms. so your giving tubs to some people and not giving opportunities to others. like very kind of mild way of segregating society. does that make sense? you guys deserve these offs, you guys don't. didn't think of it as particularly evil but then the guy gave us this idea of what his future dream for the world of the internet was. he was like, i have this dream that some day -- didn't sound like martin luther king. he says, have this idea, this is what i hope to see in advertising. he said i hope to see some day i get offeredded transcribes to aruba and jet skis and i never again have to see another university of phoenix ad because those aren't for people like me. and everyone around me laughed.

and i was like, what? what? i thought that whatever happened to the democracyizing force of the internet. this is the goal of the people constructing the modern internet, which is to segue segregate people and file them by class so we, the technologists, the people who are lucky, who are getting scored well, score high, we have opportunities, we have like toys and things that we like to play with, and then people on the other side of the spectrum can be preyed upon. >> i make money off you from your vacations and i make money off. the by exploiting universities that giveworthless degrees. where can i make money and if you're in a certain class i can make money by exploiting them. >> you become that the way tailored advertising work, there's an auction for the eyes only whomever you're talking about and it's very espirit got

alters the demographics. and again, if you're talking about someone like me, they're very good at finding me and showing me yarn because i'm like very vulnerable to alpaca. they know my number. right? and that's okay because, like, i love mayor and sometimes i'm like, i want that yarn. but like for in people, the most profitable thing is to get that person to take out a federal loan, which goes straight to a for-profit college, saddleeses them with debt and dot not give. the an -- does not give them an actual education. we have seen for-profit colleges closed down riotly which is great news. it was an enormous -- still is a relatively large industry but the type oven tour cappists talked to my company i had never seen a university of phoenix ad. didn't know what it was

honestly. it was 2012 issue think. looked it up and i saw that apollo group, the parent company of the university of phoenix, was the single biggest google ad buyer that quarter itch realized i'm not going to see the failure of this kind of algorithm. everyone here saw the failure of the triple-a ratings on mortgages. everyone in the world saw that. because it was so loud when it exploded. but who is going to see the failure of this kind of algorithm? because we have segregated ourselves online to such an extent that when i was -- with so many false prey, that kind of trap, we do not see it. we blame them for it. how are we doing on time? i can keep going. great. where do i jump in here? okay. i've got, like, ten questions, i'm going to smoosh them into

one. so, one of the things you talk be in your book is potentially we have people who are building these algorithms take something equivalent to a hippocratic oath and certainly there's no ethics training for people like us. i didn't -- you don't even -- makes people uncomfortable in the computer science department to talk at ethics. so, there's training is a good part but i think the question i came away with is, if there's money to be made by exploiting people, it seems like that's going to happen. we saw if with the mortgage-backed securities but we saw it with enron. back in 2001, phone calls from enron trader, if you remember, shut down the power grid in california, just to make money on energy futures, and they're talking about the old people who are getting screwed, basically, because they have to pay these enormous rates for their electricity. seems like the market is set up where if we can use these

algorithms, if we can use anything that is vaguely wind the rules to exploit people to make money, we'll do it, and i wonder what your thoughts are on how we change that part of the system. >> right. so i'm not expecting all corruption to end. right? period. but i do want one particular form of corruption to be challenged, and that is the corruption where people claim that because of an algorithm, it is beyond morals. every algorithm has imbedded values in it. it's just that we -- the people who create the algorithm, deploy them and own them, are the ones that get to decide what the morals are. so we need to start challenging that. and when i -- there's a very -- i have a very specific reason to define the algorithms like i do. the weapons of mass destruction. because i perform triage.

i perform triage on algorithms. tell them where to focus our attention, and for those algorithms, many of them are accidents, accidently terrible algorithmses. some are just terrible because people who are greedy will always want to do that. so, i do think that for the algorithms that rise to the level of maybe good down to the level of something that is potentially weapon of mass destruction, we need to have more scrutiny. there needs to be laws about that. a lot of this stuff there will be no free market solutions to this. the free market will not solve this problem. it actually makes money. it is profitable to be discriminatory. so you can't ask someone, please stop being discriminatory because it's not great for society, even though you make more money doing it. that's just not going to happen.

that's why we have laws around things like fair lending. we have the fair credit reporting act. we have equal credit opportunity act. these are -- and the discrimination laws that were written in 1970s that essentially regulated things luke fica scores and they're not perfect but they're a hell of a lot better than we have for what is going nonbig data. so i'm basically arguing that, none one, the public as a whole stop trusting algorithms. push back. demand accountability. because we have secrecy you have no accountability. so especially when you're being assessedded a your job you should be able to know how that i happening. how am i being assessed exactly? you should be able to ask that question it but there also has to be rules and regulations around things that rise to the level of widespread and impactful. >> i think just to kind of close this out. a point you make clearly in book, one i have been thinking

about, if you have an al grow rhythm that seems unfair or should work differently, then that's because of a human, basically are right? if you have an algorithm discriminating and doesn't consider race but does consider zip codes that's because whoever programmed it didn't think that usesy codes -- didn't think it would drum senate on race. of you have a chat bot that starts viewing holocaust denial and nazi stuff because whoever programmed it didn't think that maybe we need to think about minorities and women because they get harassed online all the time. you have an industry made up of people who are greedy or ignorant of the problems people not in my class, they can be ignorant of the problems their algorisms create, and theirs accountability issue you raise next,ed to know about the and nothing optimizing the output. >> well said.

just want to add one the last thing. there's ban lot of recent publicized examples of spectacular failure's of algorithms coming from, like, the twitter bot and facebook algorithm stuff. that's what we get to see. most of the shitty algorithms that are ruining peoples lives are never seep by the public. they're business-to-business dealings, like the juan that was the personality fest for kroger's supermarket. so if we think that things that -- if things that are known to be under public scrutiny are that bad, imagine what is going on with the -- that can scrutinize these things. >> you're taking my mic now. right? >> okay. >> thank you. [applause]

>> this is incredibly interesting. talking about the algorithm as a device for controlling sort of -- establishing or maintaining a control system, and the algorithms is the device nat none offers us understands so it's presented as valid. it reminds me of other thing ins in society that accomplish the same function, like perhaps constitutional law or things that no one really understands but they're legitimatizing a structure that probably we ought to be getting to the bottom of. these other things have usually a -- like a priestly class that sort of is the only people who understand it. you guys come from this in algorithms. what is the class and what are they thinking about this and are you a renegade from this class or is everybody thinking about this? what is the deal?

i don't know any of these people really. >> thank you. yes, it is priestly class, and much to my shame, a lot of the people who are in this class prefer it that way. a lot of -- it's like actually pretty flattering to told you magical because year mathematician. not enough of us are worried about our impact on the larger society but die want to say that when i started this project, four years ago, i quit my job to write this book, i was really panicking because i didn't see nip around me who was worried about this. but now, four years later, have an entire community of people, including computer scientists, sociologists, anthropologists, technologists, that are all super concerned with this stuff and in particular i've been hanging out with some people who, like me, are interested in developing tools and auditing

tools specifically, to, like, audit the black box to look into the black box in some sense, and i don't want to make that mysterious, so what i mean by that is, if you -- in a sociological experiment we're borrowing from sociology. if sociologist want to see whether a hiring practice was racist that it would send a bunch of applications and with reasonable -- similar qualifications and have black names and white names and see if the white names get more call back for interview. you can do that with an algorithm, too. similar things. it's crude and it's the first generation of auditing tools for algorithm but the type of thing we neil to standard doing. so long story long, we do have work to do, and our priestly class of technologists who work

with da a to but we -- data but we have a growing group of people who want to think be this. and also, like, a field that should be developed, right? i feel like it's 20 years it there will be conferences before the this thing. right now there's nothing. as far as i know there's not even a journal that is interested in publishing things like this but there will be. and by the way, i would start -- say that the beginning of this, the field has already taken root in data journalism. if you look at republicca, the recent work with auditing the recidivism. we talk about residism models because the models that ups use to sentence people to jail, and they cyst, and one of the -- racist and one reporting team at republicca got a bunch of dataly to a foia request and found it

to be racist. but this is the early stages. >> thank you. this is wonderful. i'm a physician, a. bit late. but i wanted to get two areas in which this might have something to do with medicine. there's tremendous confidence in the medical community -- could you speak up? >> i'm sorry. i'm a physician. big data comes up in two ways. the first is that the medical community and the business community are convinced that all the data going to at the electronic medical records is going to give us brand new insight into taking care of people and health, and i wonder -- it's not your field but i wonder what you think of this. i'm skeptical. the second thing is, the other -- another big push in medicine is to force individual physicians to take risks with a patient they take care of.

that is to say if a patient doesn't require so much here, you do well. if the patient requires more care, then you anticipate, you're screwed. and we're not insurance companies itch wonder if there are algorithms that will be developed so that physicians and insurance companies,s, even though everything is supposed to be on the up and up and we don't do preexisting conditions, they'll use information that is publicly available or reasonably publicly available to furring out who the risk and who is not. >> great question. i am not an expert in your field, as you said, but i want to say that i worry as well as you do about all the hype around how much data in the medical field, and so i would focus as a nondata person -- and i would focus even though i am a data person, if i were you, on, like,

accountability, and explicable testing protest procedures. let me explain with -- by analogy. this problem with the teach are stuff, i said it before but i'll say it gowned. there's no ground troops. compared to what? how do we know this is any good? how correlated is it to the thing we know is good? it's so bad, you have to actually demand the kind of accountability that would make you trust it if martians came down from mars and said i have a way of fixing medicine you're like, i don't trust you so convince me. this has to be convincing in a real way. right now we just -- people just say, big data and throw the out there like it's already convincing. but you and the doctor need to also be convinced you need to be shown why it's really good, how

it is agrees with you in ways you absolutely know are true and therefore you start trust trusting in ways you're not quite sure yourself. having said that -- an example of how algorithms -- >> testing -- >> so, there's an example i wanted to throw out there, which is that an algorithm -- just imagine we have really good big data algorithms for medicine. if we did haven't that and we can predict future illnesses well, that is not necessary lay good thing. and that's one of the thing people get wrong when day talk about precision medicine. which is that, like, in the hands of your doctor trying to help you stay well, it is probably a good thing. if it is done well and that's a big if. but in hands of an insurance company who can charge you more for future illnesses that's a terrible thing. some would was not addressed by

obamacare. obamacare stops preexisting conditions, not future conditions. and finally, imagine that in the hands of walmart and i'm not saying it us but on the large moyer who gets to choose who to hire and not hire based on future insurance costs. that would be really bad. so i'm just saying, like, there's no guarantee. even if a algorithm is accurate, it's going to be used for good. >> just wanted to chime in on the end of that. i'm kind of in this space, building the algorithms and they terrify me. so we were talking about, we just published a paper where we're looking at people who are starting treatment in aa. going to their firstollics anonymous meeting elm we analyze theirs twister profiles from before they start going to a and can predict with 85% of 90% accuracy if they going to stay sober.

a lot of colleagues work 0ing on this. group looked at women' and the their twitter accounts over the course of their progress innocence say and on the day of their birth could develop if they've'll develop post partum depression or not. great for your doctor. right? if you can go to your ob/gyn and she can push a button and say we need monitor you for this, that's got great, but for aural the cases you mentioned that's scary, and there's work in this community i'm involved in, looking at things like diabetes 0, beesty, heart disease -- obesity, heart defense, and conditions like ptsd and depression, and science is interesting that we can do this. orbit the ick mix indications and the fact there's no regulation over how you use can the output of the algorithms and no one can know if you're doing it, think is terrifying, and i spend a lot of my time going around talking to big companies saying you need to be careful about this, because bad thing is will happen.

so that is a really scary space, especially being down there, building and looking at algorithms. i have a very utopian view of where we're going. >> we're all terrified, right? i didn't even know that you could do that with twitter. that's awful. >> so, talking about the medical profession, reminds my of hipaa, and so in relation to the kroger vocational testing thing, why isn't that kind of information as sensitive and hence as worthy of being protected bay law like the pin hipaa law as the medical information that is protect bid he pippa la. >> thank you for saying that, sir. that's one thing call for in my book.

it would be even batter if you had read the book. >> i don't have the perfect algorithm on picking everyone so the hands in the audience so bear with me. >> my names jamal, i'm a reporter on higher education. one things ways hoping cuckoo address us is potential benefits in the case you cite, for instance, with the teacher example, it seems as though it was just the use of the information, maybe a wrong analysis but seems like it actually helped identify in the instance of cheating, and education, so, i wonder what you would you say to that. it's not really the algorithm but the way that they're used, and along similar lines what do

you say to businesses in the second example that you cited in terms of using personality tests that for. the it's a matter of efficiency. and that it's a matter of saving time on the front end of the hiring process. i'm sure you anticipated that. i'm curious as to your response to that. >> tanks. with respect to torcher value added model. the only reason there was cheating is because the structure of the model itself was -- had too many incentives so it basically -- when you have enough innocent testify is in play and high pressure and high stakes, it causes cheating. so, i don't think it makes -- it's circumstance lay reasoning to say one good thing that happened from this regime is it detected cheating. i'm not an education expert. so i don't actually have solution for education. but i did -- the thing wanted to point out is like -- it's kind

of the lowest bar of all my examples because it's just a bad model statistically speaking. now, i'll tell you my evidence for this. besides usingings in like some people were getting scores for classes they didn't even teach and it was like a computer error. crazy stores like that but also this -- i told you guys that the "new york post" shamed all of those teachers. so there was this high school math teacher, gary, in new york city, who did something clever itch couldn't get the access at the source code so i couldn't actually look at this model, but what he did was he found teachers that had two scores because they main taught seventh grade math and eight grate brought you expect -- this is supposed to be an overall score as teacher to be consistent. 78 for the seventh grade math,

marsh 82 for ailing grade math etch he plotted that and found uniform distribution, which is to say practically as likely to get a 90, comma zero as to get a 40, 40. just all over the map. there was no -- in other words, almost a random number generator. so there's no reason to think this is informational. when i say almost -- 24% correlation. which is very low. which is to say you couldn't possibly take a given teacher and say this is your fault you're a bad teacher because you good at low score. you can't do that. not strong enoughed but it's no a zero percent correlation, marsh at a district level you can glean useful information but i just want to throw in, you would only be able to glean information about test scores, right? test scores are already funneledmentalitily not enough information to understand whether students have good teachers.

so, again, like not an education expert but i don't think so. now, in terms of the hiring practices, my complaint with the kyle beam's personality test is that it was original and secret. right? assuming that we get evidence eventually that it is essentially a mental health assessment. which we think it is. we have evidence for that. so, like, assuming that's true, we can't figure out directly because it's secret, but assuming that true, the problem there is that it's illegal. it's against the law. there's lotoses laws around hiring. you have to be -- you can't discriminate when you hire in all sorts of ways. don't actually have probable limp with using big data to help filter resumes but has to be legal at the very lest and

should be transparent as well so people understand what they're being telephonessed for. -- what they're being tested for. >> you paint a very challenging picture and frightening picture of the misyou of algorithms and that sort of thing. the average facebook posts has no idea what aen algorithm is and those people mow e most affected by the failure of the housing crisis still don't know that any algorithm had nothing do with it at all. how do you get this information out besides from buying your book? threat important because most people still don't have any idea of the impact of algorithms on their lives. yeah. that's why i wrote the book. i feel like people don't really understand. most people at least as of like 2012 oar so when the survey was done didn't reallyize that facebook has an algorithm.

so, they are not even aware of the algorithm. they're starting to fade. example, discussion around the trending news stories, people are starting to see that they're not seeing everything. that -- and some populations have, like -- does trickingy things and figure out how to get friends to see their posts if you just post something your friends might not necessarily see it. heard about this teenaged population that puts brand names into the post, like coke, and then it gets pushed out to the rankings so your friends get to see it, which is fine. that's like a public goal so first make people aware of the algorithm and then make them skeptical. >> one interesting point on this, if you look at surveys

from, like, the pew studies of internet in american life, a great representative study, awareness of the algorithms but also concern about the impact. it's very clearly core lefted with socioeconomic status and also race. so people who are poorer are less worried about these algorithms and less aware of them. which of -- which if youer talking about being exploited by algorithms the people who are most likely to be exploited -- i'm not going to be exploited that much but the people who are most like live to be exploited are also less worried and less aware so it's kind of a cycle, there's nothing to worry about. nothing bad is goes to happen but you're the target. so that mikes its even more complicate sod their stewed study would be interesting to look at. your book is a great way to get this out there i like to go around and talk about it but i think we just need to massive education campaign because these algorithms are so powerful and

so impactful on our lives that it's kind of a basic kind of literacy we need to start develop ago to at least know there, if even if we don't know the details how the work. >> one more thing. it's not enough. it's not enough to say, everyone should be aware and everyone should protect themselves. because it's exactly what jen just said. the reason i wrote this book is because i saw this time and time again as a class issue. this is -- these are tools of social control and the beam who are at the top are controlling the people at the bottom and it's not enough to i say you should have been aware of the possible exploitation i would have done to you. that's why we actually need rules about this. it's not just like everyone for themselves. that's not good enough. >> so,. >> it's like karaoke.

close to your mouth. >> i'm a data scientist and a stat senator blog and i work in health care payments, and i work in healthcare payment and so a plot of our data modeling is using algorithms to determine things like payment, which you talked about. my question is, as building on your point that it's okay to be using al go from things like hiring practices or for making some sort of large scale decision where big data would help but i wonder -- this is probably covered in your book -- what you think a good solution is in terms of the algorithms. thes? what is the correction mechanism here for dealing with this inequity that apprises in application. there's a way that data scientists can be thinking about these inequitable implications when they're building the algorithms from the ground up. >> yes, thank you. so, the thing about the promise

of big data is that it has not been fulfilled yet but still exists. the promise of big data is that we can make things more fair, and i think we can. i mean, i'm not holding my breath but i think we can do that. part of is this realization that just because it's an algorithm, we're not done. part of that is developing tools to audit and see whether it is currently fair. part of that is developing tools to say how could defiction the algorithm? it's nat fair. how do we make it fair? so the good news is that when you look -- let's compare a company that has, like, discriminatory approximates for hiring with an algorithm that that discriminatory approximates for hiring. the problem with the company is if you interview people and ask them how the choose who to hire they will lie to you, maybe they actually don't reelize they're being discriminatory. people don't realize they're

buys buy assed. an algorithm will not lie. once you actually have a successful algorithm, that's good. but you have to have this sufficient trust in the algorithm to be able to know that it is actually doing good job. >> hello. can you hear me? okay, good. my partner and i did a fair amount of work on bernie's campaign earlier this year, and we experienced a lot of ex-aspiration and frustration at the fact that bernie's campaign -- and i'm sure the other democratic campaigns -- used dat to from a company called van, and what we were finding is that the data was sending us in terms of canvassing to a lot of neighborhoods that in our opinion did not match up with the demographic wes knew to be bernie supporters so it felt frustrating that the data didn't

seem to be working for us and hour higher ups didn't see that just because there was data, certain demographic, didn't mean that was the best demographic target. but beyond that on a broader scale i'm concerned about the fact that seems to be contacting a certain type of voter and not another type of voter. we seem to be going a lot of wealthy neighborhoods and not a lot of less affluent neighborhoods and so certain sorts of people are being reminded to vote and others aren't. so, i wanted to ask if you looked at van and if you hadn't, please ask you to look at van. and in terms of -- if there's a place where i think this algorithm, the existence of it is possibly actually not the best thing for society. don't the there should be a private company that owns the data on voters and lends it out to campaigns. what can we do about that? where is the power struggle here?

>> great question. i have a chapter on politics and data, and the subtitle of my book is, how big data increases inequality and threatens democracy. and this is the kind of the second part. trust democracy. i think just having, ace sir baited inequality at into within affects democracy bus some people have such confusing and highly scheduled lives they don't have time to be civically engaged but putting that aside, the threat to democracy i see coming from big data is the political million growing starting stuff and it is un -- microtargetting stuff. think you're absolutely right that targeting only special people and the special people typically are either people that can be counted on to donate money so they have bigger boyses

or people in swing stayed and are swing votes, a very small sliver of the population, and it also is action ex-sasser baiting they don't spend money on people that they think would definitely vote or definitely think will not vote. they only spend money on people that may vote. but if somebody votes but votes, they -- if somebody doesn't vote they're typically ignored. so the sort of -- a very socioeconomic leaving in terms of voting. but the larger issue is this. the larger issue of politics and data is this. what is efficient for campaigns is infee efficient for democracy. what is efficient for campaigns is to know everybody here, profile you perfectly, and send

you the message that the campaign wants you to hear. that not what is good for us as a group. what is go for us as a group is a broad public discussion of how the candidates stand on various issues. but that's not what is happening with political microtargetting, so if i -- if rand paul target me hi would find things i agree on him on, which is break upping the big banks and then if i went to his web page to think what the says about other stuff he would kole the misa cookie and say, oh, she is the one who wants to break owl big banks. show her that. they're controlling the information i have about them. that's not democratic. we want information about candidates to be utterly open, and we want to have more information about them than they have about us and that's not what is happening. >> let me follow up on that eye. going to play the let me freak

you out more role. understanding who people are going to vote for is something we're pretty good at. in the last election we had 97% accuracy in guessing who people would vote for. here's the scary part. if you remember in the last election, they had a little thing that some of you probably saw that said, hey, here's all your friends who have voted, or 20% of your friends have voted. and they studied the impact that had on people. and something like they could get half a percent more people to vote if they showed that than if they didn't. so they want to get stuff -- they find other people who the think will vote for donald trump and then show them the things that gets more people to vote but don't show the it to the people who will vet for somebody else. then suddenly they're having a really significant impact on the election itself. don't think they have any

aspirations -- relatively apolitical in these sorts of things. you're giving me a skeptical look. i'm not calling facebook out specifically but the fact its the companies that control what you see with these algorithms has the option to have real impacts on elections by understanding who gets to see stuff theft going to make you go vote and hiding that from people who don't, and we haven't seen evidence of that happening yet, but that is a real possibility with the technology we have today, and i think one of the most concerning things in the this political space. i. >> i look skeptical because opopulation of people on facebook is already not bipartisan, so if they showed everyone that would be fair. but they show it to everyone while have more impact on the democratic election. >> we have time for one more question.

>> so i look for an online advertiser that specializes in political web sites, and something that's happened over the last year is that a lot of major advertisers have gotten really skittish about running ad's rite wing web site as the right wing has been radicalized. half noble intentions and half being able to say possibly. saying, don't worry, naacp, we don't sell t-shirts 'owheel who read breitbart but when we pull out of market we look for who comes and replaces us, and everybody who is replaces us are online advertisers who sell weapons or videos about how hillary clinton has a video that if you see -- if any american

sees will guarantee she is not elected and this is the really intense ideology segregation that is happening on the internet, is because mainstream advertisers are pulling out of right wing web site are web sites and the impact loom is increasing because even thy ad are get buyingassed, and i think it was better just main stream, like google is selling them cars as opposed to people telling them really big knives. and this is something that is done through algorithms because breitbart, they're hundred's thousands you don't and you discover through algorithms that account managers, we are selling advertising to racists? so any are wonder if you could comment on this phenomenon. >> it's -- i think larger than my book.

i don't know how to address it. the growing partisan divide in our country is not singularly due to algorithms. the way live in silo e lowed paces on facebook and talk to each other echo chambers is part of itself but not the only part. i don't know how to -- i agree tate a problem. we'll agree on that. >> all right. well, thank you so much, cathy, for coming out, thank you, jen for joining us and thank everyone else for coming out. a great talk and a great conversation. we'll have becomes available for purchase in the book store and congratulate will be here signing afterwardses. >> thank you, guys.

applause [inaudible conversations] >> booktv record awe to programs. on monday, associatingallist looks at the political divide in america at book culture book store in new york city. she was rently named a finalist for the national book award for nonfiction. and on tuesday, at powells book store in portland, oregon to -- a panel of authors will books challenged due to their ken tent, wednesday as part of george mason's -- claudia

cowlell provide a hoyt of enemy health. and then thursday, the examination of the relationship between eleanor roosevelt and lower recent hickcock. saturday, bradley will receive this year's book award which recognizes the best book that advances conservative principles for his recent biography of russell kirk and on sunday we are with live with gerald horne. and that's a look at the time at the awe their proms booktv is covering this weekend. many events are open to the pockment look for them to air in fear future on booktv on c-span2. >> historically, diarrhea was -- there's a -- dr. william ago her had a great quote.

father of modern medicine. he was the one who said dysentery has been more featal fatal to soldiers that powder or shot. in the mexican-american war, most soldiers died from dysentery. and you would have field camps and you have the mess tent where you're preparing food, no refrigeration, and an open pit latrine and flies unbelievable number of flies attracted by if any bodies were around so the flies are lanning in the latrine material and then buzzing over to the beans, which are sitting there so the flies are on the can which means they land on the crap and then have path generals on their feet and then inoculates the identified which sits there two hours and then the whole cam gets dysentery,

yeley fever, -- yellow fever, whatever it now it's -- there's good hygiene on bases, there's -- the bases have air conditioning so the whole dine facility can be sealed. you don't have to open a window so no flies anymore but it is a problem if you're, say in special operations going out in a small unit, in a small village in snow hall ya andem en, they're eat what the locals eat and the water is often not safe and the food and the rates of dieee with a those folks are twice the rate of ordinary enlisted zoning. if you were going to -- i don't to the -- take down osama bin laden and you're witness if extreme gastroinestinal urgency that's a problem. >> do they have any miracle drugs.

>> i went all the way -- i went to djibouti, africa are for the diarrhea chapter. that's the kind of gal i am...