[laughter] >> host: helen benedict is the author of this book, "the lonely soldier: the private war of women serving in iraq." she is also the author of this novel, "sand queen," based on the same research, "a novel." helen benedict has joined us here at columbia university. ..

and that these books were written in part out of desperation. i was having to think earlier with a grad student and we were reflecting on how we struggled in some of our economic stat classes. "naked economics" was written almost by accident in the sense that i've been assigned a class to teach economics to journalist. i called my -- i was unsuccessfully trying to sell a book on the gambling industry. never got that went in. it was a good book to be written and i said to her, i have to teach this economics cost a bunch of journalists. the textbook would be inappropriate. i can't find anything that would convey why they should care about this. that wouldn't consume with the math and the questions. it was a long pause and she said, you are going to write it and it's going to be called economics for poets and i'm going to read it. so that's a "naked economics" was born.

and wants that kind of found a niche among people who been scared away, or bored to death, but economic classes, we said let's go back and lets to statistics. anyways even more mathematically daunting on the other income it's even more -- economics has always been kind of economics. statistics, think about it, you go back 15 years when you give someone your credit card and they did that little slide thing and went into a bucket somewhere and that was filed somewhere. so nobody had a digital record of what you bought. computing power was more expensive and more cumbersome. fast forward now to the point where they will scan the book, they will take your credit card data, and i'm going to read about how alarming them might become, not necessarily a, but the confluence of the ability to

digitalize that and then cheap computing power means that you know more about people than we have ever known. and those data are cheap and easy to manipulate, use, study, for good and for ill. what i thought i would do is read for short sections. the first is a motivation for the book. this will revisit my uncomfortable relationship with math. then talk about a probability that is older but give you some sense of the power of statistics, how if you really know the underlying math you can use it for good. i'm going to give you one that he think is kind of a wakeup in terms of should all be a little more aware of what's happening with the data that we are throwing out. and then i'm going to finish, a bunch of open questions that i think this is cyclical to inform but we don't have the answers to. one for example, is what scoble going to look like in 10 or 15

years with head trauma and those type of things. a lot of statisticians another researchers looking at that question. but i do think there will be some interesting developments over the near-term for the. let me start with the introduction of the book. the introduction is called why a calculus but love statistics. this is a totally true story. i've always had an uncomfortable relationship with math. i do not like numbers for the sake of numbers. i am not impressed by fancy form of said no real-world application. i particularly dislike high school calculus for the simple reason that no one ever bothered to tell me what i needed to learn it. what is the area beneath? who cares? now that my daughter is going through it i still don't care. one of the great moments of my life occurred during my senior year of high school at the end of the first and most of advanced placement calculus but i was working away on the final exam. admittedly less prepared for the

course that i should have been. i had been accepted to college a few weeks earlier, so as i stared at the final exam question, they looked completely unfamiliar. i don't mean that i was having trouble answering them. i mean that i didn't even recognize what i was being asked. i was no stranger being unprepared for them but to paraphrase don rumsfeld, i usually knew what i didn't know. this exam looked even more greek than usual. i flipped through the pages and then more or less surrendered. i walked to the front of the classroom where my calculus teacher, whom we'll call carol smith -- in the original draft had said carol miller because that's her name. the publisher said we're not going to do that. so carol smith was proctoring the exam. i said mr. smith, i do not recognize a lot of stuff on this exam. we had a contentious

relationship at the point to suffice it to say that mrs. smith did not like me a whole lot more than i liked her. yes, i can now admit that it sometimes use my limited powers to schedule all school assemblies just so that her class could be canceled. yes, my friends and i did have flowers delivered to mrs. smith during class from code a secret admirer, unquote just so we could see what would happen. and yes, i did stop doing any homework at all once i got into college. so when i walked up to mrs. smith in the middle of the exam and said the material did not look familiar, she was well, i'm sympathetic. charles, she said loudly, to me that she was facing the rows of desks to make sure the entire class could hear, if you had studied, the material would look a lot more familiar. this was a compelling point. so i went back to my desk. after few minutes brian, and

this is his real name, a far better calculus student and i walked to the front of the room and whispered a few things to mrs. smith. she whispered back and then a truly extraordinary thing happened. lass, i need your attention. it appears that giving you the second semester exam by mistake. [laughter] >> we were far enough into the test but that the whole exam had to be aborted and rescheduled. i cannot fully describe my euphoria. i would go on and left a merry a wonderful we have three healthy children. i publish books and visit places like the taj mahal. still, the day my calculus teacher got her comeuppance is a top five life moment. [laughter] now, i way of disclosure, the fact that nearly filled the makeup exam did not distinguish,

did not diminish in any way from this wonderful experience. so that's where i come from when it comes to quantitative exercises. there is a sad footnote to this, which is by the time i got to graduate school, i was feeling a little bit more confident. i'd gone to math camp and was thinking of getting my math feet underneath me. the woodrow wilson school, they had the resources where they could offer most of the icon and start classes with the without calculus. and by the second semester i was thinking you know what, i'm ready for calculus. i had kind of growing into this. i felt really confident so i took calculus and i kind of got enough. unfortunately, the class without calculus were taught by ben bernanke. [laughter] so much hubris meant that i did not study natural economics with the current fed chair. which is unfortunate.

so that is where i come from. i have in life because i do public policy, -- the enormous power of these tools and the mouth, but really it's about can you tell me what i need to know this. so i want to read the is a section from chapter five which is basic probability. the subtitle is to not buy the extended working on your $99 printer. that you can probably figure out. instead of this story that some of you may room, this may seem all but hope you recall this. in 1981, the joseph schlitz brewing company spent $1.7 million on what appeared to be a shockingly bold and risky marketing campaign for its flagging brand, schlitz. how many of you are never schlitz beer? good. so, you know, the campaign

didn't really work. and it's good for this time of year. at halftime of the super bowl in front of 100 million people around the world, the company broadcast a live taste test putting schlitz beer against michelob. boulder yet, the company did not pick random group thinkers -- beer drinkers. it picked 100 michelob drinkers. this was the culmination of a campaign that had run throughout the nfl playoffs. there were five live television taste test in all, each of which had 100 consumers of competing brands, conducted blind taste test between the supposedly favored beer, and schlitz. each of the. taste offs was promoted aggressively just like the playoffs themselves, as they watched schlitz versus bud live during the afc players. how many of you remember that? that's good. the marketing message was clear. even. drinkers who think they're like

another brand will prefer schlitz in a blind taste test. for the super bowl spot, live at halftime of the super bowl, schlitz hired a former nfl referee to oversee the test. given the risky nature of conducting blind taste test in front of huge audiences on live tv, one can assume that schlitz produced a spectacularly dilution severe. that was not the case. schlitz needed only a mediocre beer and solid grasp of statistics to know that this point, a term i do not use lightly when comes to beer advertising, would almost certainly work out in its favor. most beers and the schlitz category tasted about the same. go back, you should remember the '80s, all the beer was bad. ironically that is exactly the fact that this advertising campaign was exploded. the typical your trigger off the

street cannot tell schlitz from the globe to miller. in that case a blind taste test between any two of the beers is essentially a coin flip. on average, have the taste testers will take schlitz and half will pick the beer it is challenging. this fact alone would probably not make a particularly effective advertising campaign. a matching quote you can't tell the difference so you might as well drink schlitz. [laughter] and schlitz absolutely positively would not want to do this taste test among its own loyal customers. roughly half of whom would pick the competing beer. it looks bad when a beer drinkers supposedly most committed to your brand pics the competitive in a blind taste tests which is exactly what schlitz was trying to do with competitors. they did something very clever. the genius of the campaign was conducting the taste test exclusively among beer breakers -- beer drinkers who said they'y

like another beer. if it is really going flip, then roughly half the blood wiser or michelob drinkers will end up drinking schlitz. half of all bud drinkers like schlitz better. and it looks particularly good at halftime of the super bowl with former nfl referee in uniform conducting the taste test. still, it's live television. even if the statisticians at schlitz determined with loads of previous private trials that the typical michelob tranquil picture let's have the time, one of the 100 michelob drinkers take the test during the super bowl turn out not to? a blind test is equivalent to a coin toss, but what if most of the tasters chose michelob just by chance? after all if you lined up the same 100 guys and into political and, it's entirely possible that they would flip 85 or 90 heads. that's the kind of bad luck to taste test and would be a disaster for the schlitz brand,

not to mention it would $1.7 million to live television audience. statistics to the rescue. if there were. if there was some test assisted superior, i have in mind statement man but that would takes a different direction. this is what he or she would have swooped in and unveiled the details of what statisticians call a binomial experiment. the key characteristic of a binomial experiment is that with a fixed number of trials, so 100 taste testers for an example to each with two possible outcomes, schlitz or michelob. and the probability of success is the same in each traffic from us and the probability of picking one. with th other is 50%. and i'm defining success as the taste of picking schlitz. we also assume all the trials are independent and that one blind taste testers does it has no impact on any other taste testers. with only this information, a

statistical superhero contact with the probability of all the different outcomes for the 100 trials. 50 t52 schlitz and 40 michelob 0 when schlitz the 39 nicholas. those of us are not statistical superheroes came get a computer to do the same thing. the chances of all 100 taste testers picking michelob were, does anyone want to take a guess? this number is to be. it's one with, about 27 numbers after it. essentially you're more likely to get hit by an asteroid while watching a trial. more important, the same basic calculations can give us the probability for a range of outcomes such as the chances that your than 40 of the taste testers take schlitz. these numbers would clearly have

assuaged this schlitz marketing folks. let's assume schlitz would've in place if at at least 40 of the 100 schlitz. an impressive number. remember, given that all the men taking the live blind taste test, i think tha they're all m, have professed to be michelob drinkers. and out, at least that good was highly likely. if the. test really is like the flip of the coin. the basic probability tells us that there was a 98% chance that at least 40 of the taste testers would pick a schlitz. and 86% chance that at least 45 of the taste testers would. in theory this was not a very risky gambit at all. to do and want to guess what actually happened at halftime? [inaudible] [laughter] >> at halftime of the 1981 super bowl, exactly 50% of the michelob drinkers chose schlitz in a blind taste test. so two important lessons.

one, probability is remarkably powerful tool. and, too, many lingers in the 1980s were indistinguishable from one another. this chapter will focus primary on the first lesson. so that should make you encouraged about the power of probability of statistics in general. so now i'm going to make you scared. so this is the end of the book and it's a question from one of the questions is who gets to know what about you? last summer we hired a new babysitter quencher private house i began to explain our family background, i'm a professor, my wife the teacher. she cut me off she said oh, i know. i googled you. i was simultaneously related i did not to finish my spiel and mildly alarmed by how much of my life could be cobbled together from a short internet search. our capacity to gather and

analyze huge quantities of data, things ar i referred to earlier, the marriage of digital information with cheap computing power and internet, is unique in human history. we are going to meet some new rules for this new era. let's put the power of data in perspective with just one example from the retailer target. this is a story that is told in a new times magazine and i summarized the. like most companies, target strives to increase profits by understanding its customers. that's a very good thing. to do that, the company hires statisticians to do the kind of predictive analytics described earlier in the book. they use sales data combined with other information on consumers to figure out who buys what and why. nothing about this is inherently bad, for it means that when you go to talk at your likely to find things you want to buy. let's go down for a moment on just one example of the kinds of things that statisticians

working in a windowless basement at corporate headquarters can figure out. i don't know it is a windowless basement. i'm assuming. target has learned that pregnancy is a particularly important time in terms of developing shopping patterns. pregnant women develop what they call a retail relationship that can last for decades. as a result, target wants to identify pregnant women, particularly those in their second trimester, and get them into the stores more often. a writer for the new york times magazine followed the predictive analytics team at target as it sought to find and attract private choppers but i can assure you target deeply regrets this. [laughter] but i am very appreciative. the first part is easy. target as a baby shower registry in which pregnant women register for baby gifts in advance of the birth of their children. these women are already target

shoppers and they have effectively told the store not only that they are pregnant, but when they are due. so how far along they are. here's the statistical twist. target figured out that other women who demonstrate the same shopping patterns are probably pregnant, too. for example, pregnant women often switch to understand and lotions. they begin to buy vitamin supplements. a start buying extra big bags of cotton balls if this is true. who knew? the target predictive analytics gurus identified 25 products that together made possible what they described as a pregnancy prediction score. the whole point of this analysis was to send pregnant women pregnancy related coupons in hopes of hooking them as long-term target shoppers. how good was the model? "the new york times" magazine reported a story about a man

from minneapolis who walked into a target store and demanded to see a manager. the man was irate because his high school daughter was being bombarded with pregnancy related coupons from target. quote she still in high school and use in her coupons for baby clothes and cribs? are you trying to encourage her to get pregnant? the store manager apologized profusely. he even called the father several days later to apologize again. only this time, the man was less irate. it was his turn to be apologetic. he said, it turns out there's been some activities in my house i haven't been completely aware of, the father said. she is due in august. the target statisticians have figured out that his daughter was pregnant before he did. all right, this is not even the creepiest part.

that's their business, and it's also not their business. it can feel more than a little interested. for that reason, some companies now mask how much they know about you. for example, if you're a pregnant woman in your second trimester, you may get some coupons in the mail for cribs and diapers, along with a discount on a riding lawnmower and a coupon for free bowling socks with the purchase of any pair of bowling shoes. to you, it just seems fortuitous that the pregnancy related coupons came in the mail along with the other junk. in fact the company knows that you don't fall. they know you don't cut your own lawn. it's merely covering the track so that when it does doesn't seem so sneaky. all right. let me finish with another question. i actually like the way of

finishing because there are good things about statistics. are scary things that statistics, and then there are places where we are watching it unfold right now in real-time. so i think are six or seven questions at the end of the book. one of them is how can we identify and reward good teachers and schools? my wife, full disclosure, is a high school math teacher. we need good schools and we need good teachers in order to have good schools. thus it follows logically that we are to reward good teachers and good schools while firing back teachers and closing bad schools. how exactly do we do that? test scores give us an objective measure of student performance. yet we know that some students will do much better on standardized tests for other reasons that have nothing to do with what's going on inside the classroom or the school. the seemingly simple solution is to evaluate schools and teachers on the basis of the progress that the students make over some period of time.

what did students do when they started at a certain classroom with a particular teacher, what do they know a year later? the difference is the value added in the classroom. we can eve even use this to geta more refined setting by taking into account demographic characteristics of the students in a given classroom. race, income, performance and other tests which can be a measure. if a teacher makes significant gains with students who are typically struggled in the past, then he or she can be deemed highly successful. we will now if i would teacher equality -- quality. how devious and he says physical evaluations work in practice? in 2012, new york city took the plunge and published ratings of all 18,000 public school teachers on the basis of a value added assessment to get measured gains in the student test scores while taking into account other

issue. "los angeles times" published a similar data to los angeles teachers in 2010. both new york and los angeles, the reaction has been loud and mixed. arne duncan has generally been supportive of these kinds of value added assessment. they provide information where none previously existed. after the los angeles data were published, secretary duncan told the times, silence is not an option. the obama administration provided financial incentives for paying and promoting teachers. proponents rightfully point out that they are huge potential improvement over systems of which all teachers are paid according to the uniform summer schedule that gives zero way to any measure of performance in the classroom. on the other hand, many experts have warned of these can teacher assessment data have large margins of error and can deliver misleading results. the union representing new york teacher spent more than $100,000

on a news paper added a campaign built around the headline this is no way to write a teacher. opponents argue that the value added assessments create false precision that would be of use by parents and public officials who do not understand the limitations of this kind of investment. this is a case where everybody is right up to a point. the results were giving teacher often based on a single test taken on a single day by a single group of students. all kinds of factors can be to random fluctuation. the correlation in performance from year to year were a single teacher that uses these indicators is only about .35.

so it means if you really good you should be good year after year. and they would argument-it. on the other hand, the correlation from year to year performance for major league baseball players is also around .35 as measured by batting average or hitters and era for pictures. there's one tool in the process. a day to get less noisy when authorities have more user data for a particular teacher with different classrooms of students. in the case of the newark city rating, principles in a system have been prepped on appropriate use of the value added data in their inherent limitation. the public did not get that briefing. as a result the teacher assessments are often viewed as the definitive guide to quote good and the bad teachers. we like rankings. just think of "u.s. news & world report" -- believe me, they're in for a drubbing in several

chapters but even when the data did not support such precision. he offers a final warning of different -- this is a really interesting study. it is better to be certain of the outcomes were measured such as the result of given standardized test, we have to be sure to attract with what what we care about in the long run. so everything to this point i assume the test scores are in it was should care about. and better test scores matter. some unique data from the air force academy was just not surprisingly that the test scores that glimmer now may not be gold in the future. the air force academy, there's a lot in a book about the importance of getting good data and how hard it is to get data to study certain things. the academy here is unique -- the other military academies are the same way, and that it randomly assigns its cadets to different sections of standardized core courses, such as introductory calculus.

this randomization of eliminates any potential selection effects when comparing the effectiveness of professors. so you don't get to pick your professor. you don't even get to pick your class. overtime you can assume all professors and students have various aptitudes to unlike most universities were students with different abilities can select in or out of class at the air force academy also use the same syllabi and the singing sands in every particular course. scott carroll and chen scores, professor at the air force academy exploited this elegant arrangement to answer one of the most important questions in higher education. which professors are most effective? you're all giving students who look like each other, same readings come same test, doing a good job. here's the answer. the professors with less experience and fewer degrees from a fancy universities, these professors have students who typically do better on

standardized exams for the introductory courses but they also get a better student evaluations for their courses. clearly these young motivated instructors are more committed to the teachings in the old crusty guys with ph.d some places like harvard. the old guys must be using the same yellowing notes for use in 1978. they probably think powerpoint is an energy drink, except they don't know what an energy drink is either. the data tell us that we should find these old codgers, or at least let them retire gracefully. but hold on. don't fire anybody yet. the air force academy study has another relevant finding. about student performance over a longer horizon. they found that in math and science students who have more experience and more highly credentialed instructors in introductory courses do better in their mandatory following courses and student of less experienced professors in the introductory courses. one logical interpretation is

that the less experienced instructors are more likely to teach to the test in introductory courses. this produces impressive exam scores and happy students when it comes to filling out the instructor evaluation. meanwhile, the old crusty professors whom we nearly fired one paragraph ago, focus less on the exam and more on the important concepts, which are what matter most on the following courses and in life after the air force academy. clearly, we need to evaluate teachers and professors but we just have to make sure that we do it right. a long-term policy rooted in statistics is to develop a system that reports a teachers real value added in the classroom to the sections are meant to be ambiguous because they work in progress. so i will stop there and be happy to answer any questions about this book, "naked economics," or anything else that might be on your mind.

thank you. [applause] >> i want to thank you for writing this book and also providing "naked economics." i hope this book sells as well as the only other statistic bestseller in history which is called how to lie with statistics. wait, this is a sequel to how to lie with statistics. when i was with my publisher discussing a next book, that book was published by norton who reads behind the. and he said update this. that book was written in the 1950s and sold a million copies. as mentioned but it is not coincidence. the one with your question. >> there are at least 8000 lobbyists who will buy a copy spent i wasn't taking away the right thing. >> i'm interested in getting to policymakers to take away the right thing. i worked with the clinton white house and one of the first things we did is ahead is us all memos from the president by charles schultz which was -- we

did have a nice book like this. some wondering if there's a foundation that might send $20,000 by 535 copies -- >> i hope so. [laughter] >> if they bite you they would get a 20% discount so tony 15%. in addition to the, what should we do to get policymakers a little more aware of a misleading most of the numbers they get everyday our? and particularly the poll results that they rely on so much. >> first of all, i'm not sure everybody cares. some of it is delivered. i think the best thing we can do, this actually applies to economics and statistics. i think what we keep seeing things we have to move away from the mechanics. so when i was taught statistics, it was his attitude calculate this. you don't -- you've got a personal computer on your desk made after 1985, they can do

that for you. what you need to know is what you're actually doing. this like a hand-held grenade launcher but it's not that hard to use. but it's very hard he is appropriately. it would be better off telling you not where to point. so i think the most important thing we can do is step back, talk about what's likely to make statistics go off the rails, whilwhere you're likely to get really bad day. you get really bad head and you will plug it in the most -- i think will spit out the wrong conclusions. i found myself having to go back and relearn the things i've been taught. and a lot of times it was oh, wow, that's it? a? i understand that. now understand. but the first time around not so much. so i think again, both in economics and statistics, how to

be a more intuitive approach for why this works and how it can go egregiously wrong. gates foundation maybe. >> very glad -- >> and everyone should have one for the office and one for home. you don't want to be carrying them back and fourth. >> thank you for your talk. i'd like to talk about -- heavily studied, been promoted, take it can reduce your risk of breast cancer by half. conversation i had with the doctor who is a specialist, and looking over the statistics i asked him, isn't it more accurate to say that it doesn't reduce the risk by half, but it reduces the risk for half the women who take it? and how does an individual woman know whether or not it's going to be, is she going to be in have that works with half that doesn't work?

no answer. yeah, you're right. so what do you have -- >> i have no knowledge of those trials. there is a fair bit of talk about clinical trials performance for a product. in part because you end up with some -- that's when police were statistics can steer us wrong. i will give you a very basic point that applied to pharmaceutical products, it is simply the median. so the problem with the mean, and this gets explain a little more comfy as it gets old but outliers but if you look at me in household income in the country it's like it's going up in a nice healthy way. of course, that can be explained by slow of income growth at the top. people in the are not given any wealthier at all. that's one drawback of the mean. so you might say we should just never use and meaning. we should use the medium which is the middle point of the this addition to the way i explain it in this, 10 people sitting on a

barstool and each make $40,000 a year, and bill gates walks in insist on the 11th of barstool, the average income in the bar goes through the roof. the median doesn't a little bit in the case the meeting is the right one because none of those 10 people have gotten any richer. when you look at pharmaceutical outcomes, this guy, i forget his name, who wrote a book called the mean ashton the median is not the message. we looked at median the problem is if there is a drug that is successful for 40% of the people who take it, and they live for another 25 years, but safety president of the people don't live at all, doesn't mean life in expectancy is going to look very, very low. so this is one more descriptive -- people are affected in radically different ways choosing the statistics matters, or choosing both. the other place that clinical trials, up is they appear to be

a lot of public -- if i went out and did went out and get the status and watching watching television for eight hours a day does not make you healthier, and i shopped it around the ally going to get it published? the problem is all research is based on probability. so if 100 of us got into this, a few of us are going to find just by chance that these folks who sat on the couch and watch tv happened to get healthier. so when they shop the study which is what an outlet, 95 of us have watching tv doesn't do us any good. but a few of us have found that watching tv is correlated to healthy outcomes. someone will say you can watch tv and fix heart disease? we've got to publish that. and so there is -- it's a serious problem in the medical community that if you don't look at all the studies that are being conducted, the ones that are of most interest, the ones

that show this works, who wants to read a study that something doesn't help cancer? they are trying to do better. the medical journals are now requiring that you registered studies in the beginning, but there's been kind of a study of the studies tended to suggest we may not be getting a terribly accurate -- that doesn't directly at you you question whs probably the best i can do about the knowledge about drug. i will say that you these very cautious data. >> i notice you're a professor at dartmouth college. that was when my first teaching job a few years ago. i was a john wesley young research professor in the math department. >> great contributed to math and computer science. >> a great place to start your academic career. anyway, my thesis was written in

probability. i didn't statistics also. i just wanted to make a few comments before coming to my question. you spent a lot of time in the book about -- i was teaching a business stat course which a colleague of mine described cynically as follows, i asked him what's the difference between business stat and sophisticated business stat course is one in which every observation and your data set has a dollar sign in front of it. anyway, i spent time -- you can't prove simple limit theorem in a basic course. beautiful mathematics but you are just way over the heads of students. but anyway, you can get an intuitive description of what the theorem says. it was my reward after doing that, student evaluations at the end of the semester.

professor, you talk too much. you explain to much. cut out the smalltalk, just give us the formulas. and so, i mean, i try to do what you would like me to do, and that's the reward i get to the other thing is, here's another story. i will get to my question shortly. mean, i feel like the ghost of hamlet's father. i put a tail -- this i heard from a colleague. a student wrote an evaluation of a math professor as follows. this professor, the student book, makes it awfully tough for the average student to get an a. spent i think that's one of my students now. spin i hope these are better. but anyways, i'm glad you talked

about value added modeling. i want to call your attention to a topic that i'm interested in because teachers and you are getting fired right and left because of this abusive this is a. for example, here in washington, there was a fifth grade teacher who was fired because of the value added model. what happened was, her students do not perform as well as predicted. okay, it turns out that scores of the students in previous grade level, i think fourth grade, were, benefited from cheating. that is, the staff just erase the incorrect answers and putting the correct and she. the students came in with a higher predicted value than they deserved. so she was very worried because she saw their scores from the fourth grade tests and they

didn't correlate with what she was singing the classroom. so there's this vulnerability to fraud and cheating and so on. anyways, i talked to ron, executive director of asa, american statistical association. he said there's a special committee of the asa says going, they are analyzing this value added methodology. by the way, i myself just wrote a very simple paper but which illustrates very simple binomial model that explains why the teacher can be rated very high one year and be rated very low the next you. is because of the natural variation due to chance. >> that is a really terrific point. it's going to be with us, and it is. just a couple observations around your question.

one is one of the fun parts of the book is using probability to catch cheating on standardized tests. that are private firms now, and the methodology are interesting were if you run the answer sheets to a scantron, there are things the flag cheating. so for example, a high proportion of the racers that go from wrong to right. and it tells the story, a huge cheating scandal where, i can't remember the numbers, the probability of observations pattern of erasure's they observed were someone described as the probability of having filling the astronomical so of people who happen to be over 70 told that it's very unlikely. and so again that doesn't prove anything so if they drilled down, and to your point what they discovered was that teachers and administrators were having pizza parties on the

weekend where they brought the answer sheets and all changed answers at the same time. we're going to have to do this compromise will come together and to together. that's the most pernicious egregious example. my wife was a teacher. we're in chicago for a long time. charter schools and schools that return a schools and so on. they didn't quite cheat, but it was very close. the rule in illinois is you don't have to take anything down in your room during the testing. so my wife went to a teacher development day which was focused exclusively on maximizing the surface area of your room, including the inside of the door so your students will be up to look somewhere, she income anywhere and see the answers that they may need. like that's what they study, which is probably not the intent. and alas, my kids came home and said we love that.

why you love that? because they give you chocolate at the beginning of the test. they couldn't give them a lot day but they could and put them up on caffeine. >> my question, have you ever heard of campbell's law? >> no. >> pardon me, i would again. the greater the social, economic consequences of say with the statistics such as test scores, the more likely it is that the statistic itself will become corrupted. and that is corrupting the social processing it is intended to monitor spend time writing a short piece for "the wall street journal" to summarize some of the key findings. one of the headlines is smart managers will use statistics to evaluate employees, smart and playful figure how to manipulate those statistics. the one that is quite scary in the book from new york states where they simply decide that

they would provide data on mortality rates or angioplasty, for cardiology. information is always good, it turned out that the new times are somewhat followed up and the high proportion, something like 78% of cardiologists have it delivered a change their behavior because of the evaluation. it was not to kill fewer people, it wasn't like boy, i killed a lot of history. today i'm not going to drink, right? [laughter] it was on the margins they decided not to take the most seriously obese patient. so the net effect, i hadn't heard the law but that is one of the great takeaways is that people who are sick is, the most to me, probably some kind of intervention were least likely to get it because of the evaluation. but i think it's true. >> the most dangerous place in the world is a hospital.

>> campbell's law will be the next edition. it's a name for an important concept, thank you. >> so, there's a space in some quarters that data mining and machine learning, do away with, you know, machine learn your way to do a cure for cancer. some people, that's on one side of the house. the other side is you can't take the people out of the statistics. starting with a hypothesis and testing it from a very base. enlargement on both sides of that, that machines are messy. what do you think? >> i think it's not -- i think, we're talking breaking the about using high-powered computers for medical diagnostics. and i think there's a lot of

potential budget there. particularly to the number of medical errors and everything. my caution is, one of the books early in the book is a distinction between precision and accuracy. and in time a statistical model can give you a false sense of precision. in the book, one example, one quite serious but if anyone is my wife gave me one of those calls rangefinders for christmas. i grew up playing golf and you look at the pine trees and to look at the pin, i'm about 150 yards. you get the rangefinder and you can shoot it out the pen in your 148.7. this is amazing. my golf kept getting worse. [laughter] i was hitting trees and is hitting the center. at one point i hit the municipal parking lot were all the police officers parked. it was a bad. then i went back and read the instructions and it was 50 meters, not yards. [laughter] [laughter] so yeah, i was exactly -- it was

way wrong. it wasn't just that i was wrong, it said instead of looking like this, i wasn't looking around anymore. so one of the points of the book is a false speedometer is worse than no speedometer at all. if the split hombre is faulty or just staring at a. nose pedometer we have no choice but to look around at other cars and see what's going on. a series example in the book, and this i think made at the end into answering your question is the valley of risk models wall street a doctor. user probability-based models that reported to collapse all the risk of the from into a single dollar figure. so 99 times out of 100 the firm would not lose more than just some over whatever period the model had been calculated over the next 24 hours. these models as precise as they were in many cases assumed that housing prices would not fall. so you get something -- the

underpinnings are rotten and it spits out something in meters that you're now looking at, i think it's quite dangerous. so the model that was about it added, medical outcomes, medical diagnostics are a tool, but if you get too far away from human judgment or you let the false precisions dove your sense of kind of what's really accurate, i think it's quite dangerous. i don't think we're going to replace humans anytime soon. that was a great question and one we will arrest with for a long time. >> can you address in the book, you say statistics for political chicanery, well, in two sentences i heard -- obama's campaign talked about how to use

the internet, very successfully. but then there's the kind of lambasting obama, we're 10 times more likely to been food stamps and that sort of excuse is taking away from wal-mart's policy, for example, spent this not an explicit chapter that addresses statistics in politics. instead, you, just about every other chapter, the way the book is laid out, his our probability chapter two, chapter three, chapter three is -- every one of those potential abuses is something that tends to show up in politics. so i use a lot of economic data. of course, the presidential campaign and others are always trying to spin economics in a way that makes him look good. so sometimes there are those

examples. there's a whole chapter on polling. and, of course, polling is the point of the realm what it comes to politics. there's a lot of discussion about if you've got bad data, it doesn't matter how big your sample is, the polls will show whatever you want to show. so it doesn't come up explicitly but it's in there quite often. this is one of the things that was a surprise to me. i was fortunate enough to meet frank, editor in chief of gallup came to dartmouth and spoke to my class to talk about the challenges of polling. what is the irony of polling right now is unlike a lot of the other technological things, is actually getting harder to have a sample that were reflects the actual published the it's cheaper than ever before to gather a lot of data. think about how you traditional do. he traditionally do a phone example but if you a good polling operation you just a random set of very coast.

well now, everyone of my students has a cell phone and they gave me the phone number and it is 213 and 405, and they all live in hanover, new hampshire. you've lost track of who lives where. you have rich people with call waiting who aren't going to answer calls from strange to get kind of lonely people who will, right? [laughter] and just now got a lot of young people without land lines at all when they may have to cell phone fundamentally land line. so now had to move with an assertion that land -- one landline for hustle. ironically they're going back in some cases to the methodology they started with which is in person. actually going back out door-to-door which is what they did in 1950. i think that you take away is be aware of large samples because they're really expensive. click here if you think we're doing a great job.

mom, click faster. spin the ones this is the jumping the line was from a commercial finance company. and a little boy baby says that's about the same chances of getting mauled by a polar bear and a regular the in the same day. >> my favorite story, there's a geico i think the story came out after the book was written to a guy who got struck by lightning while on his way to buy a lottery ticket. [laughter] and my thoughts were he got a low probability of any. just not the one he was looking for. so thank you very much for coming out here about statistics. [applause] >> tell us what you think about our program this weekend. you can tweet us at booktv, comment on our facebook blog or send us an e-mail. booktv, nonfiction books every weekend on c-span2.

>> you are watching tv on c-span2. here's our primetime lineup for tonight. >> that all happens tonight on c-span2's booktv. >> fred lucas is a white house correspondent and the author of a new book, "the right

frequency: the story of talk giants who shook up the political and media establishment." who are the giants you're talking about? >> of course you have today rush limbaugh, glenn beck. this goes all the way back to the very beginning. people like walter winchell, and it shows the trajectory of how talk radio begin, really was the beginning of great you. and i kind of you this as a history of the united states since 1920 through the lens of talk radio. >> going all the way back in what year are we talking about? we are actually talking about the '20s. hb is called the dean of commentators. the first to talk about in your. we go through walter winchell who was one point he was a hard-core new deal supporter, hard-core fdr supporter. he shifted during the 1950s, became a strong anti-communist, big john mccarthy supporter

and yet a radio show, syndicated newspaper column. he was a huge composing the 50 during that time. fast forward, go through the 1960s when it was a major force, the fcc was able to broadcast the industry. there's two chapters on his. one called challenge and harass when the johnson administration use -- to use to political enemies. one of the people interviewed here, i interviewed mark fowler, he was reagan's fcc chairman who helped dismantle the fairness doctrine. once you that revelation out of the way it allowed people like rush limbaugh to get a foothold in and that led to a real explosion of conservative and partisan ism on both sides. most on the conservative side. >> that was going to be my follow-up question in regards to 1922 today, the predominant

voices, was there a pattern? spent interesting enough, one of the things i write about in this book is how during the new deal you had the major metropolitan daily leaning to the right. they were very anti-fdr. most underrated and broadcast to, talk radio were very pro-new deal, very pro-fdr. we kind of have the opposite today. but yeah, it's come it became sort of a domain for the rights and the late '80s with rush limbaugh. he got his national show in 1980. and then it grew from there. during the clinton administration proud one of the best things happen for talk we're in a sense it gave lots of material and continued on to it actually grew during the bushes we go all out with folks that thought what will happen to talk