course, we didn't know how things would end up on november 6, 2012. but, um, i looked at how he developed his governing strategy and his electoral strategy, and it really culminated in november. so this is the back story to what happened in this presidential campaign. .. these books were written out of desperation. i was having a drink with a grad student, and we were reflecting

on how we struggled in some of our economics and stats classes. now naked economics" was wherein almost by accident in the sense i had been assigned a class to teen economics to journalists. i was unsuccessfully trying to sell a book on the gambling industry. never got that done. there's a good book to be written. i said to her, i got to teach this economics class to bunch of journalists. a textbook would be inappropriate. i can't find anything that would convey why they should care about this. there was a long pause, and she said, no, you're going to write it, and it's going to be called economics for poets, and i'm going to read it. so that's how naked economics was born, once that found a niche among people who had been scared away or bored to death by economics classes, we said -- we being debby north top and i --

let's go back and do statistics which is in any ways even more mathematically daunting. on the other hand, it's -- checks has always been economics, but statistics, just a couple of things have happened over the course of everybody's lifetime in this room, go back 15 years, when you gave someone your credit card and they did the slide thing and went into a bucket somewhere and that was filed somewhere so there was nobody had a digital record of what you bought. computer power was more expensive and more cumbersome. you fast forward now to the point where they'll scan the book, take your credit card data, and i'm going to read how alarming that it might become, not necessarily here. but the confluence of scanner data, the ability to digitalize that. and then cheap computer power, means we know more about people than we have ever known, and that data -- those data are

cheap and easy to manipulate are, to use, to ask study, to good and ill. i'm going to read four sections. the first is my uncomfortable relationship with math, and then talk about a probability event. it's older but gives you some sense of the power of statistics, how if you really know the underlying math you can use it for good. i'll fifth you one that i think is a wakeup in terms of, should we all be a little more aware of what's happening with data we are throwing out in every direction, and then i'm fog to -- going to finish a bunch of open questions that statistics will help inflame but help inform. one is, what is football going to look like in 15 yearses and researchers are looking at that question. i think there will be some

interesting developments over the near term for that. let me start with the introduction of the book. at it called, who i i hated calculus but love statistics. this is a totally true story. i have always had an uncomfortable relationship with math. i do not leak numbers for the sake of numbers. i am not impressed by fancy formulas that have no real world police. i particularlitive liked high school calculus for the simple reason no one bothered to tell me why i needed to learn it. what is the area? who cares? and now that my daughter is going through it, i still don't care. in fact one of the great moments of my life occur during my senior year of high school. at the end of the first semester of advance placement calculus. i was working on the final exam, less prepared for the course. i had already been accepted to college a few weeks earlier so as i stared at the final exam

questions they looked completely up familiar. i don't mean i was having trouble answering them. i didn't understand what i was being asked. to paraphrase don rumsfeld, i usually know when i didn't know. this exam looked even more greek than usual. i flipped through the pages of the exam and then more or less surrendered. i walk the front of the classroom where mychal includes teacher, whom we'll call carol smith -- in the original draft it was, whom we'll call carol miller, because that her name -- the publisher said we're not going to do that. carol smith was proctoring the exam. i said, mrs. smith, i do not recognize the stuff on this exam. we had a contentious relationship. so suffice to say mrs. smith did not like me a whole lot more than i liked her. yes, i can now admit i sometime use my limited powers as student

association president to schedule all school i semi just so her class will be cancelled. mess -- yes, my friends and i did have flowers delivered mrs. smith, from a secret admirer so we could laugh in the back to see what happened. and i didn't study any math at all when i got to college. so when i walked up to mrs. smith and said the material did not look official she was, well, unsympathetic. charles, she said loudly, ostensibly to me but was facing the rows of desk to make sure the spire class could hear. if you had studied the material would look a lot more familiar. this was a compelling point. so, i slunk back to my desk. after a few minutes, brian -- we had these discussions -- a far better calculus student than i, walked to the front of the room

and hisserred a few things to mrs. smith. she looked back and then a truly extraordinary thing happened. class, i need your attention, mrs. smith announced. it appears i've given you the second semester exam by mistake. we were far enough into the test period that the whole exam had to be aborted and rescheduled. i cannot fully describe my euphoria. i would go on in life to marry a wonderful woman, three healthy children, i've published books and traveled. still, the day mychal includes teacher gotmer comeupans was a top five life moment. now, by way of disclosure, the fact that it nearly failed the makeup examine did not diminish from this wonderful experience. so that's where i come from when it comes to quantitative exercises.

there's a sad footnote to this, which is that by the time i got to graduate school i was feeling a little more confidence. i had gone through math camp and was thinking i was getting my math feet underneath me. and the woodrow wilson school could offer most of the econ classes with or without calculus, and you can decide and by the second semester i was thinking, i'm ready for calculus. so i took it and got beaten up, and unfortunately, the class without calculus was taught by ben bernanke. so my hubris meant i did not study macroeconomics with the current fed chair, which is unfortunate. so, that is where i come from. i have in life -- because i do public policy and things like that -- come to appreciate the

enormous power of these tools and the math, but it's about, can you tell me why i need to know this? so what i'm going to read now is a section from chapter 5, which is basic probability. the subtitle its, do not buy the stepped warranty on your $99 printer. that you can probably figure out. instead i'm going to tell you a story some of you may remember. this may seem a little old but i hope you recall this. this is a story that begins the chapter, 1981, the joseph suites brewing company spent 1.7ing me fa dollar for what appeared to be a shockingly bold and risky campaign for shlitz. how many remember that beer? the campaign didn't really work. at it good for this time of year. at halftime of the super bowl, in front of 100 million people around the world, the company

broadcast a live taste test, pitting schlitz against a key competitor, michelob. bolder yet, the company did notice pick random beer drinkers, it picked 100 michelob drinkers. this was the culmination of a campaign that had been throughout the playoffs. there were five live taste tests, which which had 100 consumerrers of competing brands conduct a blind tase test between their favorite beer and schlitz. it was promoted --, watch live during the afc playoff. how many remember this? the marketing message was clear. even beer drinkers who think they like another brand will prefer schlitz in a blind taste test. for the s. bowl spot, schlitz hired a form nfl referee to over

see the task. given the richesy nature of posing test tests on live tv, one can assume that schletz produced a spectacularly good bed. that was not the case. schlitz only needed a mediocre beer and statistics. most beers in the schlitz category tasted about the same. you should remember in the '80s all the beer was bad. ironically, that's exactly the fact this campaign exploited. assume the typical beer drinker off the street cannot tell schlitz from budweiser, from michelob and miller in that case a blind test between any of the beers is a coin flip.

on average half the taste testers will pick schlitz and half the beer it's challenging. this fact alone would not make a particularly effective advertising campaign. quote, crowd can't tell the difference so you night as well drink schletz. and sclhitz would not want to do this taste test among its own loyal customers. roughly half of whom would pick the competing beer. it looks bad when the beer drinker supposedly most committed to your brand chooses a competitor as a blind taste test, which is what schlitz was trying to do to competitors. the genius of the campaign was conducting the test test exclusively among beer drinkers who stated they preferred another beer. if the blind taste testes just a coin flip, half of the budweiser, miller, michelob drinkers, will pick schlitz.

half of all bud drinkers like schlitz better, and what looks particularly got al halftime, with a former nfl referee in uniform, conducting the taste test. still, it's live township. even if the statisticians at schlitz determined that the typical michelob drinker will pick schlitz half the time, what if the michelob drivenners taking the test turn out to be quirky? the bottom line test is equivalent to a coin toss. what if the tasters chose michelob by chance? if we lined up the same guys and asked them to flip a coin, it's entirely possible they would flip 85 or 90 heads. that's the kind of bad luck in a taste test would be a disaster for the brand, not to mention it would waste $1.7 million for the live television coverage. statistics to the rescue. if there was some kind of

statistic superhero, i have in mind sick sigma man. this is when he or should would have swooped into the schlitz corporate headquarters and give the denails of the a trial. the key of the experiment is we have a fixed number of trials, 100 taste tester-s -- each with two possible you cans, schlitz or michelob and the probability of success is the same in each trial. i'm assume egg the probable of picking one beer over the other is 50% and i'm defining success as the tester picking schlitz. we also assume the trials are independent. one decision has no impact on any other taste tester. with only this information, a statistical superhero can calculate the probability of all the different outcomes for the

hundred trials. those of us who are not statistical superheroes can use a computer to do the same thing. the chances of all 100 taste testers picking michelob were -- anyone want to take a guess? the number is too a big. it's one with -- about 27 numbers after it. you're more likely to get hit by an asteroid while watching the trial. more important, the same basic calculations can give us a cumulative probable for a range of outcomes, such as the chances that fewer than 40 of the taste testers picked schlitz. these numbers clearly assuaged the fears of the marketing team. let's assume they would have been messes with if 40 of the 100 taste testers picked

schlitz. remember all the many taking the test had professed to be michelob drinkers, and an you can at least that good was highly likely if the beer test really is like the flip of a coin, then basic probability tells us that there was a 98% chance that at least 40 of the taste testers would peck schlitz and an 86% chance that 45 of the taste testeres would. in theirly this was not a risky gambit. anyone want to guess what happened at halftime? at halftime of the 1981 super bowl, exactly 50% of the maybe lobe --ers chose schlitz in the bottom line taste test. two important lessons-probability is a remarkably powerful tool and, two, many leading beers in the 1980s were indistinguishable from one another. this chapter will focus on the

first lesson. all right. so, that should make you encouraged about the power of probability and the statistics in general. so now i'm going to make you scared. so this is actually at the end of the book and it's a question -- one of the questions -- who gets to know what about you? last summer we hired a new baby-sitter. when she arrived i began to explain our family brown. i'm a professor, my wife is a teacher. she cut me off and said, i know. i googled you. i was simultaneously relieved that i did not have to finish my spiel, and mildly alarmed by how much of my life could be cobbled together from a short internet search. our capacity to gather and analyze huge quantities of data, things i referred to earlier, the major of digital computing with the internet is unique in

human history. we're going to need some new rules for this new era, let's put the power of data in perspective with just one exam from the retailer target. this is a story that was told in "the new york times" magazine. i'm spiesing it here. like most companies, target strives to increase profits by understanding its customer. which is a very good thing. to do that the company fires statistickicses to do the -- figuring out who buys what and why. nothing about this is inherently bad. it mean when you go to target they're likely to be carrying things you actually want to buy. let go down for a moment to one example of the kinds of things the statisticians working in the windowless basement at corporate headquarters can figure out. i don't mow if it was

windowless, but all the statistician i know were pale and working underground. target learned pregnancy is a particularly important time in terms of developing shopping pattern. pregnant develop a retail relationship that can last for decades. as a result, target wants to identify pregnant women, particularly those in their second trimester, and get them into the stores more often. a writer for "the new york times" magazine followed the predictive an lit -- analytic team. target has a baby shower registry in which pregnant women register for baby gifts in advance of the birth of their children. these women are already target shoppers and have effectively told the store not only when they're pregnant but when they're due. here's the statistical twist.

tarring figured out that other women who demonstrate the same shopping patterns are probably pregnant, too. for example, pregnant women often switch to unscented lotions, begin to buy vitamin sub policemens. start buying extra big bags bagf cotton balls. who knew. the target predictive analytic gu ruiz identified 25 products that together made possible what they describe as a pregnancy prediction score. the whole point of the analysis was to send pregnant women, pregnancy relate coupons in hope of hooking them as long-term target shoppers. how good was the model? a magazine reported a story about man from minneapolis who walked into a target store and demanded to see the manager. the man was irate because his high school daughter was being

bombarded with pregnancy related coupons from target. quote, she is still in high school and you're sending her coupons for baby clothes and crib? are you trying to encourage thor get pregnant? the store manager apologized profusely. even called the father self days later to apologize again. only this time the man was less irate. it was his turn to apologetic. he said, it turns out there's been in activities in my house i haven't been completely aware of. the father said. she is due in august. the target statistickic. >> s figure out his daughter was pregnant before he did. this is not even the creepiest part. that is their business and also not their business. it can feel more than a little intrusive. for that reason, some companies

now mask how much they know about you. for example, if you are pregnant woman, and your second trimester, you may get coupons in the mail for cribs and diapers, along with a discount on a riding lawnmower and a coupon for free bowling socks with the purchase of any pair of bowling shoes. to you it just seems fortuitous the pregnancy related coupons came in the mail along with the other jung. in fact the company knows you don't bowl, you don't cut your own law. it's merely covering the track so is that what it knows doesn't seem so spooky. all right. let me finish with another question. i actually like the way i'm finishing. are there are good things about statistics, scary things about statistics, and then there are places where we're watching unfold right now in real-time. this is some of the most interesting stuff. one of the questions at the end

of the book is how can we identify and reward good teachers and schools? my wife is a public school math teacher. so she has has been involved in this realm. we need good schools and we need good teachers in order to have good schools. it follows logically we ought to reward good teachers and good schools and firing bad teachers and closing bad schools. how do we do that? test scores give us an objective measure of student performance, yet we know that some students will do much better on a standardized test for other reasons that have nothing to do with what is going on inside the classroom or the school. the seemingly simple solution is to evaluate school and teachers on the basis of the progress that their students make over some period of time. what did students know when they started when a certain classroom, what did they know a year later. the difference is the value added in that classroom. we can even use statistics to

get a more refined sense of this value, by taking into account the demographic characteristicked of the students such as race, income, performance on other tests, which can be a measure of baseline aptitude. if the teacher makes significant gain with student whose typically struggled in the pass, then they can be deemed to be highly effective. voila. we can now analyze teacher quality and the good schools are the ones full of effective teachers. how do these statistical evaluations work in peninsula? in 2012 new york city published ratings of all 18,000 public school teachers on the basis of value add ed assessment, los angeles times published similar data for los angeles teachers in 2010 in both new york and los angeles, the reaction has been loud and mixed.

arnie duncan, u.s. sackett of education, has generally been supportive of these value added numbers, after the data was published, the obama administration has provided financial incentives for states to provide value add educators for paying and promoting teachers. proopinion anyones point out they're a huge potential improvement over systems in which all teaches are paid in a system -- many experts warn these assessments have large margins of error and can deliver misleading results. the union representing the new york city teachers spent more than $100,000 on a up in advertising campaign, built around the headline, this is no way to rate a teacher. opponents argue the value added

assessment creates a false impression used by parents and public officials who do not understand the limitations of of this. everybody is right up to a point. a man works extensively with value added data for teachers, warns that these are inharen day noisy. the results are based on a single test, taken on a single day, by a single group of students. all kinds of factors can lead to random fluctuations, anything from a particularly difficult broken student to a broken air conditioning unit clanking away in the classroom while testing. the correlation of performance from year to year for a single teacher is only about .35. if you're really good, you should be year after year and it would argue by be higher than that. on the other hand the correlation in year-to-year performance for major league

baseball players is also around .35 as measured by batting average for hitters and e.r.a. for pitchers. the data are just one tool in the process for evaluating teacher performance. the data get less noisy when authorities have more years of data for particular students and particular classrooms and students. we can tell more about an athlete over many seasons. in the case of new york city teacher rating, principals have been prepped on the appropriate use of the value added data and their inherent limitations. the public did not get that briefing. as a result the teacher assessments viewed as a definitive guide to, quote, the good and bad teachers. we like rankings. think of u.s. news and world report. they're in for a drubbing in several chapters -- even when the data do not support such precision. data offers a final warning of a different sort. this is a really interesting

study. we better be certain of the outcomes we're measuring, such as the results of a given standardized test. we have to make sure they track what we truly care about. so everything that this point i assume the text scores are an end we some care about. unique data from the air core academy suggests no, surprisingly, the test scores that glimmer now may not be gold in the future. the air force academy -- there's a lot in the book about the importance of get going data and how hard it is to get data to enable you to study certain things. the academy here is unique in that it randomly assigns its cadets to different sections of standardized core courses such as introductory calculus. this randomization eliminates any potential selection effect when comparing the effectiveness of professors. you don't get to pick your professor, don't even get to pick your class.

so over time we can assume that all professors get students with various aptitudes. unlike most universities where student office different abilities can select in or out of classes. the air force academy uses the same exams in every particular course. scott carroll and james west, professors as uc davis and the air toast academy, exploited this elegant arrangement to answer one of the most important questions in higher education. which professors are most effective? you're all getting students that look leak -- like each other, same tests. who is doing a good job? the professors with less experience and fewer degrees from fancy universities. these professors have students who typically do better on the standardized exams for the introductory courses and get better student evaluations for their courses. clearly these young, motivated ininstructors are more committed to their teaching than the old

crusty guys with ph.ds from places like harvard. the old guys must be using the same yellowing notes they used in 1978. probably think power point is an energy drink, except they don't know what an energy drink is, either. obviously the data tell us we should fire these old college jerries or let them retire gracefully. but hold on. don't fire anybody yet. the air force academy study had another relevant finding about student performance over a longer horizon. carolyn west found math and science, the students witch more experience and highly credential ed instructors in the introductory course does better in they mandatory follow-on courses than students in the introductory courses. one logical interpretation is that the less experienced instructors are more likely to teach to the test. in the introductory courses.

this produces impressive exam scores and happy stunts when it comes to fill ought instructor evaluation. meanwhile, the old crusty professor,s, whom we nearly fired, focus less on the exam than on the concept which is matter most in the follow yawn courses and life after the air force academy. we need to evaluation teachers and professors, we just have to do it wrong. in the long-term policy challenge is to develop a system that rewards a teachers rival uaddded. this is a work in progress but it's not going away. so i'll stop there and be happy to answer questiones about this book. or neglect else that might be on your mine. thank you. [applause] >> i want to thank you for writing this book and also for writing naked economics. i hope this book sells as well

as the only other statistics best seller in history, which is called how to lie with statistics. >> well, wait. this is a sequel to how to lie with statistics in the sense that when i was with my publisher about a book, and he reached behind him and said update this. that book was made in the 1950s and sold millions of copies. >> at least 8,000 lobbyists will buy a copy. >> i'm interested in getting the policymakers to take away the right thing. i worked in the clinton white house, and one of the first things they did when he walked in the door was handle memos to the president by charles schultz, which was a nice explanation of economics. we didn't have a book like this to explain statistics.

i wonder if there was a book that we could buy 535 copies -- >> if you buy it here you get a 20% discount. in addition to that, what should we do to get policymakers a little more aware of how misleading most of the numbers they get every day are? and particularly the -- the poll results they rely on so much. >> i'm not sure everybody cares. some of it is deliberate. i think the best thing we can do -- when we teach these things we have 0 move away from the mechanics. when i was taught statistics, it was here's how you calculate this test. you don't need to. if you've got a personal computer on your desk made after 1995, it can too that for you. what you need to know is, what you're actually doing. it's like a hand-held grenade launcher. it's not that hard to use.

very hard to use appropriately, and we would be better off telling you where not to point it than instructing how it works. so i think the most important thing we can do is to step back, talk about what is likely to make statistics go off the rails, where you can get really bad data, and you plug it into the most elegant formulas, it will spit out the most wrong of conclusions. so i think there has to be a much spin fewtive foundation of that's going on and more explanation where these formulas come from. i had to go back and relearn the things i had been taught, and a lot of times it was, oh, wow, that's it? i understand that. now i understand. but the first time around, not so much. so i think that, again in both economics and statistics, got to be a more intuitive approach for why this works and how it can be egregiously wrong. and i think sending lots of books to lots of people would be a good start.

gates foundations maybe? >> very glad to see -- >> one for the office and one for home. you don't want to be carrying it back and forth, you're thanks again. >> thank you for your thoughts. i'd like to talk about -- heavily studied, being promoted, reduce your risk of breast cancer by half. conversation i had with a doctor, who is a specialist, and looking over the statistics i asked him, isn't it more accurate to say that it doesn't reduce your risk by half, but it reduces the risk for half the women who take it. and how does an individual woman know whether or not it's going to be -- is she going to be in the half that works or the half that doesn't work? no answer. you're right. so, what do you have to -- >> i had no knowledge of those trials. there is a fair bit of talk

about clinical trials for pharmaceutical products, though in part because you end up with some -- that's one place where statistics can steer us wrong. 'll give you a very basic point that applied to pharmaceutical products, and that is simply where you no the mean or the median. so, the problem with the mean -- this gets to explain -- it gets pulled by outliars. you look at mean household income in the country, looks like it's going up in a healthy way. that can be explained by explosive income growth at the top and people who we would consider to be the middle aren't get anything wealthier at awesome that's one drawback of the mean. so you might say, we should just never use the mean, always use the median, the middle point in the distribution, and the way i explain it, iryou got ten people sitting on a bar stool and each make $40,000 a year. and bill gates walks in and sits down on the 11th bar stool, the average income in the bar

goes through the roof. the median does not move at and all in that case the median is the right one because those people didn't get richer. when you lochte pharmaceutical outcomes, there's a guy who wrote a book called the median is not the method. when you look at median, the problem is if there's a drug that is successful for 40% of the people who take it, and they live for another 25 years, but 60% of the people don't live at all, the mean life expectancy is going to look very low, and the median is the one we should look at. so this is one where descriptive statistics and people are affected, shooing row right statistic matters. the other place that clinical trials come up is, there appears to be a lot of publication bias in the sense that nobody likes -- if i went out and did a study that said, watching television for eight hours a day does not make you healthier, and

i shopped it around, am i going to get it published? they would say, who thought it nose the problem is, all research is based on probability. so if 1200 of do this, few of us are going to find just by chance that these folks who sat on the couch and watched tv happened to get healthier. so when they shopped this study, which is really an outliar. 95 of us found that watching tv doesn't do any good but a few of us found that watching tv is correlated with a healthy outcome. people say you can watch tv and fix heart disease? we have to publish that. so it's a serious problem the medical community that if you don't look at all the studies being conducted, the ones that show, hey, this actually works, who wants to read a study that something doesn't help cancer. they're trying to do it better. journals require that you

register statistics at the beginning. but it's a study of the studies toned suggest we actually may not be getting a terribly accurate view of which of these things are effective and which are not. it's the best i can could to answer your question without specific knowledge of that drug. i will say you have to be very cautious with the data. your question was a good one. >> i notice you're a prefer at dartmouth college. that was my first teaching job. >> john leslie young research department. >> and laurie snell was my mentor there. great place to start your academic career. nye nay my thesis was written in possibility and i have done statistics also, and i just wanted to make a few comments before coming to my question. you spend lot of time in your

book about the simple -- i was teaching a business stats course which a colleague of mine described cynically as follows: i asked him what is the difference between boston stats and stats. he said, well, business stats course is one in which every observation in the data set has a dollar sign in front of it. anyway, i spent time -- you can't prove the theom but you can give an intuitive description what the therum says. here's my reward after doing that. student evaluation at the end of the semester. professor, you talk too much. you explain too much. cut out the small tuck. -- small talk. just give us the formulas.

>> so i mean, i tried to do what you would like me to do, and that's the world i get. the other thing -- here's another story. i'll get to my question shortly. i mean, i feel like the ghost of hamlet's father. i could a tale up fold, freeze thy young blood. a professor, a student wrote, makes is awfully touch for the average student to get an a. i think that's one of my students now. >> i hope things are better at dartmouth. >> a great pleasure to work there. but anyway, i'm glad you talked about the value added modeling. call your attention to -- it's a area i'm interested in. teachers i know are getting

fired right and left because of this abuse of statistics. for example, here in washington, there was fifth grade teacher who was fired because of the value added model. her students did not perform as well as predicted. okay? it turns out that the scores of the studentness the previous grate level -- i thick fourth grade -- benefited from cheating. that is, the staff just erased the incorrect answers and put in the correct answers. so, students came in with a higher predicted value than they deserved. so, she was very worried because she saw their scores in the fourth grade tests, and they didn't correlate with what she was seeing in the classroom. so there's this vulnerability to fraud and cheating and so on.

and anyway, i talked to the -- the executive director of the asa, american statistical association, and he said there's a special committee of the asa that is going to -- they're analyzing the value-added methodology. i myself just wrote a very simple paper -- i have not submit ford publication -- which illustrates very simple model that complains why a teacher can be rated very high one year and rated very low the next year. because of this natural variation due to chance. >> oh, yeah. that's a really terrific point. it is going to be with us and it is noisy. that's why i can like the description. a couple observations around your questions. one is one of the fun parts of the book is using probability to catch cheating on standardized

tested. there are actually private firms now and the methodology, if you the answer sheets through a scantron, there are things that flag cheating. so, for example, a high proportion of erasures that go from wrong to right. you expect that to be -- it tells the story that the huge cheating scandal where -- i can't remember the numbers but the odds are -- the probabilities of observing the pattern of erasures they observed we are -- someone describe as the probability of having -- going filling the astrodome with peel who showed up and happened to be over seven feet tall. very unlikely. that doesn't prove anything so they drilled down, and to your point, what they discovered was that teach experts administrators were having pizza parties on the weekend where they brought the answer sheets and they all changed the answers at the same time. we're going to have to do this, might as well come together and do it together. that's obviously the most

pernicious, egregious version of cheating. my wife is a teacher and there were high-stakes testing and they didn't quite cheat but it was very close. the rule in illinois you don't have to take anything down in your room during the testing. so, my wife went to a teacher development day, which was focused exclusively on maximizing the surface area of your room, including the inside of the door, so that your students will be able to look somewhere, ceiling, anywhere, and see the answer they may need. like that's what they stud yesterday, which is probably not intent. and last, my kid, who were also going to public school, came home and said we love the isat. we get chocolate at the beginning of the test. they couldn't give them a latte, but they could. a them up on caffeine.

my gosh. >> my question, have you ever heard of campbell's law? >> no. >> pardon me. i wrote it down. the greater the social and economic consequences associated with a statistic, such as test scores, the more likely it is that the statistic itself will become corrupted. and that its use will corrupt the social process it was intended to monitor. >> this is great. i'm writing a short piece for the wall street journal to summarize some of the key findings, and one headline is smart imaginers use statistics to evaluate employee; smart employees will figure out how to manipulate the numbers. >> in new york state they simply decided they would provide data on mortality rates for angioplasty for cardiologists, and my inner economist says

information is always good. turns out they -- "the new york times" or somebody followed up and a high proportion of -- something like 70-80% of cardiologists said they deliberately changed their behavior because of the evaluation the they way changed their behavior was not kill fewer people. boy, killed a lot yesterday, today i'm not going drink, right? it was -- [laughter] >> it was on the margin they decided not to take the most seriously ill patients. so the net effect -- i hadn't heard the law but that is one of the great takeaways, the people who were sickest, most in need were the least likely to get it because of that evaluation. but i think it's true. >> you can prove that the most dangerous place in the world is a hospital. >> oh, yeah. >> this is good. thank you. campbell will be in the next edition. it's name for an important concept. thank you. >> sorry.

feel silly for asking this but i'll do it anyway. so there's space in some quarters that data mining and machine learning, do away with scientific method and kind of machine-learn your way to a cure for cancer. and some people are -- that's on the one side of the house. the other side is we can't take the people out of the statistics and we make sure that someone kind of starting with hypothesis and testing it from a theory-based approach. and i've seen reasonable arguments on both sides of that. what do you think? >> i think it's not an either/or. we were talking about actually using high-powered commuters for medical diagnostics, and i think there's a lot of potential value there particularly in a number of medical errors. my caution is -- one of my

opinionness the book is distinction between precision and accuracy, and any kind of statistical model can give you a false sense of precision, and in the book, my wife gave me one of those golf range finders for christmas, and i grew up playing golf and you look at the pine trees and look at the pin and say, it's about 150 yards. you get the range finder and shoot it at the pin and you're 148.7. this is amazing. and my golf kept getting worse, and i was hitting trees and i was hitting a sand trap. at one point i hit the municipal parking lot where all the police officers park. it was bad. and then i went back and read the strucks and it was seat to meters, not yards. [laughter] >> so, yeah, i was exactly -- it was way wrong. and it wasn't just i was wrong. it's that in looking like this, wasn't looking around anymore. so my point in the book is, a

faulty speedometer is worse than know speedometer at all. but if the speedometer is faulty, you're staring at it itif you have no speedometer you have no choice but to look around. the serious example in the book and this may in the end answering your question -- is the value at-risk model that wall street adopted. probability based model that purported to collapse the risk of the firm into a single dollar figure. so 99 times out of 100, the firm would not lose more than this sum over whatever period the model had been laked organization the next 24 hours or in the week. these mod ol's, as precise as they were in many case assumed that housing prices would not fall. so you get something -- lots of sophisticated people building fancy models and the underpinnings are rotten and spits out something in meter

that you're looking that a narrow way. it's dangerous the models with the value-added or medical outcomes or medical diagnostics, are a tool, but if you get too far away from human judgment or you let the false precision dull your sense of what is accurate, it's quite dangerous. so i don't think we're going replace humans anytime soon. no hal in space odd di. -- space odyssey. it's a great question. >> can you address in the book you use statistics for political chicanery -- well in two senses i heard of a man who managed obama's campaign, how they used the internet and statistical calculations, verse successful. then there's the kind of bomb bass bombastic, under obama

you're ten times more likely to be on food stamps and that skews the thinking away from wal-mart's policies, for example, to the fact that -- >> right. right. an explicit chapter that addresses the use of statistics in politics. would be a good chapter. instead, just about every other chapter -- the way the book is laid out, here's the power of probable, chapter two, three, the abuses of probability. chapter three is regression, analysis. and every one of those potential abuse is something that tends to show up in politics. iy a lot of economic data, and of course the presidential campaign and others are always trying to spin economic indicators in a way that makes them look good or the other goo look bad. so sometimes there are those examples. there's a whole chapter on polling, and polling is the coin of the realm when it comes to politics and there's a lot of discussion about it. you have bad data, doesn't

matter how big your sample is, or if you deliberately self-select your data, the poll will show whatever you want it to show. at one point i will add on the polling -- this is one of the things that was a surprise to me. frank new fortport is the editor of gallup, and he came and talked and one irony of polling right now is unlike a lot of the other technological advances, it's getting harder to conduct a sample that actually reflects the general population, and the reason is, it's cheaper than ever before to gather a lot of data. but think about how you do it? you do it with a phone sample. if you good polling operation you would use a random set of area codes. that's how your figure out geographical. now every one of my students has a cell phone and they give you the phone number and it's 213, and it's 405, and they live in

handover, new hampshire so you lost track of who actually lives there you have rich people with call waiting who aren't going to answer calls from strangers. you have lonely people who will. right? and you've now got a lot of young people without land lines at all or they may have two cell phones or a land line and a cell phone. so now we have to move away from the assumption that one land line per house hold. so he talk about how they're actually going back some cases to the methodology they started with, which is in person, going back out door-to-do, which is how you've did it in the 1950s. so i think the general takeaway is beware of large samples because they're really expensive and unless somebody is dropping a lot of may -- click here if you think we're doing a great job. mom, come here, click faster. >> the topic of your book that jumped into mind is the

commercial for a finance company and the baby who is the spokesperson says, that's about the same chance as getting mauled by a polar bear and a regular bear on the same day. >> there's a guy -- i think the story came out after the book was written. a guy got struckly lightning on his way to buy a lottery ticket. my thought was he got the low probability advantage. just wasn't the one he was looking for. thank you very much for coming out here about statistics. [applause] >> fred luke case the author of a new book, the right frequency, the store, the radio giants who shook up the establishment. who are the radio giants? >> of course you have today, rush limbaugh, glen beck, but

this book goes back the very beginning, walter winchel and shows the trajectory of how talk radio began, what the beginning of radio, and i view this as a history of the united states since 1920, through the lens of talk radio. >> host: so, going back, what year are we talking about? >> guest: we're actually talking about the 20s and -- called the dean. commentators, the first guy i talked about in here. walter winch winchell, at one point he was a hard-core, new-deal supporter, hard-core fdr supporter. he shifted, became a strong anticommunist and he had a radio show, newspaper column. a human imposing media figure. go through the 1960s, when the

famous -- a major force that the fts was able to regulate the broadcasting industry, and two chapters on this. one called challenge and harass, the johnson targeted political enemies. one of the people i interview in here, mark faller. reagan's ftc chairman who helped dismantle the fairness doctrine. once you had that regulation out of the way it i allowed people like rush limbaugh to get a foothold in and that led to an explosion of consecutive and partisan voices on both sides, mostly on the conservative side. >> host: that was my followup question in regards to from 1920 to today. the predominant-for- -- was there a pattern, left, right, left? >> guest: well, interestingly enough of. one of the things i write about is during the new deal you had

the major metropolitan dailies leaping the right. very antifdr, and most of the radio and broadcasting voices were very pro new deal, very pro fdr. kind of have the opposite today. and -- it became sort of a domain for the right, and the late 80s, with rush limbaugh, got his national show in 1988, and then drew from there during the clinton administration, that is one of the best things that happened for talk radio in the sense that it gave lots of material and continued on, and actually grew as well during the bush years. a lot of folks who thought what's going to have to talk radio when they don't have bill clinton to kick reason anymore? actually thrived during that period. you have glen beck, sean hannity , major national voice. >> host: what does it say to talk radio now in comparison to

cable news outlets? we have seen people from talk radio move move over to cable news outlets. sean hannity, glen beginning. sean hannity is one of the only guys to have maintained both a presence in tv and radio on a long-term basis. glen beck was in cable news for a while. has his own cable network now internet-wise, but didn't last long on fox news. or hlm. rush limbaugh had a beer four-year stint on tv but -- had a brief four-year stint on tv but both over those guys, they're domain was radio and that's where they got the big ratings from. >> talking with fred lucas, the story of the talk radio giants who took up the political media establishment. >> thanks very much.