[inaudible conversations] >> so a short break now, about ten minutes, as you heard, in this conference on problems facing minority students seeking a college education. we will return to live coverage

when it resumes shortly, as i said, about ten minutes from now. while we wait, more on education now as the new school year is just getting underway for many school systems around the nation. we look at challenges facing public schools in a discussion on this morning's "washington journal." >> host: guest at our desk is the director of the national school boards association. as we dive into the topic of local school boards and today's education debates, first, tell us about your group and its mission. >> guest: well, the national school boards association really represents our state associations of school boards, and they're the ones that have the membership of the local school boards. local school boards are comprised typically of folks out of the community who have an in public education. so while they're usually elected, some are appointed. typically elected, most frequently not paid, so they do this as a civic duty. and because they have a concern about good public schools in

their communities. >> host: and how many people are we talking about? how many local school boards across the country about how many are there? >> guest: we have about 15,000 school districts, and just about every district has a school board. in fact, i've always found that interesting. every state in its own way created something called a school board. it wasn't a federal mandate, it wasn't a requirement, but i think they all recognize the value of having people in the community leading the local education effort. >> host: and how would you describe your association's relationship with the education department and particularly secretary arne duncan? >> guest: well, we've worked closely with the secretary and with d.. we obviously -- with the department. we obviously have some disagreements on policy issues, but the relationship's been good, the communication's been good. >> host: does he understand the importance from your view of the role of local school boards in these overarching debates on the landscape of education in the united states? >> guest: i like to remind him. i think it's important. look, i mean, there's, i think,

some tension, right, between the federal role in education and the local school district role. i think there's always been some tension around that. i think there's probably actually a little more today than there has been in the past. we think it's important, and i think the public thinks it's important, that schools generally are led at the community level. in fact, the recent gallup poll showed that nearly six out of ten people said they think school boards should play the lead role in determining really what's taught in public schools. only 15% said that about the federal government. so i think we have to get to a place where we're really having a serious conversation about where those decisions are made and how people really feel as though the schools in their communities are their schools, that they're not an outpost of some other federal or state agency, but they really feel an ownership. >> host: and some of the tension that you're talking about reflected in the debate over the local school board governance and flexibility act for those

unfamiliar with that, explain what that is. >> guest: it's legislation that's been introduced actually mow in both the house and the senate -- now in both the house and the senate, it really is designed to try to limit the federal role. mostly what we're seeing and our biggest concern that really led to this legislation is that more and more the decision making at the federal level is coming out of the department and not even through the normal regulatory process. we're seeing a lot of mandates and directives coming out of the department that are not necessarily based on the law. so this is, again, an effort to try to restore some balance. i think it's attracted some bipartisan support, and it's generated, i think, a really healthy conversation. >> host: and who's led that support on capitol hill? who's the lead sponsor, and where is that legislation in the process? >> guest: well, representative aaron shock from illinois introduced the house bill. senator jim inhofe from oklahoma introduced the senate bill. and, again, they have co-sponsors in both chambers. i think, john, the important

piece about this legislation is that it really tries to frame the issue about how do we make good decisions about public education and engage all the parties including local school boards in that process. what we've seen i think over the last number of years -- and, by the way, this isn't just directed at the current administration -- we've seen over a number of administrations over a number of years a growing federal role that's dictating more and more about what's happening in our local schools. and i think, again, as we saw in the gallup poll results, the public is getting kind of tired of that. and i think we have to figure out a better way to have that conversation than we've had recently. >> host: and if you want to join in on this conversation with the national school boards association, you can give us a call. our phone lines are open now. parents can call in at 202-585-3880. teachers can call in at 202-585-3881, and a special line for school board members, that number 202-585-3882.

we'd especially like to hear from you in this segment as we're talking with the executive director of the national school board association. 202-585-3882 if you're a member of a local school board. all others at 202-585-3883. before we believe the governance and flexibility act, want to ask you one more question on it. would this legislation give school boards a de facto veto over rules coming out of the education department? talk about some of the concern that to points of this legislation -- opponents of this legislation have brought up. >> it would not do that, just to be clear. what we're really talking about is a process. so if the federal government, the federal department of education wishes to impose some new regulations, first of all, they need to be clearly grounded in the law. and secondly, they need to go through a process that even allows local school board members, superintendents, parents, teachers to be able to comment on them before they're actually enacted into law.

that's not been happening, and we're seeing more and more directives coming out of the federal department of education that have not gone through that process. and i think that's the essence of the legislation, but it's certainly not a veto. >> host: questions for you already coming in over twitter. wild and wonderful asks: does the average local school board have the resources to collect and analyze global data to determine what u.s. kids need to do to keep up in the world? >> guest: well, the answer to that is probably not. i mean, most school districts have very limited resources which is why i would say there is a legitimate federal role in education to do that kind of research, to share that information with local school officials. we think that's important. our concern is the overreach issue. but i think we would be the first to argue that there needs to be a balance, right, between what the federal government is doing, what local government, school districts and state governments. >> host: some concern by many people on the education debate over an overreach when it comes

to common core. can you talk about your group's stands on the common core standards? >> guest: our association supports strong academic standards for public schools, career and ready standards. common core, certainly, is one of those. there are other options that different states have been developing. we think that's important. common core began as a, an effort by the governors really to come up with a joint set of standards that they could all subscribe to. we thought that was an important initiative. as we've seen again in the latest gallup poll, there's a lot of public concern about common core. i think there's a perception, certainly, that the standards are being directed by washington, and i think that's hurting a hot. the reality is that standards, i think, are, can be very helpful to school districts in designing their curriculum and how they're delivered. but we need time to do that. we need time for teachers to be trained, to put the curriculum into place to make sure this is

dope right. and i think we're not quite there yet. but the notion of having high standards, we think, is critically important. >> host: we're talking with the executive director of the national school boards association. have a specific line in this segment if you're a member of a school board, we'd love to hear from you, 202-558-3882, otherwise lines for participants, teachers and all others -- parents, teachers and all others, those numbers are on your screen, and we will be getting to your calls now. charles is in woodbridge, virginia. good morning, you're on with tom against l. >> caller: good morning. i don't want even know where to start, but i have just two quick points, and i don't care what you're teaching, the new programs, whatever, the bottom line is that it all starts at the home. school environments have no, you know, there are a lot of kids out there that respect teachers and the principal, but there are a lot of kids that don't. the reason why is because schools have no power anymore. nobody cares about a student who

talks down a teacher right in front of everybody else. when you lose that respect, kids respect going to listen to you. when i -- aren't going to listen to you. the bottom line is we respected our teachers, and we certainly had the respect of our principal because he had the paddle. and so there's a level of discipline there that is not brought from the home. it's the home's fault, by and large, not the school's. it was so bad in high school that my sons went to with the whole dressing in the hip-hop and all that, that we took them out and sent them to a military high school. my son went in with a 1.9 and two years later graduated 3.85. want to give thomas a chance to respond. >> guest: well, i think charles is making a point that we can't emphasize enough. this is about schools working very closely with participants. we're all -- with parenteds. we're all in this together, we're all concerned about the education of children, and i think we need support. local school districts need the

support of parents in the community as they implement their programs, as they enforce, you know, behavior standards of students and so on. i mean, there are a number of examples, quite honestly. i was just thinking as he was talking about schools that have implemented dress codes for their students, and they've done it with the support of parents. and they've seen some pretty remarkable results from that. i mean, that's ooh not the panacea, but i think the larger point that you're making is an important one. this is a collaborative effort, and i think as school board members i've known over the years would be the very first to say they want to do this in cooperation with wanters. that support's critical. >> host: and we've been showing newspapers this morning from around the country on back to school and the stories in the cain county chronicle, this one a strategy to thrive after a leadership shake-up. st. peter's school aims to boost its enrollment, one of the many papers we'll show you today on

"washington journal" with back to school stories. but we want to get your thought as well. lawrence is in frostburg, maryland, as a parent. good morning. >> caller: thank you for c-span, and thank you for this excellent, excellent guest. it's certainly timely and relevant. just two quick questions and a point, if i can briefly get them in. in the state of maryland, we have a state board of education that is appointed by the governor. i hear your guest rightly talking about a great disconnect that's happening in terms of federal policy. what we have in maryland is a clear disconnect t, in my view, between our state board of education that in many cases has seen to be with lack of a better word dictatorial in terms of providing prescriptions and an actual bylaw that is just as much to obliging as state statute where they can literally -- although they're not an elected body, they can sit down together and make law that local boards of education

are obliged to follow. i think that that is something in terms of not just maryland, but other states around the union that also have a similar process in place that clearly needs to be addressed, because there's a great disconnect between local boards of and education the state board of education and parents and students who get lost in that mix who are isolated. >> host: all right, lawrence. >> guest: well, i can't speak directly to what's happening in maryland, but i will say that education is fundamentally a state function. i think every state constitution has provisions for a public system of education. and that generally is vested in state government. and so it's the states and local school districts that have that authority. many states have state boards of education, some are elected, some are appointed. they all play some role in developing state policy around education. but again, i go back, john, to what i was saying before. this really is about the different players in the system,

the leaders at the local -- >> so why don't we start at the end and put anne up first. >> thank you for the opportunity to be here. my particular paper is co-authored with widla rodriguez of the university of michigan who could not be here today. we focused on how the shortcomings of a postsecondary ratings system could lead to shortchanging hispanic-serving institutions and their outcomes. so as today's research and other research indicates, there are well documented differences in student and institutional characteristics between hsis and non-hsis that would lead hsis to have lower graduation rates on average than non-hsis. and so our analysis at the beginning of this paper, in the previous session we talked about the possibility of adjusting for some of these student and institutional characteristics. and so at the beginning of our

paper, we find that when we account for student characteristics, financial resources of institutions, policies at institutions that might influence completion, whether or not an institution has an open admissions policy, that the gaps in graduation rates -- and so that's how the most commonly-used measure being discussed right now to mosh -- measure institutional performance, six-year graduation rates, that the gap in graduation rates between hispanic-serving institutions and non-hispanic-serving institutions disappears. and i think this is consistent with particularly with what stella was talking about in the last session. so another topic that came up was the idea, possibility of adjusting for these differences. so if we were to calculate a ratings system with the data that we have available in data sets like the integrated

postsecondary education data system or ipeds, that's really the only data system that has national data on all of the postsecondary institutions in the u.s. we'll talk in this a minute about how there's problems with missing data, but it has the most comprehensive data. so what some have talked about the using is method of regression adjustment to level the playing field in terms of assessing institutional performance of minority-serving institutions and nonminority-serving institutions. >> something like what deborah santiago was talking about. >> exactly. exactly what deborah santiago was talking about. so in regression adjustment, the idea is to predict the graduation rate. based on what we would expect given the student and institutional financial characteristics, perhaps policies to promote college completion, admissions policies and compare it with the actual graduation rate.

and so if the expected graduation rate was higher than the actual graduation rate of an institution, that institution might be seen as underperforming. but if the graduation rate that was expected was below what we saw to be the actual grate rate of that institution, it might be interpreted as overperforming. and so i think one of the takeaways that's sort of been coming up is that msis and hsis are doing more with less. and so a lot of them, in fact, may not be underperforming as they appear when we only look at outcomes like graduation rates, but when we take into account these other factors, they may actually be doing better than we think. next slide. okay. so we went ahead and once we sort of established that the gaps disappeared, we then went

ahead and examined different ways of looking at regression adjustments. and so one of the problems we initially came across with this data set, ipeds, is that we can only adjust for characteristics on which there are available data. so that regression adjustments may be especially useful for looking at groups, large samples of institutions and comparing them with one another like stella flores did in her analysis, but they may be a little bit less useful in terms of looking at the performance of individual institutions when we break it out. ipeds has a lot of missing data, especially for hsis, and this is also the case with minority-serving institutions in general. and there are a lot of reasons for that, and we talk about that in the paper. we also have statistics on it, but it's a significant amount of missing data. but perhaps one of the most important things we heard in the last session is that academic preparation, the coursework that

students take, how they perform in that coursework in high school and stella flores used math as an indicator, that that's the most important predicter of college completion for students. and that data are not available, those data are not available in ipeds. so as stella flores was talking about, college completion is not a college issue, but we can't, with the current data available in ipeds, we can't necessarily take that into account as completely as we might like to yet. go ahead. so one of the things that we did in this paper was we took, we ran two adjustments where we, in one of them, we looked at student characteristics, institutional characteristics,

financial context characteristics, and we took what we could best get at in terms of pre-college which was whether or not an institution had an open admissions policy. and so that may not sound like pre-college, but that was the closest that we could get to and the most available data for hsis. and then we also, there's also a common measure of looking at institution, or at student characteristics of incoming admissions test scores. so incoming s.a.t. and a.c.t. scores. so we ran separate regression adjustments, one including the open admissions policy measure and the other one including the test scores policy. and so what we found was that whether a college seems under or overperforming depends on the variables that are included in the model. so in our analysis, nearly 30% of the institutions changed direction between when we included the admissions policy variable and when we included

the test scores variable. and so the fact that this calculation can change this much should raise red flags for the idea of financial aid to these red lights -- tying financial aid to these results. we know that in 2015 the rating system is proposed, and so we are trying to look at data issues. but if we go a step further and thinking about tying funding decisions to these kinds of ratings, that should really raise a red flag. next slide. and so this brings us to our recommendations. the first one is to collect more data known to be predictive of institutional performance. and so what came up in our haas session was the importance of academic preparation characteristics. so that's one example. another example that was raised is being able to track students

from institution to institution. and so right now the graduation rate measure used in ipeds is only students who begin at that institution and who finish at that institution and who are full-time students. and be students who finish in six years. and so when we consider that a lot of the students that we've been talking about, they may take longer to finish, they may not be full-time students, and latino students in particular are more likely to transfer. it's, those students are being left out of these conversations. so if it's possible to collect more data. i think one of the things that sylvia's going to talk about too is not only is it collecting data about the inputs and also mary beth, but rethinking what these institutions contribute. what other measures or indicators of institutional performance might there be that these minority-serving institutions and hs, -- hsis

are contributing. but the second recommendation is that we don't want to make this an unfunded mandate. a lot of these institutions don't have resources for data collection and reporting comparable measures. so providing capacity to do that would be a really good idea. i think part of this could also involve partnering with the national student data clearinghouse as deborah santiago mentioned to be able to track students. so that would be a good opportunity to track transfer students, for example, nationally, students who move institutions. resisting the temptation to implement high stakes policies based on inaccurate data and assessment is really key as well. and an important takeaway from our analysis. and finally, there are several higher education associations right now who are making efforts to find accountability metrics and to find reasonable ways of

reporting them. and some of them involve bringing in measures like transfer students. so to build on what those institutions are are already doing and involve them as partners and not reinvent the wheel, the work that's already being done at the national clearinghouse with these associations and higher ed leaders who are already trying to account better for institutional performance. thank you. >> thank you very much, photographer nunez. and we're -- professor nunez. and we're going to turn now to nicholas tillman, but before we do that, just arrived, deputy undersecretary studly is here, and want to make space for her at the table. so if you could just move down and grab another chair. >> there's a chair right here. >> okay. so we turn to nicholas tillman for the next paper presentation. >> great.

thank you for the opportunity to share my work here. a work in progress, for sure, and so i'd love to have conversations and ideas about where this could go next. but the central argument here for this paper which is called the differential impacts of college ratings, the central argument here is a very straight forward one, i think. a simple one that gets overlooked, but one that's critically important for thinking about the implications of a federal rating system and especially tying funds to a college's ratings. and that is place matters. so often in research and policy making on college access we focus on the process of opportunity; questions like did the student take ap courses, did the student apply for the fafsa, did they sort of do the right, take the right steps in order to be prepared to end roll and then per -- enroll and then persist

in college and procedural opportunity is also important. there's also the geography of opportunity which is something that matters here, and that's what is going to be explored in this paper. it draws on what we might be familiar with, this idea of food deserts which is part of a family of research on a community's built environment. we have research in social sciences that show low incomes, racially-segregated and high poverty communities across the country happen to have built environments where, for example, they have limited access to public parks, they have high density to to industries that pollute, they've got all these characteristics that define the community including, you know, access to healthy, affordable, nutrition food and food deserts is what we might be familiar with. but we should think of community college as part of the built environment of our communities. and on top of this, moving away from the process of opportunity to the geography opportunity

will help us think about how people make choices about where to go to college simply based on their built environment. not everybody has the luxury to shop around for college, to search across the country or even across the state for places to go to school. and so there are maces armed the country -- places around the country that we'll explore in a second and this paper dives into a little bit, and i think we're only scratching the surface, but there are places where these opportunities are mostly con constrained and not surprisingly cut along the lines of race and class. so if we hop to the next slide, we can see, hopefully, we can see a map of the united states. and this is broken out into all of the counties, 3,147 counties. sorry, anybody from alaska or hawaii, not included in this map. but this gives us a mental image about spatial factors that could matter in choosing a college. so what i did in this study is took all the counties, clustered

them around what are called commuting zones, and this is not something i made up, this is something that people have been using for years trying to identify the commuting patterns of counties. this means that, say, for in memphis, tennessee, for example, people cross state lines. that would be captured in a commuting zone which is a cluster of counties ca that kind of share common commuting patterns for labor. and so makes sense to me that people would also be more or less willing to also commute to school if they live in a common commuting zone. so what this paper does is it shrinks these 3,000 counties down into 700 commuting zones. and so commuting zones are the unit of analysis here. within each commuting zone, i wanted to see how many public colleges were even available. are there commuting zones that have zero public colleges? are there commuting zones that only have one public college? why public colleges? because that's where the majority of students enroll, that's most the affordable

option, and it's their mission to do so. so if on the next slide this should appear, if we just take this four-county cluster in southwest texas. eagle pass, anybody from the area, i'd love the talk with you and hear more about your thoughts on this, and i know dr. nunez has some comments as well prepared. here's a quick example, though, to try to illustrate this idea of deserts. this community, eagle pass, is represented by a large hispanic population, it is a community with high unemployment rates and low educational attainment rates. and this community, set of four counties has about 100,000 people who live there. and on the next slide you can see that a student who wanted to shop around for college in eagle pass commuting zone would, if they followed the logic of our federal policymakers, they would go to this web site, college navigator, put in their code, do a search for what kind of

colleges are nearby, and this is what they would find. can't see it, sorry. enter your zip code there on the left, and then you can say within 100 miles of my zip code how many schools are are there? there's four schools. the first two are two different -- two for-profit colleges that are the same institutions, the third one is southwest texas junior college, the fourth one is also a for-profit college. and so a student who lives in this community is probably going to be placebound, and if they're looking for colleges, have one of two options; go to the public community college or go to this for-profit school. so we have to really think about how people make choices in these commitments. if we advance the next slide, this just takes a quick snippet of some of the details that might be interesting to us in terms of the differences in these two different institutions. apologize for turning here. but we can see the middle group is the community college. it's largest program is general

studies. it has about 5,000 students, most of them underrepresented minorities. at high proportion receiving pell, lower net price than the other school, the for-profit college that serves tiny handful of students, a few dozen, mostly in cosmetology. so if you're not interested in cosmetology and you live there, you really don't have any choices where to attend. here is the tricky part, and this actually opens up a whole slippery slope about our ratings system. one of the criteria that would probably be included i would imagine because it always is a thought experiment at this point is the cohort default rate, the percentage of students who borrow in a school and default on their loan within three years of repayment. 30%, if you hit 30%, you're going to be on notice. if you do that nor three years, you lose access. that's a current cdi policy. this community college is at 24% already, and so if cdrs get included in a ratings system,

this college has a pretty high risk of being rated poorly. we can play this logic out, pretending that aid is eventually tie today a college's rating. that would probably mean that this college would lose access to aid or at least have it constrained in some way or another. and so a student, one of the 5,000 students who are attending this institution, if they're trying to find a better place to go, where are they going to go? and on the next slide, we can see that this is not just a problem in southwest texas -- and advance one more slide -- it's across the country. there are commuting zones that have a similar built environment as that as i just described. and in the paper i use this data set, i'd be happy to share and explore other questions you might be interested with this data, but i run a regression just to do a number of a simple count of institutions in each commuting p zone according to that commuting zone's economic profile, their unemployment rates and several ore characteristics that might matter and be interesting to

find some patterns in what these institutions -- i'm sorry, what these communities are like. ideally, there'd be no patterns. this would just be completely random, and it's not at all. these commuting zones are drawn along the lines of race and class just like the built environment inequalities are drawn along the lines of race and class in other ways. and so one in ten people in the u.s. live in one of these commuting zones. about one in five colleges, public colleges, i'm sorry, are located and serving these communities, and these communities tend to have growing hispanic populations, they tend to have what i'm assuming is going to be intergenerational poverty, intergenerational inequality because they have low educational attainment levels already. and so when we think about policy context, and i think the college navigator example kind of started helping us to think about the implications, we think about the implications here.

is it an unintended consequence if colleges that are serving these education deserts, and i'm not even sure i like to use that term because it does, it could have a deficit connotation to it, but i think we've got to really call it something and realize its built environment's different and these opportunities differ depending on where you live. so if a policy is going to accidentally affect colleges that are serving these communities, it's also by virtue disproportionately affecting minority communities and communities that are working cls and have already low levels of educational attainment. so we have to think about the role of place and that place matters and that the current discussion around college ratings is so focused on consumer information and consumer choices, and my hunch is that it's dominated by people who maybe don't have a lot of experience living in these communities or having institutions represented in

their frameworks. and so what we ought to think about is the role of place and geography in maybe even giving colleges waivers if they're serving one of these communities or least identifying which ones they are. thanks. >> thank you very much. professor hillman. we're going to turn to our last presenter of the morning, to sylvia. >> thanks for having me here today to talk about some ongoing research at the higher education research institute. for some of you who have been talking about weighted kinds of measures of looking at graduation completion, we've been doing this for institutions who participate in our national surveys for almost 20 years. we've been working with campuses to create student input-adjusted graduation rates, providing them equations and more recently calculators to understand how they're doing relative to the kind of students they're recruiting. the the other thing we've been doing is really using now more all of the national databases to

really look at what we call efficiency scores. so i'm going to talk about work related to those two areas. let me just start with the work that we've been doing on metrics for understanding graduation rates and degree completion, particularly in s.t.e.m. fields. both of these issues are vital to the u.s. economy, and broad access to minority-serving institutions which are mostly regionally focused are vital to the organization of diverse communities. and i think nicholas' work is actually beginning to say where you have a college has located itself so very important in terms of communities they serve. the second thing i want to say is any international rating system must be fair to students and fair to first generation, underrepresented groups. by underrepresented, i mean african-american, latino, native american and asian-american groups. what we know for those particular groups of students, those targets, low income, first generation, underrepresented, they are less likely to graduate

from any college. that is, they are more likely to graduate from some colleges, the more selective, obviously, are taking only those that they know will succeed. but when we know, when we control for a whole range of factors, we know that those students are still less likely to graduate. so let me get to the metrics, and before i move to the first slide, let me talk a little bit about them, because i'm just showing results. the raw graduation rates are not as effective in really identifying the real challenges of educating large numbers of these target populations that stand the most to gain from college. our work at harry is really to dheri is -- at heri is really to help them understand where they are and relative to peers and also how they might move forward in terms of improving their degree completion. but what we know, we know that we need to have metrics that take into account the types of

students institutions recruit and the types of resources that are available. so one of our met are ricks, i'll start -- metrics, i'll start with student-input adjustment measures. using our freshman survey, the one cohort year we started there were about 700 institutions that used this data. combine that with the national clearinghouse and ipeds data. we were able to produce predicters of graduation, and that institution, for every institution we were able to give them a score in terms of their actual versus their predicted. and so one of the things, one of the projects we've just recently completed is really looking at those institutions at the highest performers in terms of the student input-adjusted measure. and what we found of, let's see, 356 institutions that we had all that data on, about 108 were doing better than expected on all three groups, low income, first generation and underrepresented groups. but, of course, they varied in terms of their resources.

but the first metric does not take into account resources. so the actual versus expected i have have on the first slide, danny, i have basically three institutions that are 10-20 points higher doing higher than expected in terms of actually completing students. now, we put a 50% line graduation rate kind of at the center, so you'll see some institutions are below 50% but doing 10-11% points higher in terms of completion. there are some institutions that are expected to be below 50% but are doing above 50%, and some institutions, the top one is a public university, that actually started above 50%, and they actually increased that rate. so in a way we were looking at some talent development that's happening with these institutions. so let me, let's go to the next slide. all right.

so we took the 356 institutions, 108, and we tried to predict what was the characteristics of those institutions that were doing better than expected? and so the key predicters are those that you see on the slide. those institutions that actually had lower selectivity -- that is, they were broad access -- the amount of financial aid for first-time students and also the instructional expenditures per capita. and in some ways that's as it should be, students should be supported if you do have large numbers of low income students and if you have more low income, first generation and underrepresented groups, you actually need to direct your resources toward those types of students. so that's sort of one way to look at it. another way to look at it is we use the time ratings. you've gone online, time actually has a rating system, a ranking system based on graduation, affordability and accessibility, then they have a

holistic rank that combines the three, so you can see which colleges come up highest on all that. so we use the time ratings. you can go to the next slide, to compare our institutional performance indices. and really what you see is for the 16 campuses that we identified that were actually had an expected rate that was, well, let's put it -- they had an actual rate that was 10-20 points higher for low income, first generation and underrepresented students. a lot of them do rank higher on the time graduation rank, but usually those with a higher proportion of minority students are going to be ranking lower. so we know that msis, even though they're doing better than expected, that is 10-20 points higher than what they're expected given their student body, they're going to be ranking higher -- lower on any kind of similar type of rating system. the next slide is really using the time holistic rank which is

by using access, graduation and also affordability. you see then that there's no relationship between our performance index for underrepresented minority students and also this ranking system. but there are a few institutions that are very high on both hours and also the holistic rank. one particular, one hispanic-serving institution turns out to be very high on their holistic rank and very high on ours. but for the most part, others are going to be ranked lower. the other thing you find is of those in the top 16, you notice they rank lower on graduation because they probably are less affordable and less accessible, that is, the top 16 that are doing very well. so they might have smaller numbers of first generation, low income students. okay. but the next one, the next one really looks at our s.t.e.m. efficiency scores or which are a little bit different technique.

because there are some limitations and using a single cohort and also the fact that it's really focused on first-time, full-time freshmen, we actually ended up using the ipeds data using five years of data to understand completion. and really we used what's called models which compares similarly-situated institutions based on resources, enrollment and expenditures per student. this is another way of identifying top performers. so one of the things that we found, and that's a much longer paper, and there are several papers that you can actually access online, is that for the most part, most of the analysis, private institutions come out looking better on most kind of rankings. but what we find by using these efficiency scores is public institutions are more efficient. so we compare the s.t.e.m. completion rates, just the raw completion rates with the efficiency scores.

we find, for example, that minority-serving institutions, particularly hbcus and hispanic-serving institutions are more proficient at producing s.t.e.m. degrees for blacks and latino students respectively. the other thing is because these particular efficiency scores have not controlled for input, that's a different kind of measure, is more selective institutions do better than expected. well, in terms of efficiency. but what we've done is really then broken down the selectivity so you can take into account access to find, actually, the most efficient kinds of institutions. what we're doing next is trying to visit these institutions. so what's the bottom line here? any metric should take into account the characteristics of students, particularly focusing on the degree completion of low income, first generation and underrepresented minorities as well as institutional resources. i think both indicators help to identify those top performers in terms of social mobility, and i'm hoping that a ranking or rating system nationally would

actually be, actually, used for that not necessarily to basically just limit financial aid. because these institutions, and i want to reiterate what others on the panel have said, are really serving as engines of social mobility. even though their rates may not look that hot, actually we're seeing what they're able to do with the students is quite amazing. so one of the questions that came up earlier was, well, what do we do, how do we handle the input, how do we handle some of these things. for a large part now, these are only four-year institutions we looked at, was that two-year institutions are really doing the bulk of this work. and so that's additional areas to look at in the future. >> thank you very much. we have a number of respondents. i'm going to turn first to the representative from ace. >> okay. just playing musical chairs for a minute here. thank you, gary.

and on behalf of ace's center for policy research and strategy, i do want to thank the authors, the other respondents and those in attendance today for contributing to this important dialogue. turning to the paper by my colleague sitting right behind me, anne-marie nine was and her co-author, i applaud you for taking the lead on attempting to do what many have said would be a good approach to the president's ratings plan, which is to rate institutions based on estimates of how they should be performing compared with how they actually are performing. and as you'll see when you read the paper -- and i hope everyone can -- while a perfectly reasonable approach in theory as your analyses bear out, the practice of this approach is flawed. and, in fact, sheds light on how other analyses that may attempt to do something similar will run into the same danger, and that is that the very data collected by the department of education,

particularly the ipeds data as was used in the paper, is, in fact, wholly inadequate for the purposes of rating institutions. now, beyond the missing data that you cite as problematic, and it is, and beyond the instability of the redepression adjustments -- regression adjustment given a limited number of data points that fail to capture complex environments, there's really another underlying data reality which is not new i know for many people in the room, but it's something that is worth addressing here. and that is that the ipeds data -- and, again, if that is used to, you know, rate hispanic-serving institutions in particular and other minority-serving institutions, it's really not reflective of the students that attend those institutions. now, we at ace have done some looking at the national postsecondary student aid which is another data source to out of the department that is very robust. it's nationally representative,

but it's not a census data as is ipeds. and we see, to illustrate my point, that 49% of latinos are end rolled part time. so that path of latino students, these students are not going to be represented in the outcome measures in ipeds. you know, ipeds does not currently track outcome measures for part-time students. 36% of latinos delay postsecondary enrollment, and that's another group that won't be recognized. so this further means that hsis and other institutions, as i said, that serve these students well will go unrecognized. now, building on the first and second recommendations in the paper, i want to caution against the need to collect more data. what we need, and i'm certain that the authors would agree, is to collect accurate data. and, yes, this absolutely includes a better accounting of academic preparation given its relationship to retention and to completion.

so the arguments in the paper, in fact, raise another issue which is essential for consideration, and that is how the department and other agencies -- and this was referenced by a speaker earlier -- can actually empower and incentivize institutions. and i know anne-marie raised this as well to not only report out data, but to actually utilize their own data to improve student outcomes. so this means that in addition to the department needing to rethink its census data, the department and other agencies actually have within their power to assist institutions in doing a better job of utilizing data to identify performance and resource gaps and to meaningfully act on that information through institutional policy and practice. and this really means allowing institutions the capacity as was mentioned to have more institutional research efforts within their very own walls. i just wanted to say a couple more things. i really applaud nick's paper

for addressing environment. and we have some work coming up at the center for policy research and strategy to look at how state context really influences institutions and regional context as well. and i just want to say quickly we know that institutions of the same sector can reveal vastly different characteristics, you know, pending on the strength of the k-12 system that's around those institutions, the makeup of their community, the students they are serving, the policies around transfer around undocumented student access to higher education, and i really want to applaud a point made earlier. i think it was by stella flores. to look at state data and for the federal government to, in fact, partner with states and to incent states to use their data to also allow for some accountability measures. a final point, and i know i'm almost out of time, i do want to let people know that we have a paper that came out earlier this year that looks at the college

choice behavior of low income students, and there's a paper forthcoming for the white house initiative on educational excellence for hispanics which actually shows the way that hispanic students choose colleges. and if we believe that the ranking -- or, excuse me, the ratings system is a de facto ranking which i, in fact, think it will become, then we know that hispanic students don't actually use rankings. in fact, around 16% of latinos cited in heri's own data, those rankings were an important factor in their choice process. so, congratulations again to the authors on their paper, particularly nunez and rodriguez, and thank you for having me. >> mary beth. >> all right. let's see, i guess i need my -- >> yeah. >> okay. so i'm responding to sylvia hurtado and her colleague's paper, and i'm just going to be short and sweet here because i am short and sweet.

[laughter] i thought i'd get that in there. [laughter] so i really enjoyed sylvia's paper, and i wanted to, i think it's really important that you read this paper for a number of reasons. one of the things that i like the most is that in the very beginning of the paper sylvia talks about how they wanted to use metrics that were fair to both students and institutions. and i really like the use of the word "fair," because i think all too off when we see research -- too often when we see research and national reports we don't see kind of the fair use of metrics. and another thing that i thought was really interesting about the paper is that she does a really, really good job of reminding us how for so long researchers have been calling for measures that consider student inputs as well as institutional inputs. and it's been so long, right? and yet we still have all of these efforts that don't

consider student inputsful we have all of these -- inputs. we have all of these ways of looking at institutions that don't take into account who the students are that attend the institutions, yet we've been asking for so long for this to not happen. and i'm just, i guess i'm wondering when policymakers are going to start to listen to that, and maybe we could just stop it right now, right? that would be great. and the other thing that i thought was really important is that it, the work that sylvia and her friend or her colleagues are doing, it helps us to look at msis and really dig a little bit deeper and try to figure out why they are achieving a greater efficiency and success at many of these institutions. and i was really glad that you said you're going to start visiting them, right? because we, we do know that there is success taking place at a variety of different can msis, but we don't really dig

deeper. and your point about the institutional research that you wrought up earlier which i think -- that you brought up earlier which i think at some larger hsis it's not as big of a deal, but it really is a large problem at many underresourced hcbus and some smaller hsis as well. it's very difficult to collect institutional data. so i think that's really important. two things that i thought of when i read the paper, i hope that people won't take your very, very good research and start using it as a litmus test. [laughter] so i really worry about that, and i hope -- excuse me, i hope that you'll caution people against that because i could see people taking it and saying, well, it's not efficient. and what i would love to see happen to the institutions that aren't, that don't come out as being efficient is that perhaps they're given a mentor institution that is of a similar type or size or maybe within the same sector that can work with them to become more efficient. and so i hope that that'll be

one of the recommendations in all that you do, because i can just imagine someone coming along and saying, well, sylvia hurtado says, and so that would be the one thing i'd really like you to think about. but thank you, great work. great work. >> sara? >> so i'm going to make some comments across the papers, but in particular my colleague, nick hillman's paper which i want to recommend that you take a look at in full. i think this is a really important and different analysis, and he was able to give you just a sort of taste of it just now. so one of the most important findings, actually, in dr. hurtado's paper is the reconfirmation that providing grant aid and investing in terms of instructional spending pays off. in other words, the inputs actually matter. and she shows this in the paper, and it's kind of remarkable that we've lost sight of it.

and in this discussion about accountability for higher education, the question should therefore be, what is this policy effort going to do to help insure that more students get is access to those sorts of resources? if those inputs matter, what is accountability going to do to increase the likelihood that the students who need those things in order to graduate are going to get them? will it bring new resources to the institutions that don't currently have them? will it create better matches between students who need to be educated in high resource settings? and institutions that actually pez those resources. possess those resources. is there any indication that that, in fact, is going to occur? i don't think so. it's extraordinarily doubtful. in fact, what looks like is going to happen is that the institutions that don't have those things, that don't have much instructional spending for their students, grow large

numbers of disadvantaged students will simply be poorly rated and cut out. so where is that going to leave us? does it mean that disadvantaged students are, therefore, going to know because an institution was badly rated that they should avoid it? whether or not they know it, what nick's paper shows is they're unlikely to be able to do anything about it. are you going to close the poorly-rated institutions and leave them with no options? is that a better scenario? i mean, these are, these seem like pretty fundamental and basic questions to ask, but they have no answers. and they haven't been addressed. we know within k-12 schools that we closed some failing schools, we called them failing, and parents still wanted to send their kids there. why? because they're the schools in their neighborhoods. where are they supposed to go? just because someone declares the school bad was it has -- because it has a lot of black and brown kids in it momentum mean that it is, in fact, a bad

school. so the problem with this accountability movement is not its interest in institutional responsibility. it takes a starting presumption with the idea that schools don't care about the outcomes of their students. and i don't think i've ever been to a school public, private or for-profit, frankly, that actually doesn't care. there are some, but for the majority they do. the problem is this system preempts any effort to establish equity and adequacy of resources among schools, and it does nothing to gap l with the fact -- grapple with the facts of the system that expose the disadvantaged students to the most risk. that's the worst part of our system right now, is how risky it is for somebody with disadvantage to enter our system and how likely it is they're going to end up with debt in their degree. so i think if we want to experiment with this accountability, we ought to start in a safe place. we ought to start with schools that have high resources and are using substantial federal funds. more than 500 institutions today

with an average endowment of more than $100,000 per student who are using federal tax dollars to cover their cost of attendance which are over $43,000 a year. students attending these expensive schools are using federal financial aid to cover 20% of their cost of attendance. what's the national return on that investment? let's start our grand experiment in those settings. where those students are at no risk of noncompletion, but we have senate concerns about the resource -- significant concerns about the resources we are spending. why not begin this rather than in a place where we're experimenting with the poor and disadvantaged students and limiting their educational opportunities. >> thank you. [applause] anne-marie? >> i'm just going to comment briefly on nicholas hillman's paper about the case of education deserts. and as has been mentioned, one

of the most important contributions of this particular paper is the importance of geography in terms of students' college choice. that as he states, about seven in ten b students in the u.s. choose to stay close to home and how important a local postsecondary option is. and that 10% of the u.s. population lives in an education desert. as we've also heard, such institutions are more likely to receive lower scores, or they would be on a lot of the metrics that we've been talking about in an accountability system. so penalizing these institutions for low scores by awarding them less funding could strain their capacity to further serve their students and in some places could lead these institutions to close. and this would disproportionally hurt students and families for who that may be only or one of the few postsecondary options. so one of the things that i would encourage us all to

consider and for hill match to develop is -- hillman to develop is to emphasize that part of the limitation of postsecondary options in certain geographic areas is influenced by the historical context of those areas, and as laurel also raised, state context, local policy context x. this includes historical segregation and discrimination in areas such as texas and other areas of the south. where mexican-americans and african-americans have been concentrated and and often have had few postsecondary institutional options. and so at the federal level, the land grant act in 1862 sought to bring postsecondary options to less settled areas of the u.s. so is maybe education -- or the frontier at the time. but many states continue to bar the enrollment of african-americans in their colleges and universities. ..

and in the institutions were i work at has been directly impacted why this. in the late 1980s it was a legal case ruling that texas had an equitably funded his post secondary institution. particularly not funny those institutions that had served large proportions of mexican-americans, including many of those along the south

texas border. deborah sandy i was also written about this. in effect identifying that south texas was an education desert, and we see that that is still the case with one of the counties that nicholas hillman talked about. people in my own department actually teach students at the institution that nick hillman identified. we bring a program out to them. so that they can have access to doctoral education and educational leadership and policy studies. and even though i live in a metropolitan area can even though we are located in the metropolitan area we have to reach out to them. we are still held to same standards. texas is one of the 25 or so states with performanc performa. we are still held to the same standards of outcomes as of universities like university of texas at austin us are probably half as many low-income students

or fewer, and we serve over twice as many latino students. and i think an important point about the education is there maybe a ripple effect even further away from those region. because institutions in my county, it's not necessary to an education desert but we're trying to make up for what's going on in some of the counties around us. if we also get penalized, and that has an even greater ramification. i think that's important to consider in future policy. >> thank you very much. our final commentary this morning is going to be in a sense, session is try to bring researchers to talk to people about pending policy issues. fortunately, we have a policy maker here who is in the middle of this discussion and was willing to come and meet with us today and talk about her

reactions to what she's heard and what's going on in the government. so deputy undersecretary dudley is our honored guest at the session and we are very pleased to have her with us today. >> thank you, gary. it's good to be doing something with you and the center again. and i'm not the only one here who is working on this project. you'll be glad to know some of my colleagues are here as well. because of your questions are our questions. the presidency challenge, if it has done nothing since last august 22, is along with a lot of other work by many of you along the way, is moving the conversation and the focus to better questions. and the best collection of all we can frame and lots of different ways, is what institutions are successfully contributing to college completion with a meaningful education for students with the least income in the least historic opportunity. that's the way we are coming at

it, and i sense in literally this year that we're asking smart questions with as the objective. to do that we have to ask what practices, conditions, investments lead to those positive outcomes what educational choices, what kinds of recruiting and selection process, what financial aid practices in student support and linkages with jobs and the workplace will actually help us accomplish those things. and so one alternative to trying to squeeze months of thinking and reading into five minutes is to simply say yes, fair questions. every question that i've heard and read in the papers is a very appropriate one. and you would be pleased to know that the conversations that we are having our italy similar to the ones that i've heard this

morning and that i am engaged with was some of you and so my colleagues have been having with many of you over time. we are all trying to think about how to design methods that would capture important differences and advance our policy goal. my own personal tasks when we have some -- test women something to test, we be whether the places that you think are doing a good job, and oppressive job in turning access into quality completion are actually well rated in the process. that's going to be my home version of whether what we do is working. and that the one thing we worry about across the sectors, the one that you say that placer struggling, i would hesitate to let my nephew go there, are the ones we should also be identifying as worthy of concern. it should be very practical and logical. and one of the correctives is

one of the rallies is that we will not lead to a proposal. we will have continued conversations and what we think is the best possible amount of issue four series conversation. so we will watch the expertise of all of you to help us. as we think about the design let me take a few items because of want you to know at a more granular level how we're thinking about some of these. we are thinking about, for example, student academic preparation. but given that not all students do the same kinds of things on the way to post secondary experience, we're also thinking about what exists and what markers there might be. is their way to the valley what high school, and yet was inevitably comes right from high school. zip codes. for what home and what moment, do they tell us about the

academic preparation in a way that's usable enough to be helpful. we know, i haven't spoken to a single person throughout this process who visit well aware that the graduation rate is a very limited indicator of folks like you know it is going to be better, it will account better for part-time and transfer students within a couple of cycles. what can we do with that, how will that affect things over time, and will no that solve a problem or are there other ways to get at the actual population. first generation of status is another element that we are thinking about. one area that i didn't hear you talk about that's also much on our minds if you want to go on the of is the whole area of looking at postgraduate outcomes

where we have graduation. we can look at things like repayment or cohort to default we. but looking at whether people work, whether they work at above the bare minimum, whether they're better off in terms of their earnings capacity after school is an important set of questions and it's one the students and families tell us matter to the. but it like all the others is complicated by existing discrimination in the marketplace, we think about whether to use earnings information early in somewhat screwed or a little bit later after there's more stability. but a counter narrative is that the further out people are, the more the earnings may be affected by race and gender bias. by local and geographic income realities. i don't have to do all of you that this is hard work but i want you to understand that we are working as hard as you are to understand what those effects

would be. gary bass they think about some questions for you. one of them that i would have, decide how to accommodate all of this and find something that does have meaning and achieves, incorporates the kind of voice that you're looking at strongly enough, seriously enough that we get answers to what schools are indeed contributing to real outcomes for low income and previously excluded people. how do we do that to serve the dual purposes of this system to the extent that the are both consumer student family council desires to understand immediate choices that people have to make based on their own options and what might happen for student at a particular school, and that there are policy and potentially federal investment decisions to be made, would you do the same

things? would you do different things for them? and so what were those differences be? because it's been pointed out very thoughtfully that when you're trying to do two things, you may need variations for those two purposes. and yet there are challenges in different data. we would love help combat. we, it's very likely that we would have some experience and some ability to try them out and improve them before proposals came forward to congress, which would have to approve any affects on the way federal student aid or other federal resources are allocated based on a rating system. so our hope is that we would find we achieve those goals, possibly even revise it to satisfy the questions that you

have asked, and be able to use ratings for several different purposes. one is, this is the rare place where someone in this $150 billion doesn't have any accountability for how it is spent in terms of the results that are achieved. we have very soft, subjective, informal measures. but to sustained national conference in the student aid system and to be able to continue to justify that we're getting what we want as a nation from it, which is education opportunity and education results for all, we need to have a lyrical structure of support that is strong enough to keep us putting precious and highly competitive or federal funds in the student aid. one very clear purpose is also to help push the state

reinvestment in student aid, federal money cannot chase the declines we were experiencing for so long in state investment in public business. and that reorientation of the traditional bargain about the feds, the states, institutions and families and philanthropy together thing for education is something that we need to improve. fortune is been stabilizing in many states and maybe we are moving in that positive direction. but something that is, is overlooked if we think about the danger side of an allocation that follows institutions that are effective in educating first generation underrepresented

students, those who do indeed pose an educational challenge for institutions and his graduation is the reason for our investing so much in student aid, we look at the negative side of who might lose. i would encourage us all to think about the positive side of having the information that would allow us to reinforce the resources and grow the resources to those institutions who advocate of track record that many of you were talking about so that we put the resources into their hands to be able to do the things that they're doing, or do it for more students or do it with less struggle, and be the models and begins we are looking for for that kind of success. this is really obligated where many of you in itunes and on the path have been very generous with us and helping us think about these issues. and i hope we can count on you to do it as we get closer diversions that we would like to

share with you and then share with the entire public. so that we can achieve the kinds of goals that you were talking about and do it according to the tests that -- its center, right? sara goldrick-rab was talking about. because, in fact, those are my questions as well. those are the benchmarks for us any system that succeeds in understanding better who is succeeding at these important goals. and where students are truly being built and disadvantaged and poor institutions should not be able to participate in our aid system. >> thank you very much. we have time for two or three short questions but it -- come down to the microphone, or if you don't, i will ask some. this is your chance. >> i'm from the national center

and my question is short for doctor hellman. the slide you showed in the cohort default rate, for for-profit said there was a little astrid. can you explain that? >> i forgot about that part. my sense is that school, bill had to teach students to you. my guess is that they probably don't need to report that the data out to the feds. if you have an institution that is less than 30 borrowers get a different criteria for tackling default rate. so it's my hunch without one. which actually speaks volumes about how we measure things we think are pretty straight forward but, in fact, are not. not. >> that could mean zero, zero to 100. >> thank you for your research and thank you for this panel. my name is gisele hunt. i am a reporter with the black press. my question is more about the

proposed standards for accountability it seems a lot of thought has gone into the factors, but has there been any thought about helping poorly ranked schools actually can themselves around and improve? i know that as a senator was saying and we've seen what happens at the k-12 level, but is that something were trying to duplicate on a higher education level? >> go ahead. >> we have lots of programs. currently in place we do try to strengthen institutions to be able to carry out these purposes, and the entire notion of including improvement so that institutions wherever they are, if they're on a positive trajectory toward greater effectiveness on whatever measures we end up choosing would be recognized and be

protected, as everybody said, it's a little hard to talk to a something that doesn't exist but everything i say is, either people suggested or it is possible that we might. because the goal is to have lots of opportunities for people to go to schools that will serve them well. our objective is not to eliminate. it's to improve the overall system availability. so the more we can identify good practices, expedite people's awareness of them, identify innovations in any part of education that add value for educating the population that we're talking about, the happier we will all be. so there's no reason that, we're not looking to eliminate endless there are places that are truly not using federal funds to get educational value for people.

improvement would be very important. >> that was the last question, the last word. i just want to take a couple of minutes. we are famous for doing things on time in a project, and you've given us your morning and we appreciate that tremendously. what you've seen today is a very thoughtful multidimensional effort to understand and think about what can be done about the accountability proposals that the president and others have made in higher education. shattered by worries about accountability, plants that exist in a number of our states, elementary and secondary that produce counter productive impacts. different analyses have shown you many ways looking at these issues. some looking at the geography, looking at pre-college preparation to looking at a variety of institutional

resource factors, inequalities in many respects. we have a very complicated system of higher education and which the states are the primary actors in public sector. so we have in essence 50 different higher education systems in a country where the federal government has been the primary actor in the student aid area. very important. vast amount of money goes into this, and the vast amount of the hopes of generations of americans that their children will have a chance to be in the middle class. depends on having post-secondary credentials. show the stakes are extremely high here. and the reason we did this conference is we want to make sure that they're not the unintended harm done. the first rule of policymaking, especially when it's so close to the destiny of america and its people, is do no harm, do no

additional harm. try to think about ways that you can do things positively and country. i don't think anybody who is in today doesn't think that it would be better to have more, accountable institutions. it's not a simple issue. particularly it's not a simple issue as some of our analysts have pointed out when the data that you have doesn't exist. when you in fact a soundbite on the basis of a theoretical data that doesn't yet exist, you create problems that are not foreseeable on any dimension. so i think what we would be doing it as we go forward in the theory come is to be taking these papers, now that the authors have had a chance to speak to each other, we'd each other's work, hear your comments and questions, here's a very powerful response of our deputy undersecretary, to think deeply about them to revise them, we will be publishing these as they

go through peer-reviewed and our revise. they will be on our website and i assume collaborating institutions, and they will certainly. in the professional literature. we hope they will appear in congressional testimony and other considerations as this unfolds. this was not designed to be an attack on the obama administration and its proposals. it's a thoughtful way to try to contribute to an ongoing discussion, and we greatly appreciate the welcoming of these contributions into the discussion but i'd like to thank the authors and commentators and all of you who asked interesting questions today, to tell you this is just a step. this is a very high set of issues that will determine the destiny of individual students, of communities. it will affect racial equality in the trendy. it will affect mobility.

it's a very high-stakes issue so i urge all of us to continue to look at this with great interest and careful and critical judgment, and to contribute to the policymaking process that turns out to be a lot smarter than the policymaking that was done in elementary and secondary education went soundbite really were enacted into law and we have been stuck with him for 14 years. i do like to thank everybody for their participation, and to close this session now. thank you very much. [applause] [inaudible conversations]

[inaudible conversations] >> if you missed any of this discussion about education you can find it online on a real library at c-span.org. later today more a american history tv on the theme of the war of 1812 with britain and the burning of washington. tonight the battle of bladensburg and other events from the time starting at 8 p.m. eastern on c-span3.

>> here's a look at some of our program this week on the c-span networks. on c-span tonight at 8 p.m. eastern, oral arguments in the case of aclu versus clapper. the second circuit court of appeals hears the chance to the national city agencies phone surveillance program.

>> the american bar association held its annual homeland security law institute conference recently. in one of the sessions expert on transportation and cargo supply chains talked about security screening for about one hour. >> thank you, emily. thanks everyone for being here today. this is the transportation cargo breakout session. let's say we have sold-out room for this discussion. my name is jeff and i will be the moderator of today's panel. and just to reiterate by way of brief introduction and credentialing, i did serve at the transportation security administration for a couple of years and then at the department of homeland security were sector michael chertoff. three years in total with dhs and the administration, and i'm thankful not only to have

survived the expense that i am thriving. i'm excited today to have on a panel five experts in the field of transportation and supply chain security and operations. i was looking over the bios last night and noticed that four out of five of them, their last names start with f. i was time to come up with a clever name for a bailout. maybe the f team but, of course, that leaves -- that leaves larry out. okay. we can certainly insert a f into your name. that's not a very complementary name for this austere group. i will introduce them briefly in a minute but of course at this point it want to introduce themselves to you in their opening statements. the format today we a panel discussion. i will open a brief statement setting the stage for the issue

we will be discussing and then each of the panels will get a five to submit statement themselves. then i will aggressively can't e cross examine each -- know. i will delicately toss questions to the panel for about 15 minutes, and i think that based on the depth of knowledge and expertise in these areas we will have a vibrant discussion of the issue. and then i'm going to open up to you all for questions so i expect some good questions from you. get ready. as the panel the script in the brochure states, threats to transportation sector supply chain, and reduce vulnerabilities. in recent years these threats of microphone that physical to the critical. cybercrime, warfare, terrorism presents new and difficult

problems for the supply chain sectors that rely more and more on interconnected networks. this panel review the current legal and public policy issues facing critical infrastructure sectors including legislative and realtor efforts to reduce risk the impact businesses. for example, over the past year to 18 months some of you have followed this issue for a number of years before that but there's been increased activity on the government side in particular, the president's executive order 13636, the nist framework development and publication and adoption and several bills in congress, two of which have passed the house of representatives, both of which placed dh in the position of the go between, the go between the private and public sector on cyber the information sharing. and so the discussion will also

address effectiveness of current transportation and cargo security programs. and the implication of the this framework as i mentioned. by with brief introduction on the panel, a frantic, leader of the boeing commercial airplanes cyber one team in that capacity she is developing and coordinating a public-private partnership between the aviation industry and the u.s. government in order to establish aviation information sharing and analysis, isac. very, very interesting. tom is a senior cybersecurity strategist tells with the department of homeland security national protection and programs directorate. at they still haven't changed that name yet. wikileaks works on cyber city insurance, cyber risk management in support of implementation of the presence executive order, title improving critical infrastructure and cybersecurity. prior to the desert on the house homeland security committee.

tom farmer is assistant vice president for security at the association of american railroads. tom and i had the pleasure of working together at tsa and we still agreed to help me out here today. in his current role the court makes the development of railroad industry security policy among other duties, and at tsa he served as a mass transit and passenger rail security division and then larry leads and manages projects in the field of telecommunications security for aecom, a global engineering firm and he will give some insightful remarks on how cybersecurity effects certain modes of transportation. i caught a presentation that larry had given regarding airports subsidy which i found fast and frantic don't think there's much out there. i'm looking forward to your comments. and then andrew farley is the

cofounder of ict strategies, a subsidiary of command consultant group which provides strategic advice to clients seeking current and innovative inside end of order management and supply chain challenges in the u.s. around the world and he is the former customs and border protection official. certainly physical security and resilience plays a significant role protecting transportation and supply chain sector as well as the critical infrastructure as the whole. over the course of several years and i spoke to the moderator of this panel who had moderated the panel for the last several years, and i mentioned to cybersecurity and he said we have been kind of migrating from the physical to the virtual and talking more and more about cybersecurity, and i thought that particularly looking at the chronology of events after 9/11 we have certainly focused on

physical, or we evaluating physical security for critical infrastructure. and over the course of time i think there's been tremendous strides that have been made, not only on securing the curriculum structure but on resilience, public-private partnerships. but one of the things i thought was missing, or least hadn't been fully baked was cybersecurity. and now we are so interdependent on networks and communicating and operating systems using open networks as well as closed networks i thought that there were certain vulnerabilities that even me as a novice, were thinking about. so i thought that today's panel we could focus on little bit more on the cyber side of things. survey don't want to leave out the developments, new developments or news regarding the physical side of equation,

but i do want to focus on the cyber site. that seems to be a theme for this conference over all. so i look forward to hearing the panelists comments. why don't we just started with you, larry, is that okay? thank you. >> good late, early morning to join. once again thank you to jeff for the opportunity to come to speak before you today. once again i am larry jaffe. i work for a cuppa called aecom. i like to think of us as the biggest company have never heard of. we are architects, engineers, construction managers and operations and maintenance personnel. on the operations and maintenance side we operate in a couple of large facilities for the government and for private industry. i do construction management side you may have seen either

aecom on one of our subsidiary companies, fishman, on the pentagon renovation. we would construction mentioned on that and also on the world trade center reconstruction. but my group is part of the design and engineering portion of aecom, and we focused mainly on roadways, and the transportation sector. roadways, bridges, tunnels, mass transit, seaports and airports. we also have to sectors can also work in the energy sector as well as transportation. the public sector and a lot can we do a lot of critical infrastructure protection, a lot of critical infrastructure in terms of water systems and arenas like that. my group specific work in a technology editor i provide all the technologies that go into these facilities. so the telecom, audiovisual, radio systems as well as the security systems that go into. and i provide some of the

cybersecurity aspects of these systems. so control systems like this hud systems or the building management systems, you know, walkover in the back of the room consumer hits a button on the wall and the lights go to a certain brightness and all. that's a building managemenmanagemen t control system that also the ones that control the hvac and the system or perhaps the ones that maybe if you're an additional facility, well, the valves and what have you to control the flow of chemicals to the pipes. in the refinery those are all examples of control systems. in the transportation and arenas we are talking things like positive train control and signaling. also in some respect also do with things like these message signs you see on the side of the road, intelligent transportation systems that are out there. i've got a commit on the way home and i will pass through a lane that switches direction in the market is headed into d.c. and in evening the same lane is

heading out of d.c. and have always wondered what would happen if someone got hold of the controller efflux those signals around. summit wasn't driving that road every day might end up going the wrong way down the road and causing some trouble. and if someone were to happen at the same time they coordinated that was some bad weather, we could be in for a whole mess of trouble there. but what i find interesting about these operational technologies, i want to spend a minute to talk about operational technologies and information technology to we're all fully with information technologies. you have laptops, cell phones, and this is the kind of stuff we've all known has been around since the '40s and 50s. from a cybersecurity standpoint we have been doing a lot of work in the cybersecurity arena for information technology. so the pccf at your desk and the servers that in your server rooms, the cloud, the internet.

a lot of effort has been spent, a lot of time spent working on the best procedures and practices and techniques and controls for securing that infrastructure. how good it's going is a whole nother story but it's been a lot of effort in that respect. operational technologies are more of those control systems or systems that manipulate physical things. as i mentioned the lanes of traffic that go back and forth, industrial controls out in refineries and chemical plants, train control systems as i've mentioned. what's unique and different about these systems is that first off the control physical things so if they go awry, or so we were able to control them and set them up in a way that could be detrimental or harmful to personnel, to physical assets, to the environment, these are the things we want to pay attention to.

and, unfortunately, there really is very little with respect -- there's very little going on in the realm of the cybersecurity in these physical logical systems. with respect to what's going on in i.t., i.t. we've been doing this for decades. on the ot side, not so much. historically that's because most of these ot systems, operational the lg systems, they were one off devices. they were in areas nobody saw. they were specialized. they were often physical relays and things that had to physically move in order to make these controls have been. they were almost never networked together with any other systems, and it was a very, very small group of people who really understood them, who had to program them or design them. and that was the case with up until just a few years ago. most recently, i got a bill for my electric company that tells

the to go to the website they showed me day by day how much electricity i was using in my home. that's because, and i can get on line and i can go look at all kinds of good statistics on my energy uses and my home. that's because the meter on the outside of my house is in some way, shape, or form connected up to a web server somewhere that i can then access. so if i can access the web server and the web server can access some way that meter on the outside of my house, so i'm wondering is it possible for someone out to go turn off the power to my house? imagine that times 100,000 homes or businesses. so we have that is one difference between operation to decide and information technology side, but what's also interesting about the operational technology side is that these systems were built, many of them, 30, 40 years ago

and they were designed and engineered to operate 24/7 for decades your one thing you might notice on your pc at the office, every tuesday well, once a month on tuesday, your computer reboots itself in the middle of the night. this is microsoft's any down a patch to your computer to fill any holes that they discovered over the last month. sometimes it's more often, sometimes it's twice a month and your internal i.t. department may do it once a day or even more often. that's because you can turn off your computer for a few minutes and actually not a big deal. but i really can't turn off one of these positive train control systems or a refineries plumbing. i can't turn that off because, first of all, it may take a day or more than a day even to restart that equipment. so the opportunity to patch

known vulnerabilities in these operational technology systems is very limited. so that cause us to have to do whole bunch of other medications to get around that problem. that's are some of the work that i do comes in as trying to find out how all this works. so a couple of examples where this comes in. back in 2003 you might avert a csx had an issue. one or more of their systems was infected with a virus, and then cleaned that up as a matter of prudence, they shut down or took off-line some of their operational systems just to make sure there was no impact to them. and, indeed, that caused some ability for them to halt traffic around and even spilled over onto anthrax -- amtrak's schedule but the reason in 2008 in poland, a young fellow, 14, sat and watched the light rail train system that went through

the city again he figured out, and i'm not quite sure how but he figured out that the trains used infrared signals, and this is what you have on your remote control for your tv at home, that's what they used, the trains would say go, send an infrared signal to the switches on the ground to move them from one track to another. and he figured this out and he went and bought some remote controls and have them record the save and then was able to play back to the switches and he was actually in control of the switches throughout his town. welcome one thing led to another and he did manage to derail for cars and cause about a dozen entries one day. they found him and i'm hoping that they have since changed to something other than infrared. we all know about the metro train collision that occurred here a couple years back, and those caused by a faulty piece of operational technology equipment and assembly a cybersecurity event per se, but

have they had a cybersecurity program specifically focused on operational technologies they might have tested it with more often and realized that this was a piece of it could that have actually failed for quite some time. so while the operational technology systems, they originally grew up as stand-alone kind of obscure systems, what's happening now is that they are indeed getting connected to internet in one way shape or another, and they are communicating tween them originally with some custom protocols that really was very obscure, but now we're are starting to use more common protocols, internet protocol ip, you probably heard that before. and that is part and parcel to the issues that not only not do we have systems that are connected to the internet, using commonly available protocols to do it, but our adversaries have

learned about this. they know about this and they're taking much more interest in it. now, about once a year verizon comes along and they do a big mega study. last you to look at about 63,000 different security incidents across the country and across many different industries. that good news from that is the transportation industry had a very, very small amount of reported incidents. not to say that there weren't more but of this large sample there were only 24 or so incidents reported in the transportation industry. what i found interesting and different and all of the other industries that were reporting is that the highest percentage of incidents in the transportation industry will be cyber espionage type. and with a small sample like that it's hard to make an extrapolation except to get you to a sure eye brows. why is, what potentially could

our state adversary be interested in our transportation systems? so that's something we need to look at. the other thing, other systems as well, are also getting connected. you go to the airport and you see the pilots and the folk who work there. the other batches. there's been a lot of regulation involved around badging people at airport, so that regulation has cost us to sort of make these badging systems super badging systems and originally they were part of the access control system. now we have to separate them out and make them stand-alone systems, but in doing so would have to connect into the unit to allow people to self register and be able to do in ci checks and things like that. so it is essentially becoming more interconnected, interconnected world in this arena of operational technologies. and the problem that we're seeing is that there just isn't a whole lot of attention being put on by the operators.

in the energy sector, probably the most regulation and rules about how and what you have to do to protect from a cybersecurity standpoint, the energy systems here in the u.s. but as you go down towards transportation and water, those are much, much less developed. so that's the way we're trying to focus on now is trying to help these transportation issues get more to speed. so i think that's it for me. i want to thank you all again. i want to thank the institute for this opportunity and i will turn it back over to jeff. >> thank you, larry. faceting. tom farmer. >> -- fascinating. >> thank you, jeff, very much. thanks to all of you for taking time to join us for this forum. i served as assistant vice president for security association of american railroad, and we represent the major freight in the united

states, kansas city southern, cfx, norfolk southern, unit pacific as well as the two lead canadian freight roads that operate extensive in the u.s. amtrak, alaska railroad, hundreds of short line carriers, and a growing number of commuter rail. and as a real credit to this industry, in the aftermath of 9/11, the industry came together, brought in subject matter expertise in the areas of intelligence, counterterrorism, brought in shipping industry, focus on developing an integrated security plan that would be applied across the board and adapter that each participate railroad within the context of its unique operation. so that plan took effect in early 2002, literally there were still fire burning at the pentagon when this group convened. we focused effort, look it is across the board to ask assess

risk, broken up into five teams, military shipments, lifecycle of the train from where the potential one of those are in the operation of the train, communications to cyber technology, and critical infrastructure, assets that the network. what a collective effort produced by early 2002 was executed plan with four alert levels that called for increasing cq measures in those areas as threat levels escalated. so before you had a fully functioning transportation security administration, he for you have a department homeland security, we had a system that was initially utilized to evaluate the threat level in various sectors of the economy. they develop a plan along those lines and that plan remains in effect to this day. it is thoroughly exercised every year. it is updated based on lessons learned, and improvement dedicated to ensuring that we're making the right processes and

effective cortège income to make sure that plan remains vital to the lot that can be talked about in that plan. i'm privileged to represent an into shape, but what if we focus on in our association has even helped manage the overall program, is information. and so at the association and american railroad can we operate was called the railway alert network, the means by which we provide intelligence, security information across the industry to ensure we're informing awareness of potential security concern. that includes a very good chance of commuters with government, based on and she proposals with tsa, dhs, fbi to make sure we're putting forth the right type of information to ensure that we maintain a level of preparedness we need to based on the volume of threats. i think many problems in life can be better understood by analogies of baseball. if you'll indulge me i will do that here.

when the iconic moment in history of the gang is october 1951 culminate event of the 1951 baseball season in the national league. bobby thompson hit a three-run homer with one out in the bottom of the ninth propels the new giants to a win and the payment against brooklyn dodgers. the dodgers had a lead in mid-august of 13 and a half games. the giants were perceived as dead and buried at one point the dodgers manager when asked about the giants said giants? are they still in the league? they were. they want 38 other final 45 games and finished the season in a time. at a time that called for a three-game playoff. the giants won the first gang. the dodgers won the second. it came down to the finale in new york. many things happened at the polo grounds. that was the home field of the new york giants. tight gang. dodgers pitcher, leans over to jackie robinson in the dugout and says, i've got nothing left

to jackie robinson says you go back out there and pitch until your arm also. that brought to court in effect dodgers take a lead in the bottom of the ninth. then he begins to get tight. gives up his finger. gives up another single. runners at first and third. gets an out. gives up a double. now it is second and third. the manager knows his pitcher is spent. he goes out to get him. he makes a decision on who to bring into the gang. he brings in a guy named ralph franken. his second pitch goes at 60 feet and into left field seats for a three-run homer that prompts the giants announcer to exclaim, the giants when the payment. the focus is on that moment. that's the moment that remains etched in history of the king. that's the moment that if he ever say program talking about key events in baseball, that's always in the top few. that's the consequence.

when you're considering think some a security perspective, the consequence matters. far more important setting yourself up to do without to try to prevent that consequence is how it happened. let's take a look at what he should've known and completely disregard and making that decision to bring in ralph to pitch. bobby thompson was the best hitter in the gang in the 45 gang stretch when the giants came from so far behind in 13800 is average was 110 points higher. he was the prince will recent the giants were playing in the plant. on deck, they had runners on second and third. on deck was willie mays. he's a hall of famer. but willie mays and 9051 was a rookie. in that same stretch, is average plummeted. he was 40 points lower while the giants were so good. his production was way down. ralph had been a starting pitcher when the giants built that greatly but then he lost

seven of his 10 decisions when the giants made a great run and he was giving up more than two runs per gang. most telling o of all, of the dozens of pictures that bobby thompson faced during the season when he stepped up to the plate and the bottom of the night, he had 2010 home runs. six against one man. ralph. dozens of pages he faced a more than 20% of his home runs came against one guy. shirley povich, his headline item the next day in washington writing for the "washington post" and the art of pitching is dead. the thing being what happened at the polo grounds is so inconceivable as a benediction. i submit to you based on what he should've known, he made absolute wrong decision. ignored the intelligence he had developed a what does any of that have to do with what we do here? physical and cybersecurity, ma very often we focus on the consequence. sometimes we focus fully on the

consequence you can lose sight of the niche at your disposal, narrow the likely those consequences will ever come about. that focus can have all sorts of activity that are very reactive but it's important to build a step back and say, what do we have at our disposal? what do we know? how can we take what we have already and put it to better effect to ensure that we're informing preparedness at the right levels? a very important aspect is intelligence and security information to those plan to talk about they do depend upon an awareness that situations are developing that necessitate threat levels. we've got a very good partnership in physical and cybersecurity to the transportation security every shushing fetish would change the dynamic. so we in the industry proposed to tsa a set of priorities for intelligence, both physical and cybersecurity. we focus on don't spend so much time telling us what happened. focus on how it happened. so what is it in the course of,

for example, the london bombing in july 2005? it's important to know what happened on that day. but from a security perspective it's invaluable to know what was going on in the months leading up to that event. where the opportunities to make a difference? tsa responded very well to that and shifted the focus to their analysis to those priority. they produce a product a number of there is on specific terrorist operations that have allowed us to walk through the preparation time and see whether opportunity were effective security measures are committed to this but it allowed us to take that sort of information and bring it further to other constituencies where many of those unobservable indicators that perceived terrorist attacks more likely to be absurd. the likelihood a train crew member is going to see a terrorist enacted committing an act or some form of preparation, is very small. is far more likely a local police officer working in a community will get a report, a complaint, perhaps see something and we want that see something

to trigger a reaction. so as an example in the months leading up to the london bombings in july 2005, there were a whole series of indicators of concern in leeds about 200 miles away. that's what the bombers were holed up. they sublet an apartment and ulcers of events took place that were indicative of the preparations that they were undertaking that eventually came the london bombing. we take that analysis, we delete those indicators, share them to inform training of our employees and we share them with local police to inform their awareness so as they go about their jobs if they get a complaint, there's an odd smell was coming out of his apartment that was not there until these people moved in, or it's pretty odd as happened in leeds that of all the plants outside the window can only these two are dead? it's more audit that they put out this okay filming. they used to dress as devout muslims but out of the blue they dressed in western outfits to any one of those things may not be sufficient.

but maybe they're the sorts of things that will trigger a question for taking a look. that's what we want to be in preparing the similarly on the cyber center it's important to know that they suffer a deficit and cyber attack that disables 20, 30,000 are far more important is how did that happen? what happened that made it even possible? similarly with dhs and tsa we've gotten them to shift the focus to that sort of analysis. help us understand the tactics that were utilized to make that happen. ..