and then those -- that logic model builds the program performance measures chart. we use some of the outputs for the comptroller passport. -- the comptroller's report. the evaluative measures are more apparent a piece to the measures. the data collection method needs to be established by each program to ensure that date that

is relevant. that speaks to some program goals. the next one, we actually come across barriers. it is a question of how this is impacting the students. that is setting the stage for evaluation that needs to be in place for what i call research questions to be addressed are examined. lastly, once those are all in place, impact can be assessed.

the second page, the page with the arabs -- that is pretty much the building blocks. i like to refer to them as layers in order for impact to actually be measured. the third sheet, with the five different points -- this is an example of a program, athletics, but i just used, because this is sort of the rundown of the five steps. it runs down the details of the five steps. president mendoza: i am sure

everybody has questions. commissioner wynns, the you have this material in front of you? do you have questions? commissioner fewer? i am sorry. this is clarifying questions before we have public comment, so just clarifying questions. commissioner fewer: i do not see in this packet a summary or a report on the impact of the funding on the spending. is that not included in here? >> i think there is a link to our data center for all the

impact. we sort of ran statistical analyses and ran more formal analyses. commissioner fewer: step five is analyze and summarize impact. what we had been asking for on the board for a couple of years. that is in a limit to our strategic plan. i have never seen any analysis or summary of the actual impact that $30 million has every year in reaching our plan. i think this is what other commissioners had said at the last meeting.

i expected at this meeting to see some of that. we have asked for it, and we have asked for it from the board. we have never received it. i think the other commissioners, and correct me if i am wrong, have not seen that impact either. we approve this plan every year and we do not know the impact. we are coming into a horrible budget year. it is the impact and evaluation that allows us to approve $30 million every year with a clear conscience. that is taxpayer money, hard earned. it must be expensed the right way. i am a little concerned that we do not have that in front of us.

i wanted to know -- before they made these recommendations, did they see any summary of analysis of the impact and how we are meeting our district goals of access and equity to student achievement and accountability? did you actually see that and then make those recommendations? >> the outcome measures -- it measures different strategic plan concepts like student engagement, joyful learning, and access inequity. that is another goal. some of those outcome measures are described and have been

listed on the performance measure chart for each program. commissioner fewer: does that mean the cac were able to see all this information before they made the recommendation to the superintendent? >> it is available for them to get that background information. commissioner fewer: was this a topic of discussion before the cac made their recommendations to the superintendent? >> i am going to defer to my colleague, cathy fleming, who supervises those meetings. she will get back to you in detail. >> cac has at almost every meeting had a discussion on impact and evaluation, and has requested information about performance measures, and has

received logic models with data, with performance measured charts, from the beginning of the programs. but i do not recall specifically what was shared with them beyond that. commissioner fewer: are you saying to me that they did not receive the performance measures? they saw the logic model, which is what i saw when i was on the cac. we have not progressed beyond that? the cac has said for many years that if you want us to make a recommendation we have to see the data on the impact $30 million has on our district goals. i think the board has also said that. i want to say right now that it is hard to approve this without data to back it up, because every year we just repeat what we spend and we do not have any impact at all on whether or not

it joins district schools. we have had a program by a waiter for the last four years. i would -- we have had a program evaluate foor for the last four years. i would expect the report. it is hard to say ok on this when we are looking at 2014 having to go back to the taxpayers and say "will you please reauthorize this charter amendment to give us the money again when we do not have hard data to show the impact it has had." i just want to say personally that i expected to have much more data in front of me, because we requested it at the last meeting. the website does me no good right now and i cannot download it right here.

if it is volumes of information, in no way am i an expert to analyze that. we rely on staff to give us that information. i want to point out that we have come at a rate of $105,000 a year, funded a program to evaluate her for this program. today, i am very disappointed they do not have that information in front of me for me to make a sound vote on this proposal. president mendoza: thank you. vice president yee: i must say i am a little confused. i am almost in agreement with commissioner fewer. at the committee as a whole meeting, we talked about needing to see some evaluation on this and what is the process, and so forth. we asked a lot of questions on this. we really did not get much of an

answer. i specifically at the end of the discussion said, "if we do not have this in place yet, we should at least within two months come up with a plan of how to perform an evaluation." i think at that point somebody should have said, "we have some data. we have some evaluation." this is the first time i heard it was on-line anywhere. for us to look it up right now and figure it out seems a little -- i totally agree with commissioner fewer. if the information was there, why wasn't it given to us? if we had the information and we asked this question is, why didn't somebody say we had it? it is pretty annoying that it was not a good presentation. i am inclined not to support this because either there is

shortsightedness in terms of presentations, or there is some reason for not telling us that there is information. if we have this, great. i am glad we are ahead of the game and have evaluations. but that is not what we heard last week. so i am very upset with this. president mendoza: let me remind my colleagues that this is clarifying questions. you can share your comments after we have public comment. are there any other clarifying questions? i am going to wait for everyone else to hold other comments until after we have public comment, if we have any public comment. is there anyone that wanted to make public comment on this particular item? ok. commissioner wynns, did you

have any clarifying questions? commissioner wynns: i appreciate what the other commissioners have said. i hear their frustrations. one of the biggest problems with this is that we have said from the beginning -- i think we should think about what would be a logical way to do this. the budgeting process is, from the school district point of view, my impression has always been that what you are asking us to do is to basically report a status quo budget for next year. but it is really just a proposal, because then it goes to the board of supervisors, and then we start negotiating. while i agree that we should be far more constant in our own ability to signal a good

decision and -- a good data- based decision this evening, i would like for there to be forthcoming to us, not necessarily this evening, how the process could change or get better, or if it works for us. that is kind of my frustration with the budget as it is being proposed to us. it basically says, "for purposes of this budget and beginning the outline process with the city, we are proposing to roll over all the programs we have." given the requirements for the ability of the city to not give us all the money, that is a problem. i do not think that was clarified or clear about the steps in the process and whether it works for us the way it is now. president mendoza: thank you. there was no public comment, so

we will keep going around. student delegate, did you have any comments? commissioner martin -- norton? commissioner norton: i share some of the frustration that has been voiced tonight about the need for us to have a clear and were established format for how we will evaluate these programs. some of this description makes me feel like -- what measurements are we using? it just feels a little bit haphazard, i guess, based on what i am seeing. but the other thing that i think is even more troubling to me is that cac has note -- i know that they are meeting regularly and have done work. but there were not at our meeting last week and are not here tonight. that is troubling. it makes me wonder how engaged -- these are our appointed

representatives from the community that we are supposed to be relying on for their explanations. if they are not there, what that is saying to me is that for some reason they feel disconnected from this process. there is not -- but never work they are doing -- whenever work they are doing, if there were proud of it and felt it was important to us, there would be here to show it to us, and they are not. that really bothers me, and i would like to know more about that. i will talk to my stuff about why that is. i would hope there would be representatives from the cac here. the other issue i have had, and i have spoken to you about this privately but wanted to bring it up publicly, it is the whole issue of the carry over. i understand there is some good reasons for that. i do not think the decisions that were made, the various policy decisions, the issues for

why we have a carryover are necessarily a bad thing, but it does trouble me that almost 10% actually was carry over. it went to other things than what we originally approved when we approved the spending plan. i suggested to step earlier that the board needs a little bit -- to staff earlier that the board need a little bit of progress reports that some of these things are going to change. we are in a complex environment. king's costs less or more than we thought they were. people aren't available, or whatever it is. but the board needs a progress report. this money you have budgeted for this use is not going to be used for that this year because we made a policy decision to do something else with it. that would be sufficient to me. but looked at the plan and look at what we did last year.

a million dollars we thought we were spending on one thing is going to something else. that is bothersome to me. i think what commissioner wynns said about the way the budgeting is working -- it is not working for us. i think this discussion, although it feels a little bit painful because we are voicing frustration with staff, i think it is healthy. hopefully, it can be the basis for figuring out a better way to do things in the coming year. commissioner murase: i want to thank you for the side-by-side comparison of the previous year's proposal to this years. that is very helpful. in terms of the outcome, i do not see what it was we requested at the committee of the whole. i think one statement that comes close to it on one of the handouts is the question "is there is a significant

difference of secondary school students all attendance rate in 2008 between students who participated in the clinic and students who did not?" we are talking about a $30 million intervention, $30 million. we should not be asking it just participation in athletics makes a difference. what is the difference of $4.80 million to that? we want a comparison. what couldn't we do without this money? we really need to have a system statement of what difference is this money-making to the strategic goals that we have to student achievement. i hope this gives you a clear direction in the kinds of outcomes we are looking for, the statements we are looking for.

i think it is great that in this example it says those students who participated in athletics had better attendance. we want to be able to make the case that this money is essential to athletics, and therefore better attendance. finally, on the carry forward, i think we have a lot to answer for, $1.10 million. that is a lot of money that was not plowed into these programs. the causes of that are identified as overestimated fte benefit costs. that is an awful lot of estimation. i think that a lot of refinement can be done. president mendoza: thank you, commissioner. >> thank you. commissioner maufas:

commissioner chang? i would like to request of dr. khana come forward. i think you are unfairly -- not that the doctor should take a beating either. i think you are unfairly being browbeaten. this is the first time i am looking at you. you should not be taking the brunt of this. that is my perspective. it should be somebody who is getting more money to take the beating. [laughter] that is what it is. i have to say, really you are looking at the end of the rope for diplomacy. i think in the plainest way that the board can put it, we have asked for certain things. and what it feels like is the runaround. it feels like the runaround.

to be here today, asking for a vote, when we cannot get very basic information -- it seems like the runaround. if we put you off long enough, we have deadlines, and we have to make them and get over to the board of supervisors. whether that is the case or not, that is what it feels like. and i do not believe that our deputy superintendent and our superintendent want you to do that to us, as the board of education, because we have to go up there and get a beating as well. that is what happens. we go outside of this board room and take a beating for the money that is spent, a beating for the money we are not spending on other necessary items, and we cannot account for it. even off the top of my head, i

know i have asked for these members to be here. why they are not here today -- i am baffled. that is basic. that is really basic. this is one of the rare times they can speak to the board of education to report the work we have appointed them to do, and they are not here today. that is another level of the runaround. if i go to my appointees and they tell me they did not get notice, that is not right. these very basic things need to be covered by you all. let me ask, dr. connor. is this a proposed process that we have in place that we are now looking at? >> thank you for letting me speak and defend walter, because i do not think it is right for

him to get, you know, come to the first board meeting and take such a beating. the key here is walter has been religiously producing the comptroller's report, which is what he was told was all he needs to do. so the comptroller's report was that this was the inputs that go into the budget, and he has to collect from every program manager the outputs on those activities. when were they done, and what happened? he did push for evaluation to occur, at which point they had to try to come up with the evaluation measures. most of the programs, as we went and looked at them, only had process measures. they did not have outcome measures that we could go to. at that point, we started to set the stage for evaluation.

that means you have to first define the outcome measures. and so the outcome measures still continue not to be defined. we worked with criteria. now we talked about criteria in order for us to evaluate. we need to have four basic things. that is when the criteria went up. we assessed every program on the criteria. the athletic department did meet the criteria. we were able to look at student impact with the athletics department, because they had criteria. it is easy for us to crunch numbers. if we had numbers of participation, we have all the data. it is very easy to do statistically. but the key is for the programs to define this outcome measures -- these outcome measures.

it has been difficult. commissioner maufas: ok. it should not be difficult at this point in time, deputy superintendent, mr. armitrage. the you have something to say before i ask my next question? i am listening. >> just to respond to your comments regarding the carry over. within that 9.2%, you are right. that is $1.17 million. but it is important to point out for the record that we are, as a district, conservative in how the budget. for example, we place budgets at the highest step. when we calculate benefits, we also are conservative about that, because we are not sure what retirement costs are. it is better to air and have

extra than the reverse. that is our district policy. there are usually adjustments made to reflect that. our policy, given the state of the budget crisis and our already dedicated reserve, it was a policy decision to not allow that to be redirected within the program. in terms of what we promised, programmatic, speaking to the inputs, and not the outcomes, which is exclusively rpa's domain, in terms of what we promised, we met our goals of what we said we would do. there was a total promise of 85 full-time positions.

i am sorry. 86 that we said we would do. by the end of the year, all but one of those positions was filled, which was an academic coaches, which was eventually discontinued. the remaining is about three to five positions toto, the word, for various reasons, people on maternity leave, things delayed on hiring. in terms of program goals, that was essentially achieved. that is not to dismiss any of your concerns for the important of this -- the importance of this or how this needs to be as transparent as possible, or how critical this has been to the district and how much we need to be accountable. this is probably the one program within the district that has to be the most accountable, given the structure we have in the city. that is driven home. we get it. we really do. i think that as commissioner

norton and i had spoken about prior to the meeting, we do issue our quarterly report and show our burn rate of the funding. we are able to give a more augmented report to the board to keep you informed and make sure that you have that, so you are empowered to be able to speak with us and are kept informed. i appreciate how important it is an want to to understand we take it seriously. commissioner maufas: let me understand something for clarification purposes. this is directed to all of you. what you are saying is, in regards to the outcome and the ability to have access to that information -- you are saying to me that is something that is in sort of the silo of the research.