Cheerleading

Obviously the central feature of education debate in America over the last decade has been student test scores.  The best thing that has come out of this is that the public realizes how broken many of our nation's schools are.  Although, one could say, the worst thing to come out of this is the public perceptions that many of our nation's schools are broken.

The problem is that a distinction is rarely made between broken communities and broken schools.  When a community is disadvantaged, so to will be their children, their students.  These students are harder to teach, and despite the best intentions of an otherwise well-trained, well-qualified staff - who would likely see great success is an advantaged community, the poorer performance of the student body will cause the school to be labeled as "broken".  This inadequacy of language leads to a dismal public perceptions of teaching in general.  Why are so many teachers and schools failing us?

Test scores in general are an adequate picture of the achievement gap in America.  But they are also incredibly flawed.  And when districts begin using them punitively, to punish teachers whose students get lower scores under the assumption that poor scores are the result of bad teaching, these flaws translate into poor evaluation systems, and ultimately unfair labor practices.

Of the many issues with state testing, I'll share my experience with just one.

I teach at a continuation school, so tests are essentially worthless. Kids come from deplorable circumstances, often show up high, many only because their probation officers mandates it. Most do practically zero work in class. So the idea of "trying" on a state test is absurd. Guess what - we're a "program improvement" school.

Anyway, there is a certainly a spectrum of interest in "trying" on a test. You can imagine that the more invested a child is in his education, the more likely he will be to do his best on a test.

A testing coordinator at our school was asked recently what to do about motivation. She replied that there are various techniques for "building enthusiasm" around the state tests. She's correct. The problem is, now we're testing a school's ability to rally student interest in taking a test. Maybe schools of education need to offer classes in cheerleading?

As a blunt tool, state testing is valuable in getting a general picture of differences in school achievement levels.  But there is simply so much nuance, for a variety of reasons and under many different circumstances, that they will only ever be of limited use. 

Instead, we need to be thinking of realistic policy solutions that target real student problems, such as wide disparities in levels of human and social capital.  Not only will this make the profession of teaching more humane and solution oriented, but it will translate into a real, lasting, scalable reduction in the achievement gap.

What Teachers (and Students) Need

A recent hard-hitting LA Times piece on teacher performance in Los Angeles schools has dropped like a bombshell.  Reporters analyzed data from 7 years of student testing, from which they were able to determine specific students and teachers.  They developed what they call a "value-added" model, creating a hypothetical projection from past scores that they then matched to future scores, so that they could compare the two - what students would be predicted to score versus what they actually did.  They then used this as an assessment of individual teacher performance.  In order to compensate for individual student outliers, the results were averaged, pointing toward what they regarded as clear trends over time.

Then they published the results, identifying teachers by name.  While I do have some skepticism regarding the sweeping claims they make, the analysis does seem very interesting and I'd like to see further investigation.  However, some findings in the story raised some questions for me.  For instance, they made this claim:

Although many parents fixate on picking the right school for their child, it matters far more which teacher the child gets. Teachers had three times as much influence on students' academic development as the school they attend. Yet parents have no access to objective information about individual instructors, and they often have little say in which teacher their child gets.

This doesn't make much sense.  We know for a fact that schools in more affluent districts dramatically outperform those from poor ones.  Yet while teachers in poor schools do tend to be less experienced, on average you would expect similar rates of good teaching across districts.  The Times story supports this.  However, the fact remains that poor schools perform worse.  The only possible explanation for this is socioeconomic differences, which has be found time and again to have dramatic effects on student readiness and what I like to call Student Capital - a given student's measure of human and social capital resources that facilitate academic agency.

This site, which maps individual school test score data across multiple states, shows clearly that the number one factor driving overall school performance is socioeconomic demographics.  I'm not sure how the article can say that teachers had 3x the influence on a student as the school.  My guess is that this is a misreading of the data.  A poor student may on his own do better at an affluent school, although I'm pretty sure I've read this benefit is actually pretty marginal.  But in order to properly test such a claim, you would have to take every student at a poor school and swap them out for every student at an affluent school.  Although from what the research tells us, the results just wouldn't be that different.  Not only are the teachers going to be of generally similar quality on average - as the findings in this story back up, but the students are still going to possess the same levels of Student Capital as they did before.

To the extent that teacher quality is emphasized to the diminishment of socioeconomic considerations, we just aren't going to make the kind of progress towards real reform that is necessary to truly making sure that every child is academically successful.

Yet the findings in the story are still important.  And teachers, and teacher unions need to take a deep breath and maintain the conversation.  Jonathan Zasloff, at the Reality Based Community, points out how some of the lessons of a story like this can get lost.  He contrasts the response of an LA teacher to the findings:

Caruso said the numbers were important and, like several other teachers interviewed, wondered why she hadn’t been shown such data before by anyone in the district.
“For better or worse,” she said, “testing and teacher effectiveness are going to be linked.… If my student test scores show I’m an ineffective teacher, I’d like to know what contributes to it. What do I need to do to bring my average up?”

with A.J. Duffy, president of United Teachers Los Angeles, responding to the story:

     The Los Angeles teachers union president said Sunday he was organizing a “massive boycott” of The Times after the newspaper began publishing a series of articles that uses student test scores to estimate the effectiveness of district teachers.
    “You’re leading people in a dangerous direction, making it seem like you can judge the quality of a teacher by … a test,” said A.J. Duffy, president of United Teachers Los Angeles, which has more than 40,000 members.
    Duffy said he would urge other labor groups to ask their members to cancel their subscriptions.

Zasloff finds Duffy's response as emblematic of teacher union intrangigence:

If progressives want to reinstitute faith in government, then we must demand the best possible results from public institutions.  And we also need to confront directly dinosaurs like Duffy who simply refuse to accept any accountability for his profession. 

I think Zasloff makes a good point. But he also has to acknowledge that teachers are under heavy attack right now, and that in such an environment people tend to get suspicious, reactionary and acrimonious. Much of the liberal establishment seems to have turned on us as well, and are pushing reckless policies without listening. But I agree with you, as a teacher the number one priority I have – and I know this may sound shocking – is that my kids are actually learning. If a standardized test can help facilitate this, then I’m all for it.

Of course, one of the big issues is, as he points out, being able to drill down and separate out the causal factors. Largest among them, yet routinely ignored, is the incredible difference in teaching environment that two schools in the same district might hold. What this means for teachers is that not only is performance going to be much different owing to demographics, but any progress made will have been much more hard won at one school versus another. This one single problem is something that neither NCLB or Race to the Top have dealt with in any serious way.

So while I completely agree with the teacher, and have a similar reaction to AJ Duffy’s response, I also know that he is on the front lines, trying to do the heavy lifting that is protecting teacher’s genuine and reasonable interests in an increasingly hostile policy environment. I think the type of testing in the story sounds promising, and it ought to be looked at more closely.

What everyone needs to remember is that what the modern public school system is trying to do is nothing short of revolutionary. It is essentially asking generational poverty to be broken on the backs of teachers. And I love this. This is why I became a teacher. But if we are really serious about doing this, we need to take a moment and look at what we are trying to force the system to do. Some teachers at poor schools will be achieving amazing things. The teacher next door may not be. But she may actually be just as competent as the teacher a few schools over who doesn’t have to deal with nearly as many issues. The reality is that you just can’t expect to put relatively equal resources into schools with wildly different demographics and expect every teacher to be amazing enough to compensate. It isn’t fair. You can’t build a transformational system around the idea that every teacher in a poor school has to be amazing.

And I guess that’s kind of the rub of how teachers are being treated today. We signed up to do this job because we wanted to help, and a certain amount of sacrifice goes with that. But it feels like society in general has seen so many Jaime Escalante movies that they think seem to think if we all aren’t working 14 hour days and coming in Saturdays we aren’t good enough. Maybe we shouldn’t have to, right? Maybe society ought to invest a little more money in its poor schools so that the job requirement isn’t super man, but maybe average man.

Imagine if we approached other public sectors this way? What if wars were won, streets were policed, fires were put out, mail delivered, etc. by spending as little as possible (indeed not enough to effect real reform), then complaining when the job wasn’t getting done that the workers just weren’t doing their jobs?

If teachers are going to be expected to get on board with looking at increased accountability, they need to feel like there's an equal policy response that addresses their legitimate concerns about the equity of the task they are being asked to perform.  Teachers wouldn't be in the profession, already making they sacrifices they do, if they didn't want every child to succeed.  And if you give them the respect they deserve, by acknowledging the complexities of their work and providing them the resources and tools to be effective, they will back you 100%.

Merit Pay is Nothing New

Merit pay, or pay-for-performance, along with charter schools and standardized testing, is one of the key platforms of today's education reformers, among them senior Obama cabinet officials like Arne Duncan.  As I've written about earlier, even taken as a group these so-called reforms are laughably inadequate to truly address the fundamental problems in American education.

In order to do merit pay, you need to have data on teacher effectiveness.  But therein lies the problem: teacher-effectiveness models must be treated with extreme caution.  In order to be fair, they must be accurate.  And how you get accurate data is a problem that has always plagued the concept.  Salary rates are currently not based on effectiveness for precisely this reason.

I was recently flipping through a handbook published in the early sixties for beginning teachers. It was an odd book really, covering everything from pedagogy to budget considerations. The references to segregation were pretty fascinating. But in a section on salary it outlined today’s debate perfectly – stating that the difficulty in pay-for-performance comes in developing an accurate model for what that effectiveness really looks like.
Merit pay be eventually be done right. Although if it isn’t I won’t be too concerned – teachers don’t do it for the money. But the idea that it is going to be an effective element of reform is naive.
That we’ve been having this debate for half a century tells you A)the concept is nothing new and B) it’s a complex issue with no clear solutions.

Student Success vs. Patient Survival Rates

Man, the education debate is depressing. Not only is teaching often one of the most difficult jobs one can do - but you have an endless parade of politicians and pundits heaping criticism on your profession after demonstrating again and again that they possess an embarrassing lack of knowledge.

As part of the Obama administration's race to the top, states are being encouraged to use pay-for-performance schemes, a hot concept among current education reformers. Many believe it represents a chance to dramatically alter the nation's troubled schools by providing a sort of market-based, competition-driven prescription for success. If we could only just look at which teachers were doing a good job, and compensate them for it, we would create an atmosphere that rewards success and punishes failure. The process has been likened to other industries where monetary rewards are based on outcomes. Mayor Bloomberg of New York city recently pointed the finger at detractors:

In New York, the state legislature passed a law last year that actually tells principals ‘you can evaluate teachers on any criteria you want, just not on student achievement data.’ That’s like saying to hospitals ‘you can evaluate surgeons on any criteria you want, just not patient survival rates.’

Unfortunately, while the idea is not without merit, it is highly problematic.

Comparing teacher evaluation to patient survival rates is absurd. It displays a fundamental ignorance of basic issues. One is reminded of those who would make generalizations from government spending to household spending. But hey - its just common sense, right?!!

For starters, "patient survival rates" imply highly controlled situations with clear parameters and guidelines to follow. From Wikipedia:

"Relative survival is calculated by dividing the overall survival after diagnosis of a disease by the survival as observed in a similar population that was not diagnosed with that disease. A similar population is composed of individuals with at least age and gender similar to those diagnosed with the disease."

Teacher quality is certainly important - why wouldn't it be? But the problem is in drilling down how to precisely determine that in a way that scales up, allowing districts to set policy. But in any given classroom, the variables effecting student outcomes are immense. Any teacher who has taught at multiple schools with different demographics knows that some classrooms can be orders of magnitude more demanding than others. How then to determine which teacher s performing better?

For example, at school A, the 10th grade Biology class is reading at or above grade level. Most parents can be counted on to enforce academic performance both in and out of class. There are few absences. Students are generally well-fed, groomed and comfortable. Resources are available for field trips (thanks in part to active PTO fund-raising), and most students have access to high-speed internet connections.

At school B however, the 10 grade biology class is reading at the 6th grade level. Few parents can be counted on to enforce performance, many don't return calls or numbers have been disconnected. Absences are frequent, as well as tardies. Students come to class unprepared, nervous and potential for conflict is constant. There are not enough books to send home - and few would be read (or even returned) if they were. There is no money for field trips, requiring the teacher to raise money on their own time. Maybe one or two students have access to a PC at home.

So how would one come up with a formula that would apply to both situations? As any of these variables could rise and fall day to day, how would a results-based compensation system be designed that would be as fair to teachers at school B, as is is at school A?

When you can begin to legitimately respond to these questions, you are ready to enter this debate. Until then, you need to do some more homework.

Standardized Testing = Yugos vs. Ferraris

So, as a requirement to get my credential to teach Secondary Science, I'm taking an online methods course from the UCLA extension. So far its been really fascinating. This week's reading has been on the subject of standardized testing.

Another difficulty stemming from the profusion of curricular aims that customized standards-based tests try to assess is that the tests themselves can’t possibly include a sufficient number of items to shed light on a students’ status regarding the curricular aims that do make the cut in a given year. More often than not, there’s only one item, or perhaps two, to assess students’ achievement of a specific curricular aim. Teachers really can’t get an accurate fix on a student’s mastery of a curricular aim based on only one or two items. Typically, the developers of customized standards-based accountability tests report students’ performances at more general levels of aggregation: amalgamations of students’ performance with respect to collections of a variety of curricular aims, some of them very different. This is what you see when accountability results are reported at the “strand” level or at the “content standard” level. But this kind of reporting obfuscates which curricular aims students have mastered.

Excerpted from “Transformative Assessment,” by W. James Popham, Copyright 2008, Association for Supervision and Curriculum Development, Alexandria, VA.

I'm not sure if I did this right, so someone correct me if I'm wrong. But what I did was create a hypothetical sample class of 10 students, (A-J). I showed the results of 10 concepts covered in class, and their respective true performance on each. They ranged from one student (A) getting 90% correct, to another (J) only getting 50%.






























(click on image for larger view)
But as the reading pointed out, there is simply not enough time to test for every concept, and so they sample. Even if they try and target important, representative skills, you're still stuck with the problem of getting a poor reflection of what was taught in the classroom, much less that student's particular knowledge.

In my sketch, student B happened to miss one of the sampled concepts, and so scored a 50%, even though he was 80% mastery over all. And student J, only 50% over all, happened to get the sampled concepts correct, and so was scored 100%.

The author goes on to argue that what standardized tests end up showing is a student's SES (socioeconomic status) and inherited aptitudes. Which would explain high levels of correlation between test scores and demographics. To me this is basically what NCLB has highlighted, by making such a huge deal out of test scores with the public: that there are wide gaps in academic achievement across demographics.

Unfortunately, the take away for many here is simply that teachers are doing a terrible job. All you have to do is look at test scores as a measure of teacher performance and it looks like you have some really bad teaching going on. But of course the reality is that there are numerous other factors involved, and that while bad teaching is bad teaching, it is apples and oranges. I liken it to measuring the performance of two race car drivers, one in a Ferrari, and one in a Yugo. This would seem a terrible way to describe students (as cars), but what we are really comparing is the craftsmanship and materials used to create the individual. A student can be made into a Ferrari or a Yugo over the course of his life. But once he gets to the teacher, most of the raw materials are in place, and there is only so much the average teacher can do to get him up to speed.

Just a thought.