Value-Added; a start

Teachers,

I have rounded up a few articles that inform you about 'value-added' systems of evaluation.  One point to clarify, though.  I was slightly off in my general description of the process (last night in class), to save time.  Students' scores compared to averaged students' scores is not the metric; changes in students' test scores are compared to average changes in students' test scores.  So, the rate and direction of change is compared to a mean rate of change.

Sorry to muddy an already murky concept.  As you read, feel free to email questions (or submit them to Ask Us Anything).  Thanks.

***************************************************

Study backs value-added measures (LA Times, with analysis by NEPR, a 'think tank')

Study analyzes efficacy of value-added (Brookings, another 'think tank')

Study looks at effect of 'successful' teachers on students (NYTimes, with analysis by independent researcher)

Class notes, Monday, N19

Teachers,

First, a few research items:

• An example of "post-hoc storytelling" (i.e. confirmation bias?) from Neuroskeptic
• Here is a particularly insightful look at how statistics can be gamed.
• Here is a more subtle approach.

On to Mr. Palomar.  Some background?

Author Italo Calvino presents Mr. Palomar's experiences as a combination of three dimensions

• 1 - the visual or the objective world that multiple people have access to at the same time
• 2 - the cultural or the normative/evaluative world that structures conscious awareness
• 3 - the philosophical or subjective world in which only each person has access

Here is an example.  Kelsie walks into room 101 before class begins.  She is freezing cold.  Andrew, who is already in the room, wears a jacket.  Kelsie asks if she can "turn the air down."  Before Andrew replies, what does he know about,

• 1 - is Kelsie speaking to him?  what happens if the air is turned "down"?  is it physically possible for Kelsie to "turn the air down"? and other issues that relate to the objective world of which both Kelsie and Andrew participate; are they talking about the same thing?
• 2 - is it allowed by the GSE for a candidate to "turn the air down"?  does one candidate need consult with another before taking actions on the room conditions?  what norms guide Andrew's actions toward another candidate or a woman? and other issues related to the normative world that underlies the way Andrew and Kelsie experiences the world of others
• 3 - what does it mean to Andrew and to Kelsie "to turn the air down"?  Is this a question of temperature? one of the white noise of the air conditioner?  is Kelsie sincere and/or serious in her request?  is Kelsie asking Andrew for permission? does Andrew respond to Kelsie in ways shaped by prior experience between them? and other issues that relate to their subjective worlds, or what Kelsie experiences but Andrew does not (and vice versa)

A simple question between two people has a cascade of possible meaning.  The question is simple only if all participants agree on the conditions which make the question simple. These are the validity claims in an argument, statement or question.  Critical ethnography, of the kind you will partake in your research and of which Mr. Palomar undergoes, examines these claims.  It starts with the recognition that communication is a process of negotiated meaning.  To reach understanding, participants - Andrew and Kelsie in the above example - must agree to all the claims made by a statement by constructing a world of shared meaning, before action is possible.  Otherwise, cognitive errors, faulty assumptions, misread intentions and other forms of miscommunication increase the risk of misunderstanding.

Think now of the possible meanings pregnant in at that moment.  One perspective could be that Andrew, having spent the day in a room sweltering at 80 degrees, is delighted about to be in such a cool place.  His students were exceptionally loud this day, and the speaker system broke its volume control so every loudspeaker message came in at 11.  Andrew is delighted to have the consistent low drone of the air conditioner drown out peaks of sound around him.  His subjective experience varies widely from Kelsie's. The simple question sits on the nexus of a horizon of possible meanings.  Understanding the explicit meanings, by examining the three dimensions of validity claims present in the question, takes effort.

Think now of a teacher and student.  Every statement given or question asked by the teacher and student  has conditions that make it true.  These relate to the objective or visual conditions of the statement (what exactly is being stated or asked?), the subjective or philosophical conditions of each participant (what does each think and feel about the statement or question? what might the other think or feel about the statement or question?  how sincere is each party?) and the normative or cultural conditions of each conditions (what background information or culturally relevant meanings determine how each participant makes sense of the statement or question).  In order to restore shared meanings, teacher and student must position-take with the other - try to understand how the other comes at the statement or question.

Here we can recall van Manen's claim about the "asymmetry" in a pedagogic relationship, and a heavy dose of your learnings from Equity:

• Should a teacher assume that all students share all the conditions inherent in the validity claims of her experiences?
• Should a teacher demand that all students learn to accept the conditions underlying her validity claims?
• Regarding the meaning of a situation, what does it mean for the teacher to have a larger responsibility in the relationship?

Mr. Palomar gives us an example of a perhaps more foundational question: what does it require for a teacher research to examine the meaning questions of her practice?

Thanks

November 10, Class Notes

Signal v. noise: "Our ability to tease the signal from the noise has not grown nearly as fast. As a result, we have plenty of data but lack the ability to extract truth from it and to build models that accurately predict the future that data portends ... we are fooled into thinking that random patterns are meaningful; we build models that are far more sensitive to our initial assumptions than we realize; we make approximations that are cruder than we realize; we focus on what is easiest to measure rather than on what is important; we are overconfident; we build models that rely too heavily on statistics, without enough theoretical understanding; and we unconsciously let biases based on expectation or self-interest affect our analysis."

Peer review:
why do it?
what can it do?
what can I get from it?
responsibility of the reviewer?

advantages include:
practice questioning
gaining new perspectives, advice on our writing
potential for self-reflection and analysis
learn about new styles for communicating
adopting the teacher perspective (how to give helpful criticism)
form of assessments
find out what others in the field are thinking

concerns include:
worrying about how comments will be received
uncomfortable
need tact and thoughtfulness

Bracey reminders:

• show me the data
• do the math
• are groups/situations comparable? (the value of context/ecology of schools)
• beware simple explanation for complex phenomena
• correlations are not causations (what else might account for the outcomes?)
• ask 'so what?'

Statistical v. practical significance - p <.05
Even if a study is proven effective, is it worth implementing?

Identity claims

Validity claims

• objective (visual)
• subjective (philosophical)
• normative (cultural)

****************************************************
For our next class, please read the first three episodes of Mr. Palomar ("Mr. Palomar at the beach").   Some suggestions to answer the question,"how should I read Mr. Palomar?" follow.  Please contact me for further in depth discussion.

Focus on how validity claims and truth conditions structure the experience of Mr. Palomar.  Think of his escapade as data gathering and reflection on the data he gathers.  Why does he find what he does?  Consider what conditions - subjective and normative - lead him to see an objective condition.  There will be a constant interplay between which validity claims are backgrounded (tacit, taken-for-granted, implicit) and the conditions that lead Mr. Palomar to meaning.  Help the class poke around with that process to get inside his understanding of the scene.

Focus on the similarity between what Mr. Palomar experiences and your experience in the classroom.  Granted, the comparison will not be exact.  Yet the struggles he has to make sense of his experience do mirror in some way the struggles you may have in coming to understand your teaching.  Italo gifts him with a talent for problematizing his experience.  That skill plays out in every entry.  He turns what might otherwise seem ordinary into a cascading set of questions that fuel his thoughts on the data before him and of which he participates.  He is the ultimate example of a participant observer, making sense of his setting as he participates in it.  Echoes perk up here to

• van Manen: "... tactful acting I demonstrate unwittingly what I can do ... thoughtful reflection I discover what I can do ... my pedagogical nature ..." and to
• Schon: "... they must plunge into the doing, and try to educate themselves before they know what it is they're trying to learn ..."

Remember, the reading is structured to enable you all to make sense of research analysis.  The book is a vehicle for bringing out complexity, of making the invisible visible. What do you learn about perception from each entry?  What rings true in the questions Mr. Palomar asks and the situation his perceptions put him in?  How does examining Mr. Palomar in each entry alter how you process the experience in schools?  What now will you look for or how will you look?

Third Critique Editorial Teams

Teachers,

For your final article critique, you will be part of an editorial team.  Before your writing can be submitted to me (sent 'to print'), it must meet approval from the other members of your group.  When a finished critique comes to me it will have the meet the standards of excellence (why limit yourselves to 'proficiency') of the entire team.  In other words, each separate paper is, in a sense, a group production.

This process replicates how peer review works, more or less, in refereed journals.  As the editors of an issue, you will play the role of determining what writing you feel is worthy of submission.  I look forward to both your work and the discussion of the process.

Here are the teams:
Jessica - Corrie - Jeremy - Zach

Andrew - Kelsie - Rachel - Aimee

Jay - Jacob - Kelly


Please direct all questions and comments to me as they arise.  Thanks.