Not so easy, is it?

Deborah Ball, Dean of the University of Michigan's School of Education, chairs the Michigan Council for Educator Effectiveness (MCEE) which was tasked to make recommendations to the Legislature about teacher evaluation. As part of her testimony before a joint meeting of the House and Senate Education Committees, she took some time to make the point that the work of teaching is often misunderstood and harder than most non-educators imagine.

The example she gives comes from fourth grade mathematics - multiplying two two-digit numbers (4.NBT.B.5). First, the teacher must be competent in the content. For example, what is the product of 49 and 25?

From Dr. Ball:

Obviously one wouldn't want anyone teaching third or fourth grade who couldn't do that. But, in fact, what skilled teaching involves is responding when students don't understand the material...

Knowing the mathematics is not enough. Teaching requires more than simply marking students' answers right or wrong. To be effective, teachers must be able to diagnose students' misconceptions in order to provide the support necessary for students to develop mathematically. Dr. Ball provides three examples of possible errors teachers might encounter from students multiplying 49 and 25:

When she asks the legislators how the students might have arrived at these answers, ... watch for yourself (it starts at about 2:30).

Dr. Ball does an effective job of pointing out that teaching is more than knowing and sharing content. However, this example focuses almost entirely on the "upper-half" of the Teaching-Learning Cycle: Assessment and Evaluation. Developing a plan for addressing the student-misconceptions and implementing that plan with a class of fourth graders raises the level of difficulty even higher. I hope these lawmakers now understand that there is more to teaching than what they experienced from the student-side of the desk.

Am I in the way?

I have completed the first four days of the Cognitive Coaching^SM Seminar (CCS), and I am still in the process of consolidating my understanding. In a previous post, I considered the stagecoach metaphor used in the CCS. Here, I want to address another metaphor - the cognitive coach as a mediator of thinking. From page 33 of the Learning Guide:

the mediator intervenes in such a way as to enhance another person's self-directed learning (Costa & Garmston, 2002).

When this is combined with the Four Support Functions (coaching, consulting, collaborating, and evaluating), it requires the cognitive coach to determine what sort of stance he or she ought to take to facilitate another person's thinking.

When we encounter someone with an issue, our typical response is to try to fix it. That is, we get between the person and the problem. To counter this habit, the default stance taken by a cognitive coach is to step back; this provides the person with the space to own the problem and the cognitive coach a sense of perspective. Then the cognitive coach can more easily support the individual in making choices about the path to take and becoming empowered.

Sometimes, the person with the issue is stuck and unable to see any path forward. Given an explicit request for help, the cognitive coach takes on a different stance - consultant. (This is what most people mean when they think of coaching in education.) The cognitive coach steps into the path and provides whatever leadership he or she has to offer. The support might be a model ("If it were me, here's what I might do.") or a menu ("Here is a list of things others have used in similar situations."). As soon as the person becomes unstuck, the cognitive coach once again takes a step back and returns to a coaching stance.

When two people are working together as equals, then the cognitive coach stands shoulder-to-shoulder with the other person. They face the problem together, with each contributing 100% to the effort of obtaining a solution. This support function might be appropriate when co-teaching a class.

In the final support function, evaluating, the cognitive coach stands out of the way and observes the individual's performance. A shared set of standards are used to monitor the individual's progress toward a solution. If the same person is doing the coaching and the evaluating, then his or her role in every situation must be well-defined. Furthermore, trust must be established if this shifting of roles is going to work.

===

Teachers who have gone through the CCS have told me that awareness of these support functions and the ability to move between them improved their instruction. They stress that thinking like a cognitive coach has helped them to focus on supporting the development of self-directed individuals. Mostly, this entails not standing between the students and their learning.

#TeachingReminder: Get out of the way!

Where does it hurt?

Me: (after a fairly lengthy description of open questions) So what did you hear? What are you thinking?

Teacher: (long pause) I don't know what you want me to say.

Exchanges like this happen more often than I like. People have been trained to believe that when a teacher asks a question, he or she is looking for a particular answer. We want to be good students which means providing the expected answer. It reminds me of a book discussion on Diane Rehm about how patients and doctors fall into a routine that can result in mistaken diagnoses and unnecessary tests.

Proud parents with Dr. Hilary at MSU COM graduation

This book came to mind when the teacher responded as she did. You see, I have tried to get in the habit of asking questions I do not know the answer to; it ensures that I avoid falling into an educational rut, and it makes my job as a teacher much more interesting. I tried to explain to the teacher that my question was an attempt to gather some information about what she had attended to in order to make a decision about our next steps. I said something like, "Think of me as a doctor trying to make an accurate diagnosis of your 'health' in regards to understanding of this topic. It's like I'm asking, 'Where does it hurt?'" She laughed and then went on to describe what she had understood about open questions. As the discussion went on, I would stop every now and then to ask, with a smile, "Where does it hurt?"

This is one of the ways we might use warm-ups more appropriately (see last week's post for my perspective on this problem). We could treat it as a way to diagnosis the readiness of the learners to move on or determine what issue we might need to address. (Here is an example of the latter.)

I saw an example of this earlier in the week. Students worked on some scale drawing questions, and after giving the answers the teacher asked if there were any of the questions students needed help on. A student called out a number. So far, so good. However, instead of doing more of a diagnosis, the teacher fell into the rut of showing the student how to find the answer. After a couple of minutes, the teacher stated, "That means the answer is 1/2 an inch. Okay?"

Fortunately, the student responded, "So 6 centimeters, right?"

Now the teacher could do an accurate diagnosis. The problem was not in finding the answer - it was interpreting the answer. "This is 1/2 an inch (holding his thumb and index finger close together). 6 centimeters would be more than two inches (expanding the distance between his thumb and finger). I wonder - were you thinking that 6 inches is half of a foot, so half of an inch is 6 centimeters? (She nods yes.)"

How often do we jump into treatment before doing an accurate diagnosis? I know that it seems inefficient to spend time listening to the student explain "where it hurts," but is it? Drs. Kosowsky and Wen found that it was important for physicians to take more time listening to their patients in order to get their diagnoses and treatments right. The same could be said for teachers.

What's your problem? Part III

Previously in this series, I shared about action plans (here) and how one teacher used an action plan and observation to improve her use of questions in assessing learners (here). In this post, I provide another example - this time focusing on evaluation. (I have explained before that our framework treats assessment and evaluation as different phases of the Teaching-Learning Cycle.)

The teacher being coached was using entry slips to gather data on her learners but was unsure how to interpret the data. Her question to develop her understanding literally asked, "What do I do now?" It was with this in mind that I entered the class and saw the following on the board in the front of the room.

After about five minutes, the teacher collected the sixth-graders' efforts, looked over them, and, satisfied, moved on with the rest of her lesson. 

During a lull in the lesson, I asked if I could look over the slips. With her consent, I began to analyze the assessment data. I organized it using the table shown. As you can see, there were no incorrect answers, however, there was a problem. Approximately 22% of the learners were unable to complete the task in the time provided. Fortunately, the teacher had asked the learners to show their work so that some of their thinking would be made visible to us. This would provide further insight into the problem.

I began to apply the evaluation framework by asking myself, "What can they do? What are they trying to do? What comes next?" Looking over their work, it became clear that they were fluent in comparing fractions using common denominators. Below is an example of how nearly all of the learners went about finding the first answer.

This also seems to suggest where they were approximating. By trying to apply this method (finding a common denominator by multiplying the two denominators) to all the comparison problems, some learners had run out of time. It seemed clear to me that what came next was considering alternative approaches to comparing fractions that might be more efficient. I shared the following list with the teacher:

• Using benchmark fractions like one-half and one;
• Comparing like numerators;
• Finding least common denominators; and
• Converting to decimals.

We talked about ways to introduce these strategies and how to make subtle shifts to the worksheets that she was expected to assigned. By asking the learners to look over the worksheet and match each item with a preferred method, they would be engaging in more meaningful work than simply applying a particular approach over-and-over again. As always, I left it open for this shift to be a part of the teacher's next action plan.

When does it work?

I have been sharing my experience teaching a group of fifth-graders how to problem-solve around fraction computation and using it as an opportunity to demonstrate the Teaching-Learning Cycle in action. Previously, I wrote about how I planned for a problem-solving lesson and then described my instruction during the following lesson. In this post, I want to discuss how I used assessment and evaluation to monitor the learners' progress and inform future planning and instruction.


We were using the clock model as a context for adding fractions and I wanted to gather data about whether or not the learners could determine when this model was an effective approach. I used an existing set of textbook items and asked the fifth-graders to: "Look at the expressions shown below - circle the ones that you think you could use the clock model to solve and place an 'X' through those you could not."

from Scott Foresman – Addison Wesley Math [5^th Grade]

Once the kids had completed this task, I asked them to solve one of the problems they had circled. As they worked, I gathered data on whether or not they were able to determine when the clock model could work.


In analyzing my observations, I noticed what the fifth-graders could do and what they were trying to do. First, they all recognized that fractions involving ninths and sevenths were poor candidates for the clock model. Those who chose to solve #2, #9, and #10 were also fluent in applying prior experiences with the time context. Some learners thought eighths could work (circling #4) and others struggled to see that fifths could work ('X'ing out #3, #5, and #8). These last two areas of approximation gave me some ideas about what to focus on next.


The last assessment I gave was intended to gather data about how the fifth-graders might apply the idea of context to a problem that could not be easily solved using the clock model. As a ticket out the door, I asked, "Now what could you do to solve a problem you put an 'X' through?" Based on my evaluation of these assessments, I was prepared to plan for future lessons.


What would you do next?

How will you supplement your curriculum? Part III

This series began with an overview of the Outside Resources section of the Professional Portfolio developed by each Teacher Assistant. In the second post, Brock, one of our TAs, shared his Outside Resources entry, and I admitted that I evaluated it incorrectly. The purpose of this, the final post in the series, is to explain what led to my error in evaluation and how the exit interview helped me to correct my misconception.

My original evaluation of Brock's work was Progressing. Here is how this level is described in the rubric: "The candidate collects outside resources but they are not related to the mathematics of the unit." Typically, TAs include an article/activity from NCTM that directly relates to the mathematical content of their unit. This is what I expected - my bias. 

When I read Brock's entry, I was surprised to find that he had included the NCTM Process Standards and Cambourne's Conditions of Learning as his Outside Resources. How did this relate to the mathematics found in his polynomial unit? I wrote a note reminding me to ask Brock about his choices and recorded "Progressing" in my NCATE spreadsheet.

About halfway through the exit interview, I questioned Brock about his Outside Resources. He responded that he thought the Process Standards and Conditions were important frameworks that should be implemented everyday in every lesson. My jaw dropped. How could I have missed this so completely? I had fallen into the old trap of judging instead of evaluating.

Based on my prior experiences with the portfolios, I was expecting TAs to produce a particular product - activities that would address some math content. Brock's entry was missing this product and so it was wrong. I had completely forgotten the fact that we had spent the semester discussing how the processes are as important as the content when it comes to math. I had completely forgotten to read Brock's entry with an open mind while using the questions from the Teaching-Learning Cycle to support my evaluation.

Fortunately, the exit interview provided me with the opportunity to be aware of my mistake, accept it, and make the necessary adjustment. Brock's entry was actually "Distinguished" given my new insight. He had gone beyond selecting a single activity that would enhance one topic in his unit. The Outside Resources he collected would impact every lesson, every day, regardless of what or who he was teaching. In his own words: "Including them in this portfolio is a way for me to have a constant reminder of them."

Sorry, Brock. Next time, I will be more open to alternative ideas and stick to our evaluation framework.  Next time, I'll do it better.

How will you supplement your curriculum? Part II

The previous post introduced the assessment used to evaluate Teacher Assistants’ ability to supplement their curriculum with outside resources. Typically, this entry includes an NCTM article related to the content addressed in a TA’s unit plan. Issues only arise when TAs select resources that are either unrelated to the mathematics in the unit or are not from NCTM.

(0) Unsatisfactory	(1) Progressing	(2) Proficient	(3) Distinguished
The candidate provides no outside resources from professional mathematics organizations.	The candidate collects outside resources but they are not related to the mathematics of the unit.	Appropriate resources are collected and the candidate explains how they relate to the content of his or her math unit.	The outside resources selected by the candidate either effectively extend students’ learning of a topic in the unit or provide scaffolding to students in need of support.

This year, one of our TAs, Brock Walsh, submitted the following entry. (He granted his permission for me to share this with you.) I was completely unprepared for this approach since it did not meet my experience or expectations. Consequently, I scored it as "Progressing" - surprise, I was wrong.

Outside Resources

The Process Standards from the NCTM website are a useful tool that I continuously find myself looking for and applying to my teaching philosophy. In particularly, I used these standards during one of my lessons this semester as I put myself at the front of the classroom to analyze a rational function problem on the white-board. I told the students that the purpose of this was for them to see what a good problem solver would look like so that they could reciprocate those ideas when they problem solved in the investigation. Showing the students my thought processes of problem solving, using reasoning and proof, and making connections were all things that I acted out in class for them to see. I then communicated to them what my thinking was and represented my work through graphs and charts. I had so much fun with this lesson and was able to improve on it teaching that lesson three times. I had an action plan set for that class as well looking to increase students’ engagement in problem solving and in the end, I felt that I did a very good job in my objective.

The Process Standards should always be a crucial framework to follow whenever I plan a lesson for my students. I want to make sure that when they are doing math, that they are able to somehow use these standards to show that they are mastering their work. My unit in particular needed a lot of connections to be made to understand new material. The students themselves needed to show me that they knew the material by communicating their thinking to me through formative assessments. We had projects where the students were able to represent their thinking on posters that were hung in the hallway. And lastly, they use reasoning during problem solving to work through problems and generate answers. My students will be successful when they are successfully using each of these standards appropriately.

Brian Cambourne’s Conditions of Learning is another great resource for really knowing how to engage my learners. Being an effective teacher means that I am reaching each student at the appropriate level that they should be learning. Being at Black River, we have high expectations for our students to be responsible for their own learning. In several lessons we allowed students to make approximations before we stepped in to demonstrate the correct way to problem solve. By doing that we gave them opportunity to achieve their own way of understanding the problem, which is powerful when it is accomplished. From the formative assessments, I was able to give students immediate response to their work and let them know how to make corrections to their understanding. All of these conditions truly make students progress towards better literacy of mathematics.

Cambourne’s conditions should be on my mind whenever I lesson plan. I need to know my students and present material to them in a way that they are engaged and are able to progressively develop their thinking. This is a framework that I should keep on hand and refer back to on occasion to make sure that I am reaching my student’s needs.

To be continued...

Whose job is it anyway?

I woke this morning and found this tweet from Alfie Kohn in my Twitter Stream:

It immediately reminded me of the article I wrote with my GVSU colleague, Dr. Pamela Wells - Are They Wrong? Or Did They Just Answer a Different Question?

But it also brought to mind a cartoon John Golden drew for me after I shared with him a talk I heard at a leadership conference at the University of Michigan's School of Education. This was before Twitter, so he had to wait until I drove back from Ann Arbor to tell him about it. I do not remember who the speaker was, but I do remember the message: we need to be figuring out what learners are saying and not the other way around.

So what is your paradigm?

Are multiple-choice tests capable of evaluating learning?

"The writer's [teacher's] job is not to judge, but to understand."

Ernest Hemingway

Last week, I returned to writing about the Teaching-Learning Cycle (T-LC) by considering whether or not we can reclaim assessment from high-stakes tests. Recall that evaluation follows assessment in the T-LC: What can learners do? What are they trying to do? What comes next? Staying with the theme of standardized tests, I want to explore the idea of using multiple-choice items to evaluate learning.

My favorite example of this comes from the 1983 NAEP math survey. I had recently started teaching middle school math when the results for the item shown to the right came out. Only 24% of the national sample of thirteen-year-olds answered this item correctly. Nearly as many of the learners answered (c) 31.33. This is the first time I remember examining the other possibilities and wondering, “What were the kids that selected these other responses thinking?” [This question continues to fascinate me and resulted in this article that I wrote with Dr. Pam Wells.]

Using the evaluation framework, I assume that most of those answering 12 can find the remainder of the division problem and are trying to use it to answer the question. The kids answering 31 might be finding the correct number of buses by estimating the quotient or rounding it to the nearest whole number. And the 31.33 answers suggest that these thirteen-year-olds can do the division correctly but forgot to keep the context in mind. In fact, this focus on making sense of the situation seems to be what comes next for all of those who chose (a), (b), or (c).

You will notice that I did a fair amount of equivocating in my evaluation of the different responses in the previous paragraph. This is the problem with multiple-choice items – we cannot be sure why the test-taker selected one response over the other. Was it based on understanding? Or was it a guess? Or was it an intentional effort to undermine the assessment process? Only by looking at a group of responses can we begin to reduce such errors and be more confident in our evaluation of learners’ thinking. That is why I do not like using multiple-choice items as summative assessments but use them to track the progress of classes as a whole on content we are exploring.

Still, I believe I can improve the question so that it is easier to more accurately evaluate where learners are fluent and where they are approximating. One possible way is to replace 12 (which I consider the most unreasonable answer) with “I’m not sure so any choice would be a guess.” This does not add to my evaluation burden and because the item is formative the learner is more likely to be honest if they know I will use the information to inform instruction and improve their likelihood of later success.

For even more data, do as Karen Bailey suggests (in this PowerPoint) and simply add “Explain” to the end of the item. Some possible rationales are shown to the left. These responses are fascinating and certainly can improve our ability to evaluate their thinking and tailor future instruction to support their continued growth.

It will require more evaluation effort to look through and make sense of open-ended responses and teachers need to decide if it is justified. I once heard Grant Wiggins ask of assessments, “Is the juice worth the squeeze.” But this question should apply to the test-takers as well as the evaluators. Why should we subject kids to an assessment if the task does not truly measure what was intended?

This is one of the first posts that truly did get at my reason for starting this blog. I began this post thinking I would defend multiple-choice items as a reasonable way to evaluate groups of learners from a formative perspective. While I still believe evaluation can be done using a selected response format, I’m not sure identifying what learners can do and are trying to do is as easy as I thought. What do you think?

Are We There Yet?

Assessment and evaluation are often used interchangeably, but the framework that I use when thinking about the Teaching-Learning Cycle sees them as two separate phases. Assessment is data gathering. Evaluation is analysis of data.

I was first introduced to this framework at a Learning Network Conference. The distinction made between assessment and evaluation was new to me and it took a while to get use to it. The intentionality of the split makes sense, however, as it allows me to concentrate on the task at hand. Once data are collected, then I can evaluate what they mean.

The Teaching-Learning Cycle’s perspective on evaluation also required a change in my thinking. As a math teacher, my typical evaluation approach was to judge things as right or wrong.  The Teaching-Learning Cycle reflects a more Vygotskian approach through a series of questions:

• What can the learner do?
• What is the learner trying to do?
• What does the learner need to do next?

I see the first question as representing Vygotsky’s zone of actual development. The second and third address his zone of proximal development.

I came to appreciate this view of evaluation. I think it fits nicely into an analogy for making and using rubrics that I share with preservice and inservice teachers. The idea is that evaluation is like monitoring one’s progress on a trip from Grand Rapids, Michigan to Detroit. A version of the analogy rubric is shown below.

Outcome	Description
Grand Rapids	Because of a lack of understanding or effort, you spun your wheels and never really left Grand Rapids.
?	Where are we? You made some movement out of Grand Rapids, but it is unclear as to where you are headed.
Chicago	It’s obvious you put a lot of effort into this trip, but you went in the wrong direction.  You did not meet most of the intended goals.
Lansing	You’re headed in the right directions, but there is still a lot to do before you reach your destination.
Detroit (but how)	You made it, but I’m not sure how.  There are gaps that make it difficult to follow your path.
Windsor	Whoa!  You overshot the goal. An occasional side trip is to be expected (encouraged even) but be aware.
Detroit	You made it!  You provide a unique, efficient, and insightful route from Grand Rapids to Detroit.
Detroit	Not only did you make it to Detroit, but you were able to connect to ideas from class in order to make the trip memorable.

In the analogy, any rubric ought to represent a taxonomy that supports the evaluator (teacher or learner) in determining where the learner is (what he or she can do and is trying to do) and describing a path toward success (what comes next).

My wife, Kathy, uses a simpler but no less elegant rubric she developed with her first grade learners to support their efforts to engage in independent reading. The learners use the rubric to self-assess and self-correct during reading time. They want to be successful readers and now they have a road map to show them the way.

I think this is true of all learners. Unfortunately, in the past my evaluation methods reflected more of a sorting mentality than a supporting one. My rubrics measured product instead of progress and did not foster a growth mindset.

I have come a long way since then but there is still a ways to go. I would say I am in Lansing. At least now I am heading in the right direction.