Was the scoring consistent?

The following is a statistics workshop I have used in a class for preservice K-8 math teachers. Because the topic being addressed, Mean Absolute Deviation (MAD), represents a specific procedure, parts of this workshop are more direct than what I typical do in a lesson. Still, I try to leave enough space for learners to explore the procedure and make sense of it.

The goal of this activity is to consider how to quantify consistency while addressing the idea of inter-rater reliability. It is based on the article Means and MADs and is intended to help introduce the concept of deviation using a measurement that is more accessible to middle grade learners. I updated the activity to reflect the upcoming Common Core State Standards (CCSS).

Grade 6 CCSS in Statistics and Probability [6.SP]

• 6.SP.2. Understand that a set of data collected to answer a statistical question has a distribution which can be described by its center, spread, and overall shape.
• 6.SP.3. Recognize that a measure of center for a numerical data set summarizes all of its values with a single number, while a measure of variation describes how its values vary with a single number.

Schema Activation: Turn and Talk

With your partner, discuss which set represents the greatest consistency.

Focus: Quantifying Variation

Consistency is a major concern when it comes to grading.  We want the scores that different teachers assign a piece of work to be relatively close. While sometimes variation is obvious, quantifying it can be helpful in identifying inter-rater reliability. In this workshop we will learn about a particular measure of variation called the Mean Absolute Deviation that can be used to describe how values vary with a single number.

Activity: Small Group

The table and graphs below show the scores one group of teachers assigned Ross for his exploration project in each Math Thematic assessment category: Problem Solving (PS), Mathematical Language (ML), Representations (R), Connections (C), and Presentation (P).

PS	ML	R	C	P	Total
3	4	3	5	5	20
5	4	5	1	4	19
3	4	4	5	5	21
5	4	4	5	2	20

Predict

• In which category do you think the scores were most consistent?
• In which category do you think the scores were least consistent?
• Which representation (table or graph) was most useful in addressing these questions? Why?

Determine

Find the mean distance (direction does not matter – why?) each data point is from the mean of the data set and compare it with your predictions.  Which representation (table or graph) was most useful in finding this Mean Absolute Deviation? Why?

Reflection: What? So what? Now what?

1. What were your results?
2. So what do the results say about consistency?
3. Now what are some ways to improve scorers’ inter-rater reliability?