How compelling was that single study?

Dear Editor,

Ever since Chancellor Fariña began advocating for using aspects of balanced literacy in New York City Public Schools (New York Times, 6/26/14), she has come under fire.  While her only action thus far has been to ask Lucy Calkins, a proponent of a balanced literacy approach, to do a seminar in August, the critics have cried foul. They argue that Chancellor Fariña is ignoring the results from a 2012 study of a pilot program called Core Knowledge Language Arts [CKLA] (New York Times, 3/11/12). Given the information provided about the study, however, these critics are seemingly misrepresenting the results.

Here are some of their arguments.

Daniel Willingham (on the Board of Trustees of the Core Knowledge Foundation) on 6/30/14

Three years later in 2012, the results were in: the kids in the Core Knowledge schools were reading better. The advantage was indisputable. 

Fariña was reminded of these results last week: how could she justify calling for a return to balanced literacy when another program had shown superior? 

Robert Pondiscio (worked for Core Knowledge) on 7/3/14

Yet the Education Department’s own three-year study comparing Core Knowledge with balanced literacy in demographically matched New York City schools showed the former “had significantly stronger gains than comparison school students on nearly all measures.”

E. D. Hirsch, Jr. (founder of the Core Knowledge Foundation), on 7/3/14

The New York City Department of Education recently did a three-year study comparing 20 schools. Ten used the Core Knowledge approach. Ten used balanced literacy. After analysis, the Core Knowledge results were deemed to be far better, to a high level of statistical significance. 

Alexander Nazaryan on 7/6/14

Ms. Calkins’s approach was tried by Mayor Michael R. Bloomberg, but abandoned when studies showed that students learned better with more instruction.

New York Daily News on 7/6/14

A 2012 a study found that a small group of balanced literacy schools lagged behind schools that used what’s known as Core Knowledge, ...

Michelle Gininger on 7/9/14 (at 9:51 into the podcast)

[The Bloomberg Administration] tested things out and saw, at least in that limited study, that Core Knowledge was working better for low-income kids ...

Whenever someone states, "the study shows," related to something as important as educating our children, please be wary. Our immediate response ought to be, "Where is the research? I would like to examine it for myself." In this case, many of the critics provide only a link to a New York Times article on the study from 3/11/12. It describes the study thusly:

For three years, a pilot program tracked the reading ability of approximately 1,000 students at 20 New York City schools, following them from kindergarten through second grade. Half of the schools adopted a curriculum designed by the education theorist E. D. Hirsch Jr.’s Core Knowledge Foundation. The other 10 used a variety of methods, but most fell under the definition of “balanced literacy,” an approach that was spread citywide by former Schools Chancellor Joel I. Klein, beginning in 2003.

Based on this description, it seems unlikely that the study did a direct comparison between balanced literacy and CKLA. After an unproductive search for the actual study, I only found a PowerPoint on Core Knowledge Foundation's own website sharing the results from the CKLA Pilot. This is how the Core Knowledge Foundation described the study:

There is nothing about balanced literacy in this description. In fact, there is nothing about balanced literacy or Lucy Calkins in the entire PowerPoint.

If there is more to the study that supports the critics’ claim that a direct comparison was done between balanced literacy and CKLA, then it is up to them to provide that information. It is not enough to simply say, “a study shows.” Without the actual study to anchor the discussion, arguments can drift far from the actual results.

For example, given the history of balanced literacy in New York City Public Schools, perhaps the critics jumped to the conclusion that the comparison schools were using balanced literacy. Therefore, in their minds, CKLA was compared directly to balanced literacy. While this history might be enough to support their opinion in casual conversation, research conclusions require confirmation. And no evidence was provided that confirms that the comparison schools were using the instructional approach described by several of the authors under the banner of balanced literacy.


Every study, and anyone who wants to appropriately cite that study, is limited by the study’s design. The design of the CKLA Pilot Study limits how we can use the results. If the critics are serious that they want Chancellor Fariña to use data to inform education policy, then a new study ought to be conducted. That is what the International Reading Association suggests:

With greater care in the design, analysis, and reporting, these shortcomings could have been resolved. If we could have confidence that the gains that students in the Core Knowledge schools showed on assessments of reading comprehension, as well as assessments of science and social studies knowledge, could be attributed to the resources that were introduced, then there is cause for celebration. The Core Knowledge program could, in fact, be an important element in a truly “balanced” literacy curriculum. 

To his credit, even E. D. Hirsch, Jr. supports this idea. At the end of his opinion piece, he writes:

But I agree with the schools chancellor, Carmen Fariña. The study was too small. We need a bigger one – and one that gauges long-term as well as short-term effects.

In order to conduct the study the critics want, a direct comparison of balanced literacy and CKLA, some New York City Public Schools are going to have to adopt Lucy Calkins’s instructional approach. Simply adopting some aspects, as Chancellor Fariña suggests, will not be enough. Dr. Calkins will probably even have to conduct professional development workshops for New York City educators. Ah, the irony.

Let me end on a serious note. If we are committed to doing what is best for students and using education research to inform these decisions, then we need to start having serious discussions around the data we collect. Using a single education study to push an agenda or bash a potential competitor is irresponsible, as it often ignores other important results. Intentionally misrepresenting the study in an effort to suggest that our children are at risk is nothing short of propaganda and has no place in the discussion. Let’s make “the study shows” the beginning of the conversation, not the end of it.





Sincerely,
David Coffey
Professor of Mathematics Education
Grand Valley State University
Allendale, Michigan