StickWithANose

Without my having been aware of it, a generational shift in educational discourse appears to have occurred. The education establishment has finally admitted what academic research has consistently said for nearly two decades. EdWeek:

Multiple-choice items can efficiently discover whether a student has assembled discrete pieces of knowledge across a subject. The results are also typically highly reliable, meaning the error associated with the results is low—a desirable quality for high-stakes tests. And they are are easy and cheap to score.

Such tests, though, are not ideal for identifying whether students can take multiple pieces of domain-specific knowledge and analyze, integrate, and apply them in unfamiliar contexts, Mr. Bennett said. And researchers familiar with international benchmarking argue that those critical-thinking skills are precisely the type that will be in demand as the global economy becomes increasingly knowledge-oriented.

Holy $#@^!  Could it be that the useful tools of the education lobby have looked into their cold, hollowed-out souls and discovered a genuine concern for student learning? Or is this a classic head fake toward a repackaging of the same old garbage?

What now seems to be an intractable choice between richer tasks and reliable data, though, could be mediated by advancements in technology that could improve access, cost, and reliability of performance-based testing, some experts argue.

And the federal funding, they say, could be the lever to support that work…

The high costs of scoring such a complicated assessment with an almost unlimited number of answers, he added, could be mitigated by advancements in natural-language-processing software­—essentially programming that proponents claim can judge written essays as accurately as human readers and reduce, though not eliminate, the need for costly human evaluation.

In addition, experts say, technology offers the ability to measure student understanding of concepts and processes involving critical thinking that have been notoriously difficult to assess using only multiple-choice items.

Looks like option B: repackaging. There is a long history of snake oil salesmen promising that technology is going to fix all of our education needs, but there is a fundamental truth about education that haunts policy-makers and allows the snake oil salesmen to prey on them. Education is a human process that is individual, complex, and not easily measured in a mathematical sense.

Constructing authentic formative assessments is not a mystery. There is a wealth of research on the subject.  However, it is expensive and time-consuming. It requires human judgment, and it doesn’t produce easy to read numbers for politicians to point toward come election time. What it can do is provide real-time learning assessments to both teacher and student that helps students gauge their own learning and teachers adjust lessons and curricula to meet student needs. It can also produce rich data on student achievement that offers teachers, administrators, and policy-makers alike a more nuanced perspective of macro-level trends. In short, authentic formative assessments can provide what standardized, multiple-choice exams cannot.

To be clear, I’m not against the use of any one form of assessment a priori. I am against the use of one form of assessment to the exclusion of all others… even if it is re-packaged as Assessment 2.0! Now 30% more authentic! There is a genuine need on the part of policy-makers and administrators for data. The question is what kind of data do we want and how can we get it without breaking the bank?

Here’s my plan: Instead of mandating annual assessments for all students grades 3-8 that yield low-quality data, policy-makers should mandate authentic assessments* for those grades that use random-sampling techniques for scalability. Instead of giving everybody a crappy test, we should try administering authentic assessments to smaller samples. The higher costs of authentic assessments would be off set by using tried and true techniques for scaling smaller samples of students into larger trends, and the data it would provide to education actors would be of superior quality. Or for you quant guys….

[Revenue neutral + Better Data = Winner!]

* Notice that I did not use the term ‘formative’. In the classroom, formative assessments should be standard operating procedure. It provides real-time learning data to both teacher and student. However, the subject here is large-scale education data used for identifying trends, problems, etc.