Category Archives: UDL

Goodhart’s Law and Data-Driven Decision Making

“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” [Goodhart, Charles (1981). “Problems of Monetary Management: The U.K. Experience”. Anthony S. Courakis (ed.), Inflation, Depression, and Economic Policy in the West (Rowman & Littlefield): 116.]

Educators are relying more and more on data to inform their work.  Individual teachers are using hard data to make decisions about instructional practice (Formative Assessment, anyone?) and whole states are making policy decisions based on “what the data says” (Third Grade Reading Guarantee, anyone?).

The phenomenon is not unique to education.  But, we may not be as far down the road as other professions.  Professionals in other fields have experienced their own versions of “data-based decision making” and learned much from the experience.

In the field of economics, Charles Goodhart made the opening statement in this article.  It essentially says, “When a measure becomes a target, it ceases to be a good measure.

For instance, here is a classic economic example.  A certain factory produces nails.  The owner of the factory finds that productivity and profits are at their peak when the factory produces 100,000 nails per week.  This becomes a target, in the form of a quota.  The factory workers begin cranking out some very small, shoddy nails that can be produced quickly.  The workers easily make the 100,000 target, but it has ceased to be a good indicator, because the nails that are now being produced are useless.

Here is a hypothetical example as it might happen in a school.  A school records copious amounts of data on student behavior and discipline efforts over an entire school year.  At the end of the school year, staff members analyze the data and determine that in 30% of instances of student behavior that resulted in suspension or expulsion, the student was wearing a black shirt, and that figure is far higher than any other shirt color.  Based on this data, the school proposes a new rule for the next school year… students may no longer wear black shirts to school.

The indicator has become the target.  Rather than trying to reduce or eliminate the real target (behaviors that resulted in suspensions), the school has addressed the indicator (shirt color).  There is an underlying belief that the target and the indicator are so tightly linked that reducing one automatically results in the reduction of the other.

The Third Grade Reading Guarantee

More than half of US states now have some form of a “third grade reading guarantee” in place.  These have been sparked by volumes of data about overall student performance linked to the student’s proficiency in reading by the end of third grade.  One such example from the Annie E. Casey Foundation study “Early Warning Confirmed” draws such conclusions as “children who do not read proficiently by the end of third grade are four times more likely to leave school without a diploma than proficient readers.” [http://www.aecf.org/~/media/Pubs/Topics/Education/Other/EarlyWarningConfirmed/EarlyWarningConfirmed.pdf, page 4.]

This is certainly actionable data.  But, if we place our emphasis there, will we accomplish the real target?  Or will we just destroy the usefulness of third grade reading ability as an indicator of likelihood to graduate?  In effect, will our approach just change the color of the students’ shirts?

Correlation vs. Causation

xkcd comic about correlation and causation.
From xkcd.

If education and economics aren’t enough voices, let’s let the world of statistical science weigh in.  “Correlation” is when two sets of statistical data are closely related, so that when one changes, so does the other. “Causation” is when the existence of one circumstance causes another to happen (e.g., when I push the power button on my laptop, that causes it to turn on).  If two sets of data are correlated, we sometimes leap to the conclusion that one of them is causing the other.  We look for an explanation of the phenomenon.

Another example looks at the number of highway fatalities in the US, and the number of metric tons of lemons imported from Mexico to the US.  The accompanying graph clearly demonstrates that as the number of metric tons of lemons imported from Mexico increased (from 1996 to 2000), the number of US highway fatalities reduced at an incredibly similar rate.  We quickly jump from looking at the data to analyzing it by asking ourselves, “Why would increasing lemon imports reduce highway fatalities?”  And many of us also quickly come to the conclusion that no such link exists, and the extremely close correlation is nothing but coincidence.  A data-based decision in this instance would say that if you want to reduce highway fatalities even more, then lemon imports should be increased.  And nothing in our minds tells us that makes any sense.

Graph of tons of imported lemons versus highway fatalities.
Do imported lemons reduce highway fatalities?

Do What’s Right For Kids

None of these considerations should ever trump the foundational concept of “doing what is right for kids”. Even if locking my daughter in a room by herself and giving her electric shocks while reading caused her to retain more of what she reads and score higher on standardized reading assessments, I would still fight against them because they are wrong.  Whether my kids are able to read at the third-grade level by the end of third grade is one indicator of their likelihood to graduate.  However, whether they have a passion for learning is an even stronger indicator of whether my kids will succeed at school, and in life.  Any approaches that hinder or squelch that passion are wrong, regardless of any indicators that they support.

The “Unmeasurable” Cause

I am quite certain that the conclusion from the Annie E. Casey Foundation about reading proficiency by the end of third grade is correct.  But, I also believe that we make the mistake of taking correlation for causation, and fall deep into the pit of Goodhart’s Law, when we make third grade reading proficiency (or any other measurable indicator) our explicit target.  It may well be that the most affluent neighborhoods have a high percentage of yellow houses, but we do nothing to affect the socio-economic status of a neighborhood by requiring them to paint all their houses yellow, even if we provided all the materials and labor to do so.

Reading at third grade level by the end of third grade, graduating high school, and a host of other academic achievements are all attributable to a passion for learning.  That passion is largely not measurable, and thus it confounds the success formulas of those who wish to create whole cultures of data-based decision making, where experience, intuition, and passion are ignored when they contradict the conclusion pointed to by the almighty data.

This is not to say that data should be ignored.  Data can give us great insights into the effectiveness of instructional strategies and projecting student achievement. Data can help us see holes where they weren’t clear before, or make us face up to deficiencies that we did not (or did not want to) acknowledge by our own subjective perceptions.  However, we cannot allow data to make decisions for us, especially when experienced educators know the conclusions to be (at best) unrelated, or (at worst) detrimental to students.

The next time you are in a district/building/teacher leadership meeting, and the data being presented says you should take a certain course of action, and you know that action is no good, remember Goodhart’s Law, and ask yourself if you’re really just telling the students to change their shirts.

 

Advertisements

Ten Free Online Simulation Games for Education

originally posted December 2, 2011, at http://www.ohioregion14.org/perspectives/?p=767. It’s still good, so I’m re-posting it here.

Simulations can be an engaging, effective way to immerse students in a concept. They can also be a good way to spark a student’s interest in a topic by making the factual information to be learned seem “more real”, because it has an immediate purpose – advancing in the game.

Simulations generally require students to learn facts for a purpose beyond simply repeating them on a test. Simulations also often require exercises in decision-making and problem-solving that reach into the upper levels of higher-order thinking skills. Some simulations can be very expensive, though. And just because something is a simulation does not guarantee it will be either educational or engaging!

Here are my ten favorite free online educational simulations!

10) Blood Typing
http://www.nobelprize.org/educational/medicine/landsteiner/
Why do blood types matter? See what happens when three accident victims come to the emergency room. They all need transfusions, but none of them know their blood type. Can you figure it out in time to save them all?

9) Trade and Economics
http://www.nobelprize.org/educational/economics/trade/
Why do some countries specialize in certain goods? Do imports and exports really matter? How do production decisions in one part of the world affect other countries? Explore these questions in a wonderfully developed simulation of the Heckscher-Ohlin trade theory. Playable in minutes, and great for playing numerous times in one sitting. Allows students to adjust their strategy from one game to the next in order to achieve a higher score.

8) Zapitalism
http://www.Zapitalism.com/
Nice simulation of starting with a small retail store, and taking it to profitability based on the important decisions you make. What should you sell? How high should your prices be? What are the other stores around you selling?

7) Arm Surgery 2
http://www.learn4good.com/games/simulation/doctor-hospital-games-for-kids.htm
Seriously? Arm surgery? Yep! But don’t worry, the animation is “cartoony” enough that you shouldn’t be causing anyone to faint. And the subject matter is likely to have someone in your class saying “Hey, that happened to me!”

6) Cargo Bridge
http://limexgames.com/studio/games/cargo_bridge
Triangles. Leverage. Force. Momentum. Planning and Architecture. Resource management. It’s all part of “Cargo Bridge” (and its expanding variants, like the Armor Games Edition, and Christmas Levels Pack)! Use your knowledge of structures and force to keep advancing through higher and more difficult levels.

5) The River City Project (No longer free)
(Intro) http://muveb.gse.harvard.edu/~muvers/rivercityvideos/River_City_Fall_Large.mov
I’m breaking my own rule here by including a simulation that USED to be free, but is now licensed. In a sense, it was never free… funding used to be covered by a federal grant, but that grant has expired. The fact that the simulation is still available at all is a good thing, because it’s well constructed, and plays very well. Students who do not just rush through the game will be rewarded for their attention to details and recording of interesting facts gathered from interacting with the virtual world and the people in it.

4) GCF Learn Free – ATM
http://www.gcflearnfree.org/everydaylife/atm
Simulations can be used to place students in situations they never would find themselves in, to give them an idea of what life is like from other perspectives. But sometimes, simulations can be used to give students an idea of what life could be like for them in a year or two, or ten! GCF Learn Free’s ATM Simulation is one of the latter. Using an ATM may be second nature to many people today, but for some students, this is a vital skill they will need in order to function in society. This simulation gives them a safe way to experience using an ATM for themselves. GCF Learn Free simulations include printable worksheets to go along with the online activities.

3) The POD Game
http://www.thepodgame.com/
Do your high school students think they’re good in a crisis? Let them find out with this scary simulation, funded by the Center for Disease Control and developed by the Chicago Department of Public Health and CADE. An airplane has released deadly anthrax virus over the city, and you are working in a drug dispensing center’s Point-Of-Dispensing (POD). Take the training, and then see if you have what it takes to take the role of a Medical Screener, Forms Reviewer, or Dispensing in a tense situation.

2) Mission US – For Crown or Colony?
http://www.Mission-US.org/
Take up the role of Nat Wheeler, a 14-year-old boy in Boston. The time is the days leading up to the beginning of the Revolutionary War. As Nat, participants make decisions that impact the progression of the game, while learning important facts. Teachers can register an entire class and keep track of their progress through the well-constructed simulation. Don’t miss the fun mini-game “Pennywhistle Hero”!

Two more games have been added to this series.  “Flight to Freedom” has students playing the part of a runaway slave in the time leading up to the US Civil War, and “A Cheyenne Odyssey” puts students in the northern plains in 1866 as part of a Native American tribe!

1) The Oregon Trail
The grandaddy of educational simulations! The one that basically started it all! New versions exist for mobile devices, but they aren’t nearly as heavy on requiring independent thought and planning.
Start in Independence, Missouri, and plan your trip westward to start a new life for you and your family in 1848.
Original edition FREE online
ActiveGS version
http://www.virtualapple.org/oregontraildisk.html
Java Version
http://www.virtualapple.org/J_oregontraildisk.html

How Not To Implement RtI (Response to Intervention)

A recent tweet of mine got several favorites/retweets.  It was, like many of my tweets, a sudden off-the-cuff remark sparked by something I heard.

not_rti“Tier I – Talk. Tier II – Talk louder. Tier III – ID as special ed. That’s NOT how RtI is supposed to work, folks.”

My suggestions for implementing Response to Intervention in a way that will actually benefit kids:

  • Don’t use RtI as an excuse to homogenize Tier I.  Tier II is not an excuse not to implement universal design in Tier I.  The more kids you let fall into Tier II, the more difficult you are making it.
  • Tier II is not “Tier I again, a little slower and a little louder.”  If Tier I instruction wasn’t sufficient, there should be formative assessment data that shows where the deficiency exists, so that a targeted intervention can be implemented.  Not all kids “didn’t get it” for the same reason, and the same intervention won’t work for all.
  • Tier III is not the “gateway to special ed”.  Just because a student is identified as being in need of Tier III instruction in an RtI model does not mean they have some disability.  It means the attempts at instruction so far have been unsuccessful, and a larger barrier exists that must be addressed.
  • Tier III does not allow you to delay special education referral and identification.  “States and LEAs [Local Educational Agencies] have an obligation to ensure that evaluations of children suspected of having a disability are not delayed or denied because of implementation of an RtI strategy.” (OSEP Memo, 1-21-2011.)  In other words, “Well, we think Johnny might have a disability, but we’re doing RtI with him.  If Johnny gets to Tier III, then we’ll set up an evaluation.” is a violation of Johnny’s civil rights.
  • Response to Intervention isn’t something you only do with your special education or “at-risk” population.
  • Tier III shouldn’t be a final and unchangeable designation for any student.  If a certain student is always in Tier III, there’s a problem somewhere in Tier I and Tier II.

In short, Response to Intervention is not a sorting mechanism, or a panacea for student achievement gaps.  It only works when a teacher uses it as feedback for how well their instruction is doing its job, and changes how they teach based on what helps their students learn best.

 

Learning: Motivation and Mechanics

A Challenge

My friends do something very important for me.  They make me think.

Beyond that, they challenge me to formalize my thoughts into coherent messages.  This is what happened on Monday, March 10, during the weekly #OHEdChat twitter chat.

Scott Kinkopf asked me to write more about a comment I made.

We were discussing “learning”.  Yes, that’s right, “learning”.  Sean Wheeler (teachinghumans.com) made an excellent observation about learning being sparked by a) utility, b) curiosity, and/or c) whimsy.  My graduate school friends would say that means we learn through what is useful, unexplored, and/or interesting.

I thoroughly agree with that observation, but it does just tell one side of the story.  These sparks can tell us a lot about the motivation to learn, but that is not where the journey ends.  It is not enough to want to learn.  The resources to support learning must be accessible.

“Presuming Competence”

Professor Douglas Biklin won the UNESCO/Emir Jaber al-Ahmad al-Jaber al-Sabah Prize to promote Quality Education for Persons with Intellectual Disabilities in 2012.  His work focuses on providing equitable access for all students, regardless of labels such as socio-economic status and disability.

Can you explain the concept of “presuming competence” and how it relates to inclusive education?

Helen Keller and Anne Sullivan
By Family member of Thaxter P. Spencer, now part of the R. Stanton Avery Special Collections, at the New England Historic Genealogical Society. See Press Release [1] for more information. (New England Historic Genealogical Society) [Public domain], via Wikimedia Commons
When Anne Sullivan first worked with Helen Keller, she approached her with the presumption that she was competent, that Helen’s problem emanated from her not having an effective means of communication.   Even before Anne began to work with Helen, there was evidence of her desire to communicate—she used pantomime to show her interest in making ice cream or wanting toast with butter.  But it was Anne’s introduction of spelling and words that proved liberating for Helen.The principle of “presuming competence,” is simply to act as Anne Sullivan did.  Assume that a child has intellectual ability, provide opportunities to be exposed to learning, assume the child wants to learn and assert him or herself in the world. To not presume competence is to assume that some individuals cannot learn, develop, or participate in the world.  Presuming competence is nothing less than a Hippocratic oath for educators. It is a framework that says, approach each child as wanting to be fully included, wanting acceptance and appreciation, wanting to learn, wanting to be heard, wanting to contribute.  By presuming competence, educators place the burden on themselves to come up with ever more creative, innovative ways for individuals to learn.  The question is no longer who can be included or who can learn, but how can we achieve inclusive education.  We begin by presuming competence.

interview with Dr. Douglas Biklen

Every educator realizes at some point that it is not enough to merely want all students to learn.  It is likewise not enough to believe that every student wants to learn.  Students must have available the necessary resources, tools, and skills to facilitate the mechanics of learning.

Like Peanut Butter and Jelly

Motivation to learn must be paired with competency in the mechanics of learning.  The student who is competent in the mechanics but has no motivation is bored and disconnected from education.  The student who has the motivation but lacks important skills (e.g., decoding text, comprehending text, organizing thoughts) will become increasingly frustrated, like the fox who found he could not jump high enough to reach the grapes and concludes the grapes are sour anyway.

Problem is, our foxes are being tested on how many grapes they’ve eaten.  And instead of putting the grapes where the fox can get them, we conclude that if the fox were hungry enough, he’d find a way to get the grapes on his own.

Lifelong learners have both a strong motivation to learn and a firm grasp on the mechanics of acquiring and applying new information.  They have both the spark and the fuel that results in a roaring fire for learning.

If we hope to produce lifelong learners, we must attend to both of these essential factors.  We must protect and develop each student’s innate curiosity and desire to learn, and we must foster and support the development of appropriate, practical skills for all students to access the general curriculum.

If the next Helen Keller registered for your class tomorrow, could she learn there?  She has the motivation.  Are you supporting her in developing the mechanics?

Big thanks to Scott Kinkopf, Sean Wheeler, and Mike Thayer for taking me down this road!