Goals and Measurement Traps

Good vs less bad, and not losing sight of the big picture.

If we want to move towards a better world, we have to set a direction.  That's what a goal is for.  Then we have to establish a method to determine our progress on the journey from where we are to our goal.  That's what measurements are for.

The goals we set and the measurements we use influence the results we get.  But when we zoom out, we may find that a lot of our metrics for success are not aligned with what we wanted in the first place.

For example, let's say a national planning office wants to grow its industrial sector.  For a particular factory that makes nails, they decide to measure the number of nails produced by the factory each month.  More nails are better.  With nail quantity as the metric of success, the factory begins to make millions of tiny nails that are too small to hold anything together.  Realizing their mistake, they change the goal to be to maximize the total mass of nails produced instead of quantity.  So the factory starts churning out just a few enormous nails, too big to be of any use.  This is the likely apocryphal, rather dumb, yet illustrative idea of the "Soviet nail factory."

What if a national school system wants to raise student achievement in reading and mathematics?  Policymakers could decide that annual standardized test scores offer an easy measure of progress, and they might even link these scores to funding or penalties.  The test scores might improve.  The gains, however, would come with tradeoffs.  Schools would be incentivized to reduce time for science, history, art, or activities that nurture curiosity and critical thinking, favoring only the subjects whose test scores are looked at for funding.  Teachers may manipulate results by excluding low-performing students or focusing only on students near the scoring threshold.  Most likely, teachers would adjust the curriculum to "teach for the test," spending most of the class time on test material and neglecting other parts of student education that are important but not measured on the test.  The system may end up raising test scores by sacrificing quality education in the process.  This dynamic was observed in the "No Child Left Behind Act" of 2001.

Say a hospital wants to improve its quality of care.  The hospital board may decide that reducing mortality rates (patients who die in the hospital) is a good way to measure quality of care, and they can display that information to the public.  What happens?  The hospital starts turning away the sickest patients or transferring them to other hospitals to avoid affecting their mortality score.  The hospital can now publish that their mortality rates are very low, while the actual quality of care did not improve, and may have even incentivized harmful behaviors.

What's going on here?  Goodhart's Law, named after British economist Charles Goodhart, says: When a measure becomes a target, it ceases to be a good measure.

Similarly, Campbell's Law, named after psychologist and social scientist Donald Campbell, states: The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

Simon Kuznets, an economist working for the U.S. Department of Commerce, created the first national income accounts and presented them to Congress in 1934.  This was the origin of the modern metric of GDP, or Gross Domestic Product.  GDP is a monetary measure of the market value of all final goods and services produced within a country during a specific period, usually a year or a quarter. In other words, it tracks the total economic activity that involves buying and selling at market prices.  GDP became the dominant measure of national progress in the 1940s, when governments used it to coordinate wartime production and later to track economic recovery.  By the late 1940s and early 1950s, it had become the standard benchmark for comparing countries and for guiding policy.

Since then, somehow, GDP morphed into something of a divine metric that we use as a way to measure the health and well-being of not only the aggregate economy but also individuals.  This is despite Kuznets' warnings: "The welfare of a nation can scarcely be inferred from a measure of national income,” and “Distinctions must be kept in mind between quantity and quality of growth.”

More recently, in a 2009 paper titled Beyond GDP: The Need for New Measures of Progress,  Robert Costanza et al state:

"GDP ignores changes in the natural, social, and human components of community capital on which the community relies for continued existence and well-being.  As a result, GDP not only fails to measure key aspects of quality of life; in many ways, it encourages activities that are counter to long-term community well-being."

So why have we become a nation fixated on GDP?  A few reasons, perhaps: one being that it's easier to measure it than more subjective measures of human wellbeing.  Ignoring qualitative metrics because they can't be measured is called the McNamara fallacy, and it shapes what we choose to value in society.  It's pretty hard to measure the visual beauty of a monarch butterfly emerging from a chrysalis, the dollar value of an old-growth forest, or the well-being from knowing that your neighbors have got your back if something weird happens when you're away for the weekend.  But we can measure iPhone sales, 4K TV revenue, and Hulu subscriptions.

Then there's the issue that our capitalist economic system and credit-based monetary system require infinite economic growth, as measured by GDP, to continue.  If GDP growth doesn't continue, the systems become unstable and can collapse.

When we use GDP as a measure of success, we see nature as natural resources to be exploited and turned into economic goods.  Our economic system confuses map and territory.  Most people would argue that an economic/social system should seek to maximize things like quality of life, happiness, freedom, community, etc.  But with GDP as the key performance indicator that politicians use to measure national success, the real things we want get obfuscated, and we convert our natural capital into something that can be measured and taxed.  Forests become lumber.

When we worship the god of GDP, we shouldn't be surprised when it sacrifices our health, happiness, and meaning on the altar of progress.

Environmental Innovation

There are a lot of folks out there who want to do good for the environment.  There are entrepreneurs who want to make a difference and do things in a better way, including me.  We want to replant forests, reduce plastic pollution, farm regeneratively, stop burning fossil fuels, protect wildlife habitat, build sustainably, and all the things.  In doing so, we need to be careful of what we choose to measure so we don't inadvertently incentivize the wrong behavior or confuse a few metrics for holistic goals.

You know those water dispensers where you can fill your reusable cup with filtered water?  They have a nifty counter on them that displays how many plastic bottles the water dispenser has saved.  Reducing plastic bottles is great.  You figure out that you can save a bottle every time 8 fluid ounces come out of the machine.  So you put your hand in front of the sensor, and water comes out.  You're saving so many bottles!

This example is obviously idiotic.  Running water through a dispenser doesn't save plastic bottles.  The concept relies on the assumption that you were going to drink water anyway, and if you didn't use this machine, you'd buy a single-use plastic water bottle instead.  In many cases, that's true.  And in this simple case, we understand that sticking our hand in front of the sensor isn't going to rid the oceans of plastic.  But when the goals are more complex or abstracted from our daily lives, it becomes easier to confuse goals and measures.

We need to understand the distinction between net benefits and doing less bad.  There's nothing wrong with doing something less bad than how it's being done now—we need that—but we must not confuse it with doing a net good for the environment.

For example, solar panels are not good for the environment.  They are less bad than burning coal to produce electricity, but they aren't providing a net benefit for air quality, the land, or ecosystems.  From raw materials to commissioning, they still have negative environmental impacts associated with mining, transportation, fabrication, and installation.  The same is true of any renewable energy system, from wind to wave to hydro.  Wind turbines disrupt picturesque landscapes, kill birds, and produce waste at the end of their lives.  Hydroelectric plants dam rivers, block fish migrations, use insane amounts of concrete, and disrupt natural sediment and nutrient cycling.  These systems are not good.  They are simply less bad than burning fossil hydrocarbons in perpetuity to produce electricity.

In the transition to renewable energy systems, if we use the total installed capacity of solar and wind as a measure of success, we won't achieve the desired outcome.  Even if we use the percentage of primary energy use from renewables as a metric to maximize, we'll still run into trouble.  The implication is that renewables are good for the environment, and the more we install, the better off the environment is.

But solar panels are made from purified silicon, silver, aluminum, copper, glass, specialized polymers, along with occasional rare metals, all sourced through global mining and refining chains.  Wind turbines are made of steel, concrete, fiberglass composites, copper, lubricants, and rare earth elements.  The mining, refining, transportation, and fabrication are all reliant on diesel and other fossil fuels.  Even if one day renewable energy systems are able to supply all of the energy to build more renewable energy systems, they're still not good for the environment.  Extracting and refining all the materials from the environment still involves scarring the land with mines, polluting waterways and soil, and destroying natural habitat.  We still need heavy equipment, trucks, electrical infrastructure, energy storage systems, factories, a place to put the PV panels and turbines, and a way to recycle or dispose of them at the end of their useful lives.

So, why are we racing to build renewable energy systems?  And why is this seen as a good thing for the environment?  First, fossil fuels are finite, and we're running out.  Current extraction rates cannot continue much longer.  If the economy is to maintain its size and grow, it needs more energy.

Society has adopted the goal of reducing climate change.  CO2 is one of the key drivers of anthropogenic climate change.  Because they are directly linked, we use CO2 emissions as a key measure of near and long-term climate change.  Renewable energy systems are seen as "green" because they don't emit CO2 into the atmosphere during their operation of converting natural energy flows into electricity.  Burning fossil fuels for energy emits lots of CO2.  So if reducing climate change is society's primary environmental goal, and reducing CO2 emissions is one of the best ways to measure our progress, the logic goes, then replacing fossil fuels with renewables for our energy needs is good for the environment.  Fair enough.

If we use this logic and the goal of reducing CO2 emissions as a goal, then we'll measure the installed capacity of renewables.  We treat it as a target that signifies something good for the environment.  But as we've seen, renewables don't provide a net environmental benefit.  They are less bad than the incumbent technology, but nature would be better off in every way if we didn't use fossil fuels, solar panels, or wind turbines.  As long as we view climate change and CO2 emissions as the primary environmental problems to tackle, we'll keep installing more renewables.  If we view renewable installations as something to maximize, eventually we'll cover much of the earth, and other environmental problems will become apparent.

Same thing with electric vehicles.  If we view EVs as good, then we'll celebrate when every man, woman, and child owns 5.  Or if we aim to reduce polluting, gas-guzzling cars on the road by measuring EV miles driven, we'll also distort reality.  Every mile driven still requires electricity (maybe from fossil fuels) and involves infrastructure, tire aerosols, and wear and tear on the vehicle that will eventually necessitate a new one to be made.  Maybe drivers, thinking that driving their EV is totally green, will rack up more miles overall.

Perhaps the key measurement we should be looking at is not the amount of renewable energy capacity installed, but the amount of fossil fuels still being used.  It seems like the point should be to get fossil fuel use to zero, not to maximize renewables.  It might seem like a subtle distinction, but it makes a huge difference in where we end up in the long run. Renewable energy is growing rapidly, all while natural gas, oil, and coal are hitting all-time highs every year.  The addition of renewables is not helpful, environmentally speaking.

When we tacitly assume that something is good because it replaces something worse, we get into trouble.  We measure and optimize for the wrong thing, and we issue a license to harm.

When we assume that a new technology or product—renewable energy, clothes made from 50% recycled plastic, insulation made from hemp, etc—is helping the environment rather than harming it less, we incentivize a measure that conflates the goal.  This is not to suggest that all environmentally-conscious goals, businesses, organizations, and products are wrongheaded or misleading.  Some are, but not all.

I'm suggesting that the world is messy and context matters.  I'm suggesting that reductionism oversimplifies problems and overshoots goals.  Unintended consequences often result from measurements being mistaken for goals.  The world is complex and interconnected.  It's impossible to predict how one thing affects everything else from the outset.  If a goal is to be effective at creating positive change, it must be coupled with a mechanism to recognize when it has gone too far.  Avoid metric fixation and regularly reexamine if the measurement is serving the goal.

Whether conceived by the human mind—or otherwise—our models and goals must be holistic.  To create a more beautiful world for this generation and the ones after, for this species and the countless others, we must use discipline and restraint.  Good intentions without careful consideration can cause harmful effects.

Go ahead.  Pursue that moonshot.  We're counting on you.

And beware of dragons: dragons in the measures.

They might be closer than you think.

Disclaimer: No portion of this work may be used for training artificial intelligence systems without written permission from the author.

Resources:

Costanza, R., Hart, M., Posner, S., & Talberth, J. (2009). Beyond GDP: The need for new measures of progress (Pardee Paper No. 4). Frederick S. Pardee Center for the Study of the Longer-Range Future. http://archives.pdx.edu/ds/psu/8349

European Commission, Stiglitz-Sen-Fitoussi Commission. (2009). Report by the Commission on the Measurement of Economic Performance and Social Progress. Eurostat. https://ec.europa.eu/eurostat/documents/8131721/8131772/Stiglitz-Sen-Fitoussi-Commission-report.pdf

Jha, A. K., & Werner, R. M. (2006). Relationship between Medicare’s Hospital Compare performance measures and mortality rates. JAMA, 296(22), 2693–2704. https://jamanetwork.com/journals/jama/fullarticle/204523

National Geographic Society. (n.d.). Here Be Dragons. In 5th Grade Explorer. https://education.nationalgeographic.org/resource/here-be-dragons/5th-grade/