Original subtitles

- Hi, I'm William Lidwell and this

is Universal Principles of Design.

In this movie, Selection Bias.

How the dots we collect influence the dots we connect.

During World War II, a statistician

by the name of Abraham Wald

was tasked to research how allied bombers

were being felled by enemy fire.

The idea was that by identifying areas of vulnerability

in the bombers, we would know where to add armor,

increasing the survivability of bombers on future missions.

So Wald beautifully collected data based on bomber damage

and then rendered the results of his analysis

on a diagram of an aircraft that looks something like this.

The red dots represent bulletholes and flak damage.

Looking at his data, where would you add the armor?

If you're like most people, the answer seems obvious.

You add the armor where the damage is.

Where the red dots are.

But remember, Walt's data were based

on aircraft that had survived.

That had successfully made it back.

The data did not include the bombers that were shot down.

Walt of course knew this.

He knew the data were biased in this way.

A bias called selection bias.

And because of this understanding, he drew to me,

a mind-blowingly ingenious and counterintuitive conclusion.

Wald said we should add armor

to the areas with no damage, with no red dots.

Because the bombers hit in the red areas came home.

The ones that didn't come home

must have been hit in the non-red areas.

Brilliant.

Why do people so consistently draw the wrong conclusion

when confronted with data like this?

The answer is simple.

Humans are pattern-detecting and pattern-making machines.

When we see dots, we try to connect them.

It's reflexive.

It is only when we, like Wald,

understand the perils of biases like selection bias

that we pause and make sure the dots

that have been collected are worthy of connecting.

So what is selection bias?

Stated simply, selection bias is a bias

in the way evidence is collected

that distorts our analysis and conclusions.

In the case of Wald's damaged bombers,

the evidence was biased through no bad intentions or faults.

The downed planes simply were not

available for consideration.

But this is often not the case.

People who want to persuade often cherry pick data

that support their position and exclude data that negate it.

Resulting in evidence that appears convincing

but that is not representative of the truth.

For example, climate change denialists

typically cherry-pick climate data from just the last decade

which indicates flat or declining global temperatures

ignoring the clear longer term historical trend.

How can we prevent selection bias?

It's surprisingly simple in theory,

a little harder in practice.

When you're dealing with a small population,

meaning a small number of things

about which you're collecting data,

like a classroom of students.

You collect data from everyone, all of the students.

If all members of a population are represented

in your analysis, there can be no selection bias.

Unfortunately, in most real world situations,

it is neither possible nor practical to do this.

A number of members in a population is too large,

or as with the bombers, just not available.

In these cases, the key is random sampling.

You must randomly select members

from the available population.

A truly random sample prevents selection bias.

You just need to make sure that you sample

from the full set of things you're generalizing about.

For example, Walt collected data from a subset of bombers.

The ones that returned.

But needed to generalize the results to all bombers.

This is why adding armor to the red areas was wrong.

The sample of damaged bombers was not random,

and so did not generalize the full set.

Similarly, if you surveyed a random group of Mac users,

you couldn't generalize to PC users.

So whether your knowledge of selection bias

helps you do better design research

or think more critically about how data and statistics

are used to persuade and drive decision making.

Remember the only way to connect the right dots

is to collect the right dots.