All language subtitles for NOVA S49E13 Computers v Crime 1080p WEB h264-BAE (1)_track3_[eng]

♪ ♪

SARAH BRAYNE:
We live in this era where

we leave digital traces
throughout the course

of our everyday lives.

ANDY CLARNO:
What is this data,

how is it collected,
how is it being used?

NARRATOR:
One way it's being used
is to make predictions

about who might commit
a crime...

Hey, give me all
your money, man!

NARRATOR:
...and who should get bail.

JUDGE:
On count one, you're charged
with felony intimidation...

ANDREW FERGUSON:
The idea is that
if you look at past crimes,

you might be able to predict
the future.

WILLIAM ISAAC:
We want safer communities,

we want societies that are
less incarcerated.

NARRATOR:
But is that what we're getting?

Are the predictions reliable?

CATHY O'NEIL:
I think algorithms can,

in many cases,
be better than people.

But, of course, algorithms
don't have consciousness.

The algorithm only knows
what it's been fed.

RUHA BENJAMIN:
Because it's technology,

we don't question them as
much as we might

a racist judge
or a racist officer.

They're behind this veneer
of neutrality.

ISAAC:
We need to know
who's accountable

when systems harm
the communities

that they're designed to serve.

NARRATOR:
Can we trust the justice
of predictive algorithms?

And should we?

"Computers Vs. Crime,"

right now, on "NOVA."

(computers booting up)

♪ ♪

NARRATOR:
We live in a world of big data,

where computers look
for patterns

in vast collections
of information

in order to predict the future.

And we depend on their accuracy.

Is it a good morning
for jogging?

Will this become cancer?

What movie should I choose?

The best way to beat traffic?

Your computer can tell you.

Similar computer programs,
called predictive algorithms,

are mining big data
to make predictions

about crime and punishment--

reinventing how our
criminal legal system works.

Policing agencies have used
these computer algorithms

in an effort to predict where
the next crime will occur

and even who the perpetrator
will be.

ASSISTANT DISTRICT ATTORNEY:
Here, the state is
recommending...

NARRATOR:
Judges use them

to determine who should get bail

and who shouldn't.

JUDGE:
If you fail to appear
next time, you get no bond.

NARRATOR:
It may sound like
the police of the future

in the movie
"Minority Report."
I'm placing you under arrest

for the future murder
of Sarah Marks.

NARRATOR:
But fiction it's not.

How do these predictions
actually work?

Can computer algorithms

make our criminal legal system
more equitable?

Are these algorithms truly fair
and free of human bias?

ANDREW PAPACHRISTOS:
I grew up in Chicago

in the 1980s and early 1990s.

♪ ♪

My dad was an immigrant
from Greece,

we worked in my
family's restaurant,

called KaMar's.

NARRATOR:
Andrew Papachristos
was a 16-year-old kid

in the North Side of Chicago
in the 1990s.

I spent a lot of my formative
years busing tables,

serving people hamburgers
and gyros.

It kind of was a whole
family affair.

NARRATOR:
Young Papachristos was aware
the streets could be dangerous,

but never imagined the violence
would touch him or his family.

REPORTER:
Two more gang-related
murders Monday night.

PAPACHRISTOS:
And of course, you know,
the '80s and '90s in Chicago

was some of the historically
most violent periods in Chicago.

Street corner drug markets,
street organizations.

And then like a lot of other
businesses on our, on our block

and in our neighborhood,

local gangs tried to extort
my family and the business.

And my dad had been running
KaMar's for 30 years

and kind of just said no.

♪ ♪

(sirens blaring)

NARRATOR:
Then, one night, the family
restaurant burned to the ground.

Police suspected arson.

PAPACHRISTOS:
It was quite a shock
to our family,

'cause everybody
in the neighborhood worked

in the restaurant at one point
in their life.

And my parents lost
30 years of their lives.

That was really one of the
events that made me want to

understand violence.

Like, how could this happen?

♪ ♪

NARRATOR:
About a decade later,

Papachristos was a graduate
student searching for answers.

PAPACHRISTOS:
In graduate school,

I was working on
a violence prevention program

that brought together
community members,

including street outreach
workers.

And we were sitting at a table,

and one of these
outreach workers asked me,

the university student,

"Who's next?

Who's going to get shot next?"

And where that led
was me sitting down

with stacks of shooting and,
and homicide files

with a red pen and a legal pad,

by hand creating these
network images

of, this person shot this
person,

and this person was involved
with this group and this event,

and creating a web
of these relationships.

And then I learned that

there's this whole science
about networks.

I didn't have to invent
anything.

♪ ♪

NARRATOR:
Social network analysis
was already influencing

popular culture.

"Six Degrees of Separation"
was a play on Broadway.

And then, there was
Six Degrees of Kevin Bacon.

PAPACHRISTOS:
The idea was,
you would play this game,

and whoever got the
shortest distance to Kevin Bacon

would win.

So Robert De Niro was in
a movie with so-and-so,

who was in a movie with
Kevin Bacon.

It was creating, essentially,
a series of ties

among movies and actors.

And in fact,
there's a mathematics

behind that principle.

It's actually old mathematical
graph theory, right?

That goes back
to 1900s mathematics.

And lots of scientists started
seeing that there were

mathematical principles,

and computational resources--
computers, data--

were at a point that you could
test those things.

So it was in a
very exciting time.

We looked at arrest records,
at police stops,

and we looked at
victimization records.

Who was the victim
of a homicide

or a non-fatal shooting?

♪ ♪

The statistical model starts by
creating the social networks of,

say, everybody who may have
been arrested

in a,
in a particular neighborhood.

So Person A and Person B
were in a robbery together,

they have a tie,
and then Person B and Person C

were, were stopped by the police
in another instance.

And it creates networks of
thousands of people.

Understanding that events
are connected,

places are connected.

That there are old things,
like disputes between crews,

which actually drive behavior
for generations.

What we saw was striking.

(snaps):
And you could see it
immediately,

and you could see it
a mile away.

Which was, gunshot victims
clumped together.

You, you very rarely see one
victim.

You see two, three, four.

Sometimes they string across
time and space.

And then the model predicts,
what's the probability

that this is going to lead to
a shooting

on the same pathway
in the future?

(gun firing, people shouting)

REPORTER:
Another young man lies dead.

NARRATOR:
In Boston, Papachristos found

that 85% of all
gunshot injuries

occurred within a single
social network.

Individuals in this network

were less than
five handshakes away

from the victim of a
gun homicide

or non-fatal shooting.

The closer a person was

connected to a gunshot victim,
he found,

the greater the probability that
that person would be shot.

Around 2011,
when Papachristos was presenting

his groundbreaking work
on social networks

and gang violence,

the Chicago Police Department
wanted to know more.

PAPACHRISTOS:
We were at a conference.

The then-superintendent
of the police department,

he was asking me a bunch
of questions.

He had clearly read the paper.

NARRATOR:
The Chicago Police Department

was working on its own
predictive policing program

to fight crime.

They were convinced
that Papachristos's model

could make their new policing
model even more effective.

could make their new policing
model even more effective.

LOGAN KOEPKE:
Predictive policing involves

looking to historical crime data
to predict future events,

either where police believe
crime may occur

or who might be involved
in certain crimes.

♪ ♪

So it's the use of historical
data to forecast a future event.

NARRATOR:
At the core of these programs
is software,

which, like all
computer programs,

is built around an algorithm.

So, think of an algorithm
like a recipe.

♪ ♪

You have inputs,

which are your ingredients,
you have the algorithm,

which is the steps.

♪ ♪

And then there's the output,

which is hopefully the
delicious cake you're making.

GROUP:
Happy birthday!

So one way to think about
algorithms is to think about

the hiring process.

In fact, recruiters have been
studied for a hundred years.

And it turns out
many human recruiters

have a standard algorithm

when they're looking at
a risumi.

So they start with your name,
and then they look to see

where you went to school,
and then finally,

they look at what your
last job was.

If they don't see the pattern
they're looking for...

(bell dings)
...that's all the time you get.

And in a sense, that's exactly
what artificial intelligence

is doing, as well,
in a very basic level.

It's recognizing sets
of patterns and using that

to decide what the next step in
its decision process would be.

♪ ♪

NARRATOR:
What is commonly referred to

as artificial intelligence,
or A.I.,

is a process called
machine learning,

where a computer algorithm
will adjust on its own,

without human instructions,

in response to the patterns
it finds in the data.

These powerful processes
can analyze more data

than any person can,

and find patterns never
recognized before.

The principles for
machine learning

were invented in the 1950s,

but began proliferating
only after about 2010.

What we consider
machine learning today

came about because hard drives
became very cheap.

So it was really easy to get
a lot of data on everyone

in every aspect of life.

And the question is, what can
we do with all of that data?

Those new uses are things like
predictive policing,

they are things like deciding
whether or not a person's

going to get a job or not,

or be invited for
a job interview.

NARRATOR:
So how does such a powerful tool
like machine learning work?

Take the case of a
hiring algorithm.

First, a computer needs to
understand the objective.

Here, the objective is
identifying

the best candidate
for the job.

The algorithm looks at risumis
of former job candidates

and searches for keywords
in risumis of successful hires.

The risumis are what's
called training data.

The algorithm assigns values
to each keyword.

Words that appear more
frequently in the risumis

of successful candidates
are given more value.

The system learns from
past risumis the patterns

of qualities that are associated
with successful hires.

Then it makes its predictions
by identifying these

same patterns from the risumis
of potential candidates.

♪ ♪

In a similar way,

the Chicago police wanted to
find patterns in crime reports

and arrest records to predict
who would be connected

to violence in the future.

They thought Papachristos's
model could help.

Obviously we wanted to,
and tried, and framed and wrote

all the caveats and made
our recommendations to say,

"This research should be
in this public health space."

But once the math is out there,

once the statistics
are out there,

people can also take it
and do what they want with it.

NARRATOR:
While Papachristos saw
the model as a tool to identify

future victims of gun violence,

CPD saw the chance to identify
not only future victims,

but future criminals.

First it took me, you know,
by, by surprise,

and then it got me worried.

What is it gonna do?

Who is it gonna harm?

♪ ♪

NARRATOR:
What the police wanted
to predict was who was at risk

for being involved in
future violence.

Gimme all your money, man.

NARRATOR:
Training on hundreds of
thousands of arrest records,

the computer algorithm looks
for patterns or factors

associated with violent crime

to calculate the risk
that an individual

will be connected to
future violence.

Using social network analysis,

arrest records of associates

are also included in that
calculation.

The program was called the
Strategic Subject List, or SSL.

It would be one of the most
controversial

in Chicago policing history.

ANDY CLARNO:
The idea behind the
Strategic Subjects List,

or the SSL,

was to try to identify
the people who would be

most likely to become involved
as what they called

a "party to violence," either as
a shooter or a victim.

♪ ♪

NARRATOR:
Chicago police would use
Papachristos's research

to evaluate what was called
an individual's

"co-arrest network."

And the way that the
Chicago Police Department

calculated an
individual's network was through

kind of two degrees of removal.

Anybody that I'd been arrested
with and anybody that they

would, had been arrested with
counted as people who were

within my network.

So my risk score would be
based on my individual history

of arrest and victimization,
as well as the histories

of arrest and victimization
of people within that

two-degree network of mine.

It was colloquially known
as the "heat list."

If you were hot, you were on it.

And they gave you literally
a risk score.

At one time,
it was zero to 500-plus.

If you're 500-plus, you are
a high-risk person.

♪ ♪

And if you made this heat list,

you might find a detective
knocking on your front door.

♪ ♪

NARRATOR:
Trying to predict

future criminal activity is not
a new idea.

Scotland Yard in London
began using this approach

by mapping crime events
in the 1930s.

But in the 1990s,

it was New York City Police
Commissioner William Bratton

who took crime mapping
to another level.

BRATTON:
I run the New York City
Police Department.

My competition is the
criminal element.

NARRATOR:
Bratton convinced policing
agencies across the country

that data-driven policing
was the key

to successful
policing strategies.

Part of this is to prevent
crime in the first place.

♪ ♪

NARRATOR:
Bratton was inspired

by the work of his own
New York City Transit Police.

As you see all those,

uh, dots on the map,

that's our opponents.

NARRATOR:
It was called
Charts of the Future,

and credited with cutting
subway felonies by 27%

and robberies by a third.

Bratton saw potential.

He ordered all
New York City precincts

to systematically map crime,

collect data, find patterns,
report back.

The new approach was
called CompStat.

BRAYNE:
CompStat, I think, in a way,
is kind of a precessor

of predictive policing,

in the sense that many of
the same principles there--

you know, using data tracking,
year-to-dates,

identifying places where
law enforcement interventions

could be effective, et cetera--

really laid the groundwork
for predictive policing.

♪ ♪

NARRATOR:
By the early 2000s,

as computational power
increased,

criminologists were
convinced this new data trove

could be used in machine
learning to create models

that predict when and where

crime would happen
in the future.

♪ ♪

REPORTER:
L.A. police now say
the gunmen opened fire

with a semi-automatic weapon.

NARRATOR:
In 2008,

now as chief of the
Los Angeles Police Department,

Bratton joined with academics
at U.C.L.A.

to help launch a
predictive policing system

called PredPol,

powered by a machine learning
algorithm.

♪ ♪

ISAAC:
PredPol started

as a spin-off
of a set of, like,

government contracts that were
related to military work.

They were developing

a form of an algorithm that was
used to predict I.E.Ds.

(device explodes)

And it was a technique
that was used

to also detect aftershocks
and seismographic activity.

(dogs barking and whining,
objects clattering)

And after those contracts ended,

the company decided
they wanted to apply this

in the domain of, of
policing

domestically in the
United States.

(radio beeping)

NARRATOR:
The PredPol model

relies on three types
of historical data:

type of crime, crime location,
and time of crime,

going back two to five years.

The algorithm is looking for
patterns

to identify locations where
crime is most likely to occur.

As new crime incidents
are reported,

they get folded into
the calculation.

The predictions are displayed
on a map

as 500 x 500 foot areas
that officers are then

directed to patrol.

ISAAC:
And then from there,
the algorithm says,

"Okay, based on what we know
about the kind of

"very recent history,

"where is likely that
we'll see crime

in the next day
or the next hour?"

♪ ♪

BRAYNE:
One of the key reasons

that police start using
these tools

is the efficient
and even, to a certain extent,

like in their logic,

more fair, um, and, and
justifiable allocation

of their police resources.

♪ ♪

NARRATOR:
By 2013,

in addition to PredPol,
predictive policing systems

developed by companies like
HunchLab, IBM, and Palantir

were in use across the country.

(radios running in background)

And computer algorithms

were also being adopted
in courtrooms.

BAILIFF:
21CF3810, State of Wisconsin
versus Chantille...

KATHERINE FORREST:
These tools are used in
pretrial determinations,

they're used in sentencing
determinations,

and they're used in
housing determinations.

They're also used, importantly,
in the plea bargaining phase.

They're used really throughout
the entire process

to try to do what judges
have been doing,

which is the very, very
difficult task

of trying to understand
and predict

what will a human being
do tomorrow, or the next day,

or next month, or three years
from now.

ASSISTANT DISTRICT ATTORNEY:
Bail forfeited.

He failed to appear
12/13/21.

Didn't even make it
to preliminary hearing.

The software tools are an
attempt to try to predict

it better than humans can.

MICHELLE HAVAS:
On count one, you're charged
with

felony intimidation
of a victim.

SWEENEY:
So, in the United States,
you're innocent

until you've been proven guilty,
but you've been arrested.

Now that you've been arrested,

a judge has to decide
whether or not

you get out on bail,

or how high or low that bail
should be.

You're charged with driving
on a suspended license.

I've set that bond at $1,000.

No insurance, I've set that bond
at $1,000.

ALISON SHAMES:
One of the problems is,

judges often are relying on
money bond

or financial conditions
of release.

JUDGE:
So I'm going to
lower his fine

to make it a bit
more reasonable.

So instead of $250,000 cash,

surety is $100,000.

SHAMES:
It allows people who have
access to money to be released.

If you are poor, you are often
being detained pretrial.

Approximately 70% of the people
in jail are there on pretrial.

These are people who are
presumed innocent,

but are detained during the
pretrial stage of their case.

NARRATOR:
Many jurisdictions use
pretrial assessment algorithms

with a goal to reduce
jail populations and decrease

the impact of judicial bias.

SHAMES:
The use of a tool like this
takes historical data

and assesses, based on research,

associates factors that are
predictive of the two outcomes

that the judge is
concerned with.

That's community safety
and whether that person

will appear back in court
during the pretrial period.

♪ ♪

NARRATOR:
Many of these algorithms
are based on a concept called

a regression model.

The earliest,
called linear regression,

dates back to 19th-century
mathematics.

O'NEIL:
At the end of the day,
machine learning algorithms

do exactly what
linear regression does,

which is predict--

based on the initial conditions,
the situation they're seeing--

predict what will happen
in the future,

whether that's, like, in
the next one minute

or the next four years.

NARRATOR:
Throughout the United States,
over 60 jurisdictions

use predictive algorithms
as part of the legal process.

One of the most widely used
is COMPAS.

The COMPAS algorithm
weighs factors,

including a defendant's answers
to a questionnaire,

to provide a
risk assessment score.

These scores are used every day
by judges to guide decisions

about pretrial detention, bail,
and even sentencing.

But the reliability
of the COMPAS algorithm

has been questioned.

In 2016, ProPublica published
an investigative report

on the COMPAS
risk assessment tool.

BENJAMIN:
Investigators wanted to see
if the scores were accurate

in predicting whether
these individuals

would commit a future crime.

And they found two things
that were interesting.

One was that the score
was remarkably unreliable

in predicting who would commit
a, a crime in the future

over this two-year period.

But then the other thing that
ProPublica investigators found

was that Black people were
much more likely to be deemed

high risk and white people
low risk.

NARRATOR:
This was true even in cases
when the Black person

was arrested for a minor offense
and the white person

in question was arrested for
a more serious crime.

BENJAMIN:
This ProPublica study
was one of the first to begin

to burst the bubble
of technology

as somehow objective
and neutral.

NARRATOR:
The article created
a national controversy.

But at Dartmouth, a student
convinced her professor

they should both be
more than stunned.

HANY FARID:
As it turns out, one of my
students, Julia Dressel,

reads the same article and said,

"This is terrible.

We should do something
about it."
(chuckles)

This is the difference between
an awesome idealistic student

and a jaded, uh, professor.

And I thought,
"I think you're right."

And as we were sort of
struggling to understand

the underlying roots of the bias
in the algorithms,

we asked ourselves
a really simple question:

are the algorithms today, are
they doing better than humans?

Because presumably, that's why
you have these algorithms,

is that they eliminate some of
the bias and the prejudices,

either implicit or explicit,
in the human judgment.

NARRATOR:
To analyze COMPAS's
risk assessment accuracy,

they used the crowdsourcing
platform Mechanical Turk.

Their online study included
400 participants

who evaluated 1,000 defendants.

FARID:
We asked participants to

read a very short paragraph
about an actual defendant.

How old they were,

whether they were
male or female,

what their prior
juvenile conviction record was,

and their prior
adult conviction record.

And, importantly, we didn't
tell people their race.

And then we ask a very simple
question,

"Do you think this person
will commit a crime

in the next two years?"

Yes, no.

And again,
these are non-experts.

These are people being paid

a couple of bucks online
to answer a survey.

No criminal justice experience,

don't know anything about
the defendants.

They were as accurate
as the commercial software

being used in the courts today,

one particular piece
of software.

That was really surprising.

We would've expected
a little bit of improvement.

After all, the algorithm has
access

to huge amounts
of training data.

NARRATOR:
And something else
puzzled the researchers.

The MTurk workers' answers to
questions

about who would commit crimes in
the future and who wouldn't

showed a surprising pattern of
racial bias,

even though race wasn't
indicated

in any of the profiles.

They were more likely to say a
person of color

will be high risk when they
weren't,

and they were more likely to say
that a white person

would not be high risk when in
fact they were.

And this made no sense to us at
all.

You don't know the race of the
person.

How is it possible that you're
biased against them?

(radios running in background)

In this country,
if you are a person of color,

you are significantly more
likely, historically,

to be arrested,

to be charged,
and to be convicted of a crime.

So in fact, prior convictions

is a proxy for your race.

Not a perfect proxy,
but it is correlated,

because of the historical
inequities

in the criminal justice system

and policing in this country.

(siren blaring)

MAN:
It's my car, bro, come on,
what are y'all doing?

Like, this, this is racial
profiling.

NARRATOR:
Research indicates a
Black person

is five times more likely to be
stopped without cause

than a white person.

Black people are at least twice
as likely

as white people to be
arrested for drug offenses,

even though Black and
white people

use drugs at the same rate.

Black people are also about 12
times

more likely to be wrongly
convicted of drug crimes.

FORREST:
Historically, Black men have
been arrested at higher levels

than other populations.

Therefore, the tool predicts
that a Black man, for instance,

will be arrested at a rate and
recidivate at a rate

that is higher than a white
individual.

FARID:
And so what was happening is,
you know, the big data,

the big machine
learning folks are saying,

"Look, we're not giving it
race-- it can't be racist."

But that
is spectacularly naive,

because we know that other
things correlate with race.

In this case, number
of prior convictions.

And so when you train an
algorithm on historical data,

well, guess what.

It's going to reproduce
history-- of course it will.

NARRATOR:
Compounding the problem is the
fact that

predictive algorithms can't be
put on the witness stand

and interrogated about their
decision-making processes.

FORREST:
Many defendants have had
difficulty

getting access to
the underlying information

that tells them,

what was the data set that was
used to assess me?

What were the inputs
that were used?

How were those inputs weighted?

So you've got what can be,
these days,

increasingly, a black box.

A lack of transparency.

NARRATOR:
Some black box algorithms get
their name

from a lack of transparency
about the code

and data inputs they use,

which can be deemed proprietary.

But that's not the only
kind of black box.

A black box is any system
which is so complicated

that you can see what goes in
and you can see what comes out,

but it's impossible to
understand

what's going on inside it.

All of those steps in the
algorithm

are hidden inside phenomenally
complex math

and processes.

FARID:
And I would argue that when you
are using algorithms

in mission-critical
applications,

like criminal justice system,

we should not be deploying
black box algorithms.

NARRATOR:
PredPol, like many
predictive platforms,

claimed a proven record for
crime reduction.

In 2015, PredPol published its
algorithm

In 2015, PredPol published its
algorithm

in a peer-reviewed journal.

William Isaac and Kristian Lum,

research scientists
who investigate

predictive policing platforms,

analyzed the algorithm.

ISAAC:
We just kind of saw the
algorithm

as going back to the
same one or two blocks

every single time.

And that's kind of strange,

because if you had a truly
predictive policing system,

you wouldn't necessarily see it
going to the same locations

over and over again.

NARRATOR:
For their experiment,

Isaac and Lum used a different
data set,

public health data, to map
illicit drug use in Oakland.

ISAAC:
So, a good chunk
of the city was kind of

evenly distributed in terms of
where

potential illicit drug use might
be.

But the police predictions were
clustering around areas

where police had, you know,

historically found incidents of
illicit drug use.

Specifically, we saw significant
numbers of neighborhoods

that were predominantly
non-white and lower-income

being deliberate targets of the
predictions.

NARRATOR:
Even though illicit drug use was
a citywide problem,

the algorithm focused its
predictions

on low-income neighborhoods
and communities of color.

ISAAC:
The reason why is
actually really important.

It's very hard to divorce

these predictions from those
histories

and legacies of over-policing.

As a result of that, they
manifest themselves in the data.

NARRATOR:
In an area where there
is more police presence,

more crime is uncovered.

The crime data indicates to the
algorithm

that the heavily
policed neighborhood

is where future crime will be
found,

even though there may be other
neighborhoods

where crimes are being committed
at the same or higher rate.

ISAAC:
Every new prediction that
you generate

is going to be
increasingly dependent

on the behavior of the algorithm
in the past.

So, you know,
if you go ten days, 20 days,

30 days into the future, right,
after using an algorithm,

all of those predictions have
changed the behavior

of the police department

and are now being folded back
into the next day's prediction.

NARRATOR:
The result can be a
feedback loop

that reinforces
historical policing practices.

SWEENEY:
All of these different types

of machine learning algorithms
are all trying to help us

figure out, are there some
patterns in this data?

It's up to us to then figure
out,

are those legitimate patterns,
do they,

are they useful patterns?

Because the computer has no
idea.

It didn't make a logical
association.

It just made it, made a
correlation.

MING:
My favorite definition of
artificial intelligence

is, it's any autonomous system

that can make decisions under
uncertainty.

You can't make decisions under
uncertainty without bias.

In fact, it's impossible
to escape from having bias.

It's a mathematical reality

about any intelligent system,
even us.

(siren blaring in distance)

NARRATOR:
And even if the goal
is to get rid of prejudice,

bias in the historical data can
undermine that objective.

Amazon discovered this

when they began a search for top
talent

with a hiring algorithm
whose training data

depended on hiring successes
from the past.

MING:
Amazon, somewhat famously
within the A.I. industry,

they tried to build a hiring
algorithm.

They had a massive data set.

They had all the right answers,

because they knew literally who
got hired

and who got that
promotion in their first year.

(typing)

NARRATOR:
The company created
multiple models

to review past
candidates' risumis

and identify some 50,000
key terms.

MING:
What Amazon actually
wanted to achieve

was to diversify their hiring.

Amazon, just like every other
tech company,

and a lot of other companies, as
well,

has enormous bias built
into its hiring history.

It was always biased, strongly
biased, in favor of men,

in favor, generally,

of white or sometimes Asian men.

Well, they went and built
a hiring algorithm.

And sure enough, this thing was

the most sexist recruiter you
could imagine.

If you said the word "women's"
in your risumi,

then it wouldn't hire you.

If you went to a women's
college,

it didn't want to hire you.

So they take out all the gender
markers,

and all of the women's
colleges--

all the things that explicitly
says,

"This is a man," and, "This is a
woman,"

or even the ones that,
obviously, implicitly say it.

So they did that.

And then they trained up their
new deep neural network

to decide who Amazon would hire.

And it did something amazing.

It did something no
human could do.

It figured out who was a woman
and it wouldn't hire them.

It was able to look through

all of the correlations that
existed

in that massive data set

and figure out which ones most
strongly correlated

with someone getting a
promotion.

And the single biggest
correlate

of getting a promotion was being
a man.

And it figured those patterns
out and didn't hire women.

NARRATOR:
Amazon abandoned its
hiring algorithm in 2017.

Remember the way machine
learning works, right?

It's like a student who doesn't
really understand

the material in the class.

They got a bunch of questions,
they got a bunch of answers.

They got a bunch of questions,
they got a bunch of answers.

And now they're trying to
pattern match

for a new question and say,
"Oh, wait.

"Let me find an answer that
looks pretty much

like the questions and answers I
saw before."

The algorithm only worked
because someone has said,

"Oh, this person whose data you
have,

"they were a good employee.

This other person was a bad
employee," or, "This person

performed well," or, "This
person did not perform well."

O'NEIL:
Because algorithms
don't just look for patterns,

they look for patterns of
success, however it's defined.

But the definition of success
is really critically important

to what that end up, ends up
being.

And a lot of, a lot of opinion

is embedded in, what, what does
success look like?

NARRATOR:
In the case of algorithms,

human choices play a critical
role.

O'NEIL:
The data itself was curated.

Someone decided what
data to collect.

Somebody decided what data
was not relevant, right?

And they don't exclude
it necessarily

intentionally--
they could be blind spots.

NARRATOR:
The need to identify
such oversights

becomes more urgent

as technology takes on more
decision making.

♪ ♪

Consider facial recognition
technology,

used by law enforcement

in cities around the world for
surveillance.

In Detroit, 2018,

law enforcement looked to facial
recognition technology

when $3,800 worth of watches

were stolen from an upscale
boutique.

Police ran a still frame from
the shop's surveillance video

through their facial recognition
system to find a match.

How do I turn a face into
numbers

that equations can act with?

You turn the individual pixels
in the picture of that face

into values.

What it's really looking for
are complex patterns

across those pixels.

The sequence of taking a
pattern of numbers

and transforming it

into little edges and angles,

then transforming that into eyes
and cheekbones and mustaches.

NARRATOR:
To find that match,

the system can be trained on
billions of photographs.

Facial recognition uses a class
of machine learning

called deep learning.

The models built by deep
learning techniques

are called neural networks.

VENKATASUBRAMANIAN:
A neural network

is, you know, stylized as, you
know, trying to model

how neural
pathways work in the brain.

You can think of a neural
network

as a collection of neurons.

So you put some values into a
neuron,

and if they're, sufficiently,
they add up to some number,

or they cross some threshold,

this one will fire and send off
a new number to the next neuron.

NARRATOR:
At a certain threshold,

the neuron will fire to the
next neuron.

If it's below the threshold,
the neuron doesn't fire.

This process repeats and
repeats

across hundreds, possibly
thousands of layers,

making connections like the
neurons in our brain.

♪ ♪

The output is a predictive
match.

Based on a facial recognition
match,

in January 2020, the
police arrested Robert Williams

for the theft of the watches.

The next day, he was released.

Not only did Williams have an
alibi,

but it wasn't his face.

MING:
To be very blunt about it,
these algorithms are probably

dramatically over-trained on
white faces.

♪ ♪

So, of course, algorithms that
start out bad

can be improved, in general.

The Gender Shades project found
that

certain facial recognition
technology,

when they actually
tested it on Black women,

it was 65% accurate, whereas for
white men, it was 99% accurate.

How did they improve it?
Because they did.

They built an algorithm

that was trained on more diverse
data.

So I don't think it's
completely a lost cause

to improve algorithms to be
better.

MAN (in ad voiceover):
I used to think my job
was all about arrests.

NEDY:
LESLIE KEN
There was a commercial a few
years ago

that showed a police officer
going to a gas station

and then waiting for the
criminal to show up.

MAN:
We analyze crime data,

spot patterns,

and figure out where to send
patrols.

They said, "Well, our algorithm
will tell you exactly

where the crime, the next crime
is going to take place."

Well, that's just silly, uh,
it, that's not how it works.

MAN:
By stopping it before it
happens.

(sighs)

MAN:
Let's build a smarter planet.

♪ ♪

JOEL CAPLAN:
Understanding what it is about
these places

that enable
crime problems

to emerge and/or persist.

NARRATOR:
At Rutgers University,

the researchers who invented

the crime mapping platform
called Risk Terrain Modeling,

or RTM,

bristle at the term
"predictive policing."

CAPLAN (voiceover):
We don't want to predict,
we want to prevent.

I worked as a police officer a
long time ago,

in the early 2000s.

Police collected data for as
long as police have existed.

Now there's a greater
recognition

that data can have value.

But it's not just
about the data.

It's about how you analyze it,
how you use those results.

There's only two data sets that
risk terrain modeling uses.

These data sets are local,

current information about crime
incidents within a given area

and information about
environmental features

that exist in that landscape,

such as bars, fast food
restaurants,

convenience stores,
schools, parks, alleyways.

KENNEDY:
The algorithm is basically

the relationship between these
environmental features

and the, the outcome data,
which in this case is crime.

The algorithm provides you with
a map

of the distribution of
the risk values.

ALEJANDRO GIME NEZ-SANTANA:
This is the highest-risk area,

on this commercial corridor on
Bloomfield Avenue.

NARRATOR:
But the algorithm isn't intended
for use just by police.

Criminologist Alejandro
Giminez-Santana

leads the Newark Public Safety
Collaborative,

a collection of 40 community
organizations.

They use RTM as a diagnostic
tool

to understand not just where
crime may happen next,

but why.

GIME NEZ-SANTANA:
Through RTM, we identify this
commercial corridor

on Bloomfield Avenue, which is
where we are right now,

as a risky area for auto theft
due to car idling.

So why is this space

particularly problematic when it
comes to auto theft?

One is because we're in a
commercial corridor,

where there's high density of
people

who go to the beauty
salon or to go to a restaurant.

Uber delivery and Uber Eats,

delivery people who come to grab
orders that also,

and leave their cars
running

create the conditions for this
crime

to be concentrated in this
particular area.

What the data showed us was,

there was a tremendous rise in
auto vehicle thefts.

But we convinced the police
department

to take a more
social service approach.

NARRATOR:
Community organizers convinced
police

not to ticket idling cars,

and let organizers create

an effective public awareness
poster campaign instead.

And we put it out to the Newark
students

to submit in this flyer
campaign,

and have their artwork on the
actual flyer.

GIME NEZ-SANTANA:
As you can see, this is the
commercial corridor

on Bloomfield Avenue.

The site score shows a six,

which means that we are at the
highest risk of auto theft

in this particular location.

And as I move closer to the end

of the commercial corridor, the
site risk score is coming down.

NARRATOR:
This is the first time
in Newark

that police data for
crime occurrences

have been shared widely with
community members.

ELVIS PEREZ:
The kind of data we share
is incident-related data--

sort of time, location,
that sort of information.

We don't discuss any private
arrest information.

We're trying to avoid a crime.

NARRATOR:
In 2019,

Caplan and Kennedy formed a
start-up at Rutgers

to meet the rising demand for
their technology.

Despite the many possible
applications for RTM,

from tracking public health
issues

to understanding vehicle
crashes,

law enforcement continues to be
its principal application.

Like any other technology,

risk terrain modeling can be
used for the public good

when people use it wisely.

♪ ♪

We as academics and scientists,
we actually need to be critical,

because it could be the
best model in the world,

it could be very good
predictions,

but how you use those
predictions

matters, in some ways,
even more.

REPORTER:
The police department

had revised the SSL numerous
times...

NARRATOR:
In 2019,

Chicago's inspector general
contracted the RAND Corporation

to evaluate the Strategic
Subject List,

the predictive policing platform

that incorporated Papachristos's
research on social networks.

PAPACHRISTOS:
I never wanted to go down this
path

of who was the person that
was the potential suspect.

And that problem is not
necessarily

with the statistical model,
it's the fact that someone

took victim and made him an
offender.

You've criminalized
someone who is at risk,

that you should be prioritizing
saving their life.

NARRATOR:
It turned out that some 400,000
people were included on the SSL.

Of those,
77% were Black or Hispanic.

The inspector general's audit
revealed

that SSL scores were unreliable.

The Rand Corporation found the
program had no impact

on homicide or victimization
rates.

(protesters chanting)

NARRATOR:
The program was shut down.

But data collection continues

to be essential to law
enforcement.

♪ ♪

O'NEIL:
There are things
about us that we might not even

be aware of that are sort of
being collected

by the data brokers

and will be held against us for
the rest of our lives--

held against people forever,
digitally.

NARRATOR:
Data is produced and collected.

But is it accurate?

And can the data be
properly vetted?

PAPACHRISTOS:
And that was one of the
critiques

of not just the
Strategic Subjects List,

but the gang database in
Chicago.

Any data source that treats data
as a stagnant, forever condition

is a problem.

WOMAN:
The gang database has been
around for four years.

It'll be five in January.

We want to get rid of
surveillance

in Black and brown communities.

BENJAMIN:
In places like Chicago,

in places like L.A.,
where I grew up,

there are gang databases
with tens of thousands

of people listed, their names
listed in these databases.

Just by simply having a certain
name

and coming from a certain
ZIP code

could land you in these
databases.

Do you all feel safe
in Chicago?

DARRELL DACRES:
The cops pulled up out of
nowhere.

Didn't ask any questions, just
immediately start beating on us.

And basically were saying,
like,

what are, what are we doing over
here, you know, like,

in this, in this gangbang area?

I was already labeled as a
gangbanger

from that area because of where
I lived.

I, I just happened to live
there.

NARRATOR:
The Chicago gang database

is shared with hundreds
of law enforcement agencies.

Even if someone is wrongly
included,

there is no mechanism
to have their name removed.

If you try to
apply for an apartment,

or if you try to
apply for a job or a college,

or even in a, um, a house, it
will show

that you are in this record of
a gang database.

I was arrested for peacefully
protesting.

And they told me that, "Well,

you're in the gang database."

But I was never in no gang.

MAN:
Because you have a gang
designation,

you're a security threat group,

right?

NARRATOR:
Researchers and activists

have been instrumental in
dismantling

some of these systems.

And so we continue to push
back.

I mean, the fight is not going
to finish

until we get rid of
the database.

♪ ♪

FERGUSON:
I think what we're seeing now

is not a move away from data.

It's just a move away from this
term "predictive policing."

But we're seeing big
companies,

big tech, enter the policing
space.

We're seeing the reality that
almost all policing now

is data-driven.

You're seeing these same police
departments

invest heavily in the
technology,

including other forms of
surveillance technology,

including other forms
of databases

to sort of manage policing.

(chanting):
We want you out!

NARRATOR:
More citizens are calling for
regulations

to audit algorithms

and guarantee they're
accomplishing what they promise

without harm.

BRAYNE:
Ironically, there is very
little data

on police use of big data.

And there is no systematic data

at a national level
on how these tools are used.

The deployment of these tools

so far outpaces

legal and regulatory responses
to them.

What you have happening

is essentially this regulatory
Wild West.

O'NEIL:
And we're, like, "Well,
it's an algorithm,

let's, let's just throw it
into production."

Without testing it to whether

it "works" sufficiently,
um, at all.

NARRATOR:
Multiple requests for comment

from police agencies and
law enforcement officials

in several cities, including
Chicago and New York,

were either declined or went
unanswered.

♪ ♪

Artificial intelligence must
serve people,

and therefore artificial
intelligence

must always comply
with people's rights.

NARRATOR:
The European Union is preparing
to implement legislation

to regulate artificial
intelligence.

In 2021, bills to regulate data
science algorithms

were introduced in 17 states,

and enacted in Alabama,
Colorado,

Illinois, and Mississippi.

SWEENEY:
If you look carefully on
electrical devices,

you'll see "U.L.," for
Underwriters Laboratory.

That's a process that came
about

so that things, when you plugged
them in,

didn't blow up in your hand.

That's the same kind of idea

that we need in these
algorithms.

O'NEIL:
We can adjust it to make
it better than the past,

and we can do it carefully,

and we can do it with, with
precision

in an ongoing conversation about
what it means to us

that it is, it's biased in the
right way.

I don't think you remove bias,

but you get to a bias that you
can live with,

that you, you think is moral.

To be clear, like, I, I think we
can do better,

but often doing better

would look like we don't use
this at all.

(radio running)

FARID:
There's nothing fundamentally
wrong

with trying to predict the
future,

as long as you understand how
the algorithms are working,

how are they being deployed.

What is the consequence
of getting it right?

And most importantly is,

what is the consequence of
getting it wrong?

OFFICER:
Keep your hands on
the steering wheel!

MAN:
My hands haven't moved
off the steering wheel!

MAN 2:
Are you gonna arrest me?

MAN 1:
Officer, what are we here for?

OFFICER:
We just want to talk with...

♪ ♪

ANNOUNCER:
This program is available
with PBS Passport

and on Amazon Prime Video.

♪ ♪

♪ ♪