All language subtitles for Secrets.Of.Big.Data.S01E05.WEBRip.x264-ION10

♪

Narrator: The U.S. government
suffers the most invasive

cyberattack in its history.

Nikolas Badminton: The hackers
broke into email accounts

affiliated with the head of the
Department of Homeland Security.

Ramona Pringle:
You would think they'd have

all sorts of
safeguards in place.

Narrator: Job hunters across
the world are facing

stressful interviews conducted
not by a person,

but by an algorithm
that scrutinises

their every word and gesture.

K. Alexander: It analyzes their
facial expressions;

how much eye contact they make,
and even their tone of voice.

M. Aitken: They're essentially
trying to impress a machine.

Narrator: In India, a research
student is using new technology

to help deaf people
communicate in a way

that they have never
been able to do before.

Aitken: She wants to invent an
AI program that can translate

visual 'sign language'
into English text.

Anthony Morgan: And it isn't
just limited to people

with hearing conditions.
AI can help people

who are blind or have
other disabilities.

Narrator: These are the
stories of the future

that big data is bringing
to our doorsteps.

The real world impact of
predictions and surveillance.

The power of artificial
intelligence

and autonomous machines.

For better or for worse,

these are the
Secrets of Big Data.

Narrator: In late 2020, an
employee at the Silicon Valley

headquarters of FireEye
in California,

one of the most respected and
successful cybersecurity firms

in the United States, is doing
a routine systems check

when she notices something
out of the ordinary.

Alexander: One of their
employees seems to have

two phones registered to
his network.

Narrator: While this may appear
insignificant to an outsider,

FireEye's clientele includes
some of the world's

biggest companies and top-level
government institutions,

which makes them a constant
target for cyber-espionage.

Badminton: Because of their
high profile,

FireEye is always under
threat of attack.

That's why anything unusual
in their systems

is cause for concern.

Narrator: The employee that
registered the second phone

is contacted and tells the
security team

that he has no idea why
there is another number

attached to his network.

Pringle: Alarm bells
start to go off.

FireEye can only conclude that
they've been compromised

and somebody has accessed
their systems.

Narrator: The company
immediately launches

an investigation.

After several weeks of analysis,
FireEye discovers

that not only did someone breach
their network,

but they also stole hacking
applications

that the company employs to
assess the safety

of its own clients' networks.

Alexander: This is very bad.
The tools that they stole could

be used to stage sophisticated
new attacks around the world.

Narrator: FireEye is able to
trace the intrusion back

to something seemingly harmless,

a routine software update from a
company called SolarWinds,

a leading provider of system
management tools

for network and infrastructure
monitoring.

Badminton: SolarWinds is a
major player in the space,

with hundreds of thousands
of customers around the world.

Narrator: The software,
called Orion,

is a popular network
management system.

To update it, users were
prompted to log into

the SolarWinds' development
website,

enter their password and then
the new software would be

automatically integrated
into their servers.

Alexander: On the surface,
it's a pretty standard update.

Some bug fixes, performance
improvements and such.

Narrator: But below the surface,
it's anything but standard.

Someone has managed to
insert a malicious code

into the Orion software update,

and unaware of this, some 18,000

SolarWinds customers downloaded
the tainted product.

Once the update was completed,
the perpetrators were able

to gain access to other
companies and organisations

that these customers used and
even worked for.

Including tech giants Intel,

Cisco and Microsoft.

Pringle: But what's more
concerning is that a number

of US federal agencies
are also compromised,

including the Treasury, Justice
and Energy departments

and even the Pentagon.

Narrator: The SolarWinds hack
is one of the largest and most

sophisticated cybersecurity
breaches of the 21st century.

Authorities begin to
investigate, trying to ascertain

who is behind this brazen attack

and how exactly they managed to
execute it.

Badminton: The SolarWinds hack
is what's called

a supply-chain attack.

Rather than trying to breach a
company or institution directly,

hackers identify a third party
vendor with weak cybersecurity

and use them to gain access.

Alexander: As there are many
possibilities for

who that third party is,
there are also

a few different types
of supply chain attacks.

But one common trick is to
breach businesses

that build websites.

Narrator: In a website
builder attack,

hackers compromise
companies who use

ready-made templates
to create websites,

usually digital ad agencies
or developers.

Once the business is breached,

the attackers manipulate the
core script of the template,

redirecting victims
to a corrupt domain.

Malware is then installed onto
the systems of those

browsing legitimate websites.

Pringle: Builder attacks
are very efficient

because instead of targeting a
bunch of websites individually,

hackers can gain access
to any site that uses the

doctored code, all by gaining
access to just one company.

Badminton: What we're also
seeing more and more of

are so-called
"watering hole attacks",

where hackers single out a
website that's visited often

by employees of a
certain organisation

or even a whole sector
like healthcare or defence.

Narrator: Once the target
website of a watering hole

attack is compromised, the
perpetrators distribute malware,

sometimes without the
victims realising it.

But because users trust the
site, it can also be hidden

in a file that they
deliberately download,

unaware of the malicious
content it contains.

In 2021, Google's Threat
Advisory Group discovered

a watering hole attack that
breached several media

and pro-democracy websites to
target visitors

specifically from Hong Kong.

Cybersecurity experts suspect
that the Chinese government

was behind the attack.

Pringle: Hackers are now using
'watering hole' sites

for cyber attacks against
an array of victims

in many different sectors.

Narrator: But the most popular
supply chain attack method

is third-party software
interference,

as witnessed in the
SolarWinds hack.

And as the investigation
into the breach deepens,

the ingenuity and complexity
of the operation

becomes evident to authorities.

Alexander: They discover that
the hackers originally gained

access to SolarWinds over a year
before the attack was exposed.

As a sort of trial run, they
inserted a small snippet

of harmless code into
a software update

to see what they could
get away with it.

Badminton: Once that version of
the software was published

and distributed with
their code still intact,

they knew that a full-scale
attack was possible.

Narrator: On the heels
of this victory,

the attackers then
do something strange,

vanish for five months.

Alexander: Presumably they were
working on writing the code

for the main operation,
because when they reappear,

they come equipped with
a backdoor attack

the likes of which
the world has never seen.

Narrator: Investigators are
stunned when they discover

exactly how the perpetrators
managed to introduce

the tainted code into the
SolarWinds software update.

The first step was to embed code
that informed them

whenever an employee on the
development team

was preparing new software.

Pringle: These companies
have a digital library,

and every time they
engineer an update,

the developer has
to check the code out

and then when they're done
modifying it,

they check it back in.

Badminton: This creates a
digital trail so it's easy

to track who has had
access to the files

and when they were worked on.

Narrator: Once an
update is complete,

what's called a build
process is started,

which converts the code
from human language

to computer language.

The finished software is
then stamped with

what could be described
as a digital seal,

which in most cases
makes it impossible

to tamper with without someone
being alerted.

Alexander: The hackers were
able to study the SolarWinds

build process and
sneak the code in

at the very last second
so it went undetected.

Badminton: In real world terms,
is like if someone slipped

a poison pill into a bottle
of aspirin at the factory,

only a moment before the
bottle was sealed shut.

Narrator: The resulting
malicious software update

was then unknowingly sent out
to SolarWinds customers,

giving the attackers total
anonymous access

to any Orion user who
installed the update

and had an internet connection.

Pringle: The code itself was
sophisticated but brief,

only 3,500 encrypted
characters long.

The best hackers are very
economical in their programming,

the more concise the code,
the harder it is to detect.

Narrator: As investigators
unravel how the operation

was carried out, they begin to
search for clues

as to who was behind it, but
determining the identity

of the attackers is proving
difficult.

Badminton: A lot of hackers
inadvertently leave evidence

behind, some have coding tics
that give them away based on

known previous attacks or they
might even write something

in their native language, which
can give away their nationality.

Narrator: But the SolarWinds
code is so sophisticated,

that there are no clues
to its origin.

Authorities can't find any
evidence to pin down exactly

where the attack came from, but
they have their suspicions.

In 2017, the most damaging and
expensive cyberattack in history

was allegedly perpetrated
by the Russian military.

The hack, called NotPetya, also
used corrupted software

as a delivery method.

Alexander: NotPetya was
originally a cyber-weapon used

against Ukraine by the Russians.

They breached tax software
called M.E. Doc

that is widely used by
Ukrainian businesses.

From there, it spread like
wildfire around the world.

Narrator: Investigators
discovered that NotPetya

had been specifically programmed
to make it impossible

to recover any files once
systems were infected.

It was designed to
completely destroy

any computer that
it infiltrated.

The malware targeted everything
from energy companies,

the power grid and gas
stations to airports,

banks and major corporations.

Pringle: The US government
assessed that the

NotPetya attack ended up causing
about $10 Billion USD

worth of damages world wide,
making it the most expensive

and destructive
cyberattack in history.

Narrator: US authorities begin
to suspect that Russia

may be behind the SolarWinds
hack as well, because both

used tainted software code as
a launching point.

But the problem for
investigators is

that the similarities end there.

Pringle: NotPetya was hell-bent
on destroying everything

in its path, but the SolarWinds
attack was done covertly.

They were also selective about
what institutions to target.

Narrator: Breaches as large
as the SolarWinds hack

can present an embarrassment of
riches for attackers.

With so many options, it can be
hard for them to narrow down

which companies or government
agencies they want to access.

Badminton: What the attackers do
is create a passive domain name

server system that not only
identifies potential targets

by IP address, but also
gives them a little bit

of information about each one.

Pringle: The hackers then choose

which targets are worthy
of their attention.

They mostly go
after tech companies

and high-profile branches
of the government.

Narrator: The SolarWinds
attackers manage to breach

about a dozen vital US
government agencies.

Disturbingly, they even break
into the Cybersecurity

and Infrastructure Security
Agency, or CISA,

the office at the Department
of Homeland Security

whose primary function is to
defend government networks

from cyberattacks.

Alexander: For them to breach
the very agency whose job it is

to defend against
these kinds of attacks

is a major embarrassment
for US authorities.

How does that even happen?

Narrator: According to the
Department of Homeland Security,

their system only detects known
threats and the SolarWinds

attack is unlike anything
they've ever seen before.

On top of that issue, their
processes don't involve

scanning software updates.

So even if the system could have
identified the malicious code,

they never would
have detected it,

because it was buried
in the update.

Pringle: That seems like a
pretty big oversight on the part

of the government,
who you would think would have

all sorts of
safeguards in place.

Narrator: As investigators
dig deeper,

they discover something
truly shocking.

The attackers had unfettered
access to some of these systems

for an astonishing nine months
before the hack was detected.

Badminton: Interestingly, the
hackers didn't seem to disrupt

any systems or
destroy any files.

They just kind of
silently roamed around.

Which points to one
thing - cyber espionage.

Narrator: Authorities now have
the motive for the attack,

but are still unable to find
any evidence

as to who perpetrated it.

But they can't help but come
back to their prime suspect:

Russia.

More specifically, one of the
most tenacious and cunning

hacking groups on the planet:

the Russian Intelligence-backed
APT29,

also known as Cozy Bear.

Pringle: Cozy Bear has been
responsible for some of the most

infamous hacks of
US and NATO member countries

over the past several years.

They're the cream of the crop.

Narrator: In 2016, WikiLeaks
released 20,000 emails

from the Democratic
National Committee

that they acquired after Cozy
Bear and another hacking team

believed to be tied to a
separate branch

of the Russian
intelligence service,

accessed the DNC's
internal network.

Cozy Bear camped out in the
system undetected

for over a year, actions
suspiciously similar

to the SolarWinds hack.

And the resemblance
doesn't stop there.

Badminton: Both of these
attacks have common thread,

they use cutting
edge digital tools,

and this suggests state funding.

They went after strategic
information,

rather than financial gain;

and they chose
targets of interest

to Russia's
intelligence community.

Narrator: For its part, Russia
denied any involvement

in the SolarWinds hack.

But the US was unconvinced

and imposed sanctions on them as
punishment for the attack.

Pringle: The U.S.
ultimately announced

that 10 Russian diplomats would
be expelled from the country

and 32 entities and individuals
would be blacklisted.

The sanctions also targeted
six Russian tech firms

linked to intelligence services.

Narrator: The full extent
of the damage caused

by the SolarWinds hack remains
something of a mystery.

For the government agencies that
were breached,

it's virtually impossible for
them to know the sum total

of the information the
Russians had access to.

Badminton: They do know for sure
that the hackers broke into

email accounts
affiliated with the

Head of the Department of
Homeland Security

and also several others who work

in the department's
cybersecurity division.

Narrator: For private companies,
the impact is also murky.

Microsoft reported no evidence
of stolen or leaked

customer data from the attack.

Alexander: It seems pretty
clear that the US government

was the primary target
of the attack.

The tech companies were probably
just collateral damage.

Narrator: Up against this kind
of formidable enemy,

American authorities face
difficult questions.

Was the attack just
the tip of the spear

in the escalating cyberwar
between familiar

cold war adversaries?

Is another, more destructive
hack waiting around the corner?

Most experts agree - it's
not a matter of

if it happens again, but when?

♪ [show theme music]

♪♪

Narrator: 25-year-old New York
native Sheikh Ahmed

is on the hunt for a job
as a bank teller.

He has applied for many
positions across the city,

and today he gets the news that
he has been selected

for not just one,
but eight interviews.

However, these would be no
ordinary meetings with

Human Resources representatives
or branch managers.

These are HireVue assessments,

a state of the art
recruiting tool

that uses artificial
intelligence

to assess a candidate's
worthiness

with no prospective
employer present.

Badminton: The Hirevue system
uses a person's phone

or computer camera to scrutinize
the smallest details

of their answers to a
standard set of questions.

Alexander: It analyzes their
facial expressions;

how much eye contact they make,

what words they use and even
their tone of voice.

HireVue claims that by using
these metrics,

they can determine how
enthusiastic a person is

about getting the job.

Narrator: This information is
then used to automatically

produce an employability score,

which is ranked against
other candidates.

The HireVue algorithm is part of
a burgeoning field

of artificial intelligence
called ERT,

Emotion Recognition Technology.

Aitken: ERT basically tries to
identify how someone is feeling

based on their facial
expressions

and other physical clues.

Narrator: These systems rely on
two factors - computer vision,

to accurately recognize
facial movements,

and machine learning to analyze
and decipher them.

The algorithms reference huge
image databases of human faces

that are classified by emotion,

and then try to match
them to the subject.

Badminton:
Six basic feelings are used:

fear, anger, joy, sadness,

disgust, and surprise.

Narrator: Pioneering American
Psychologist Doctor Paul Ekman

was the first to categorize
these as the fundamental

human emotions back
in the 1960s.

Ekman is considered to be the
founding father of ERT,

and his early research still
echoes in today's

sophisticated artificial
intelligence systems.

Alexander: He believed
that these were the

universal feelings
that all humans shared,

regardless of gender, culture,

location or situation.

Narrator: In 1978, he published

the the Facial Action
Coding System or FACS.

Alexander: The FACS system
categorized around 40 unique

muscle movements of the face
and called the elements of each

expression an action unit.

Narrator: For the most part,
FACS was a resounding success,

but there were issues.

The central problem was that it
was very time consuming to use,

it took up to 100 hours to teach
users the procedures,

and an hour to evaluate only one
minute of film footage.

But a promising new technology
that might help overcome

these obstacles was on the
horizon,

computer vision.

Aitken: In the early '90s,
researchers realized

that in order to take advantage
of advancing technology,

they needed a database of
standardized images

to work with.

Badminton: At this point, the
US government stepped in

and financed a program to
compile facial pictures.

They saw the potential for ERT
as a security application.

Narrator: By the
end of the 1990s,

machine-learning scientists
began to collect

and classify these archives,

resulting in robust image
datasets that provide

the foundation for much of
today's AI based research.

And emotion recognition
technology

is quickly becoming big
business.

Alexander: One early
provider of ERT services

was a startup called Affectiva.

Their technology was
sold to businesses

as a market research product,

analyzing real-time
emotional reactions to ads

and new products
in focus groups.

Narrator: ERT has since expanded
into many other areas

of business, particularly
recruitment,

where companies like
HireVue claim

that they can streamline the
hiring process

by using their systems to weed
out unworthy applicants

quickly and accurately.

Badminton: A process that
used to take weeks

now only takes a few days.

It's way cheaper and faster
than if humans were involved.

Narrator: In fact, ERT is now
so prevalent in human resources,

that there are online guides
with tips for candidates

on how to best present
themselves to the camera.

For job seekers like Sheikh
Ahmed, the process can be

an intimidating and
distressing experience.

Aitken: They're essentially
trying to impress a machine.

It really is kind of strange.

Narrator: Ahmed has spent
countless hours studying guides

on how to speak and
comfort himself,

but on the day of the interviews

he frets over something
seemingly trivial,

how to position the camera.

Alexander: A high angle might
make him seem weak and small,

whereas a low angle might
make him appear too dominant.

Narrator: And there
are other factors

fuelling Ahmed's anxiety,

namely that random sounds
might harm his score.

He turns off the air
conditioning system,

and tucks himself into the
corner of his father's

soundproof music studio, far
away from the normally

pleasant chirping of the
family's pet bird.

Badminton: Because
the software analyses

the sound of people's voices,
any outside interference

could have an impact
on his evaluation.

Narrator: Ahmed settles
into a gruelling day

and confronts the unsettling
reality of facing an algorithm

that judges every involuntary
gesture that he makes

and every word that he utters.

To critics of emotion
recognition technology,

and there are many,
this is problematic.

Most wonder if artificial
intelligence can really

interpret something as complex
and nuanced as human behaviour.

Aitken: Some argue that it's
impossible to know definitively

what a person is feeling
simply by reading

their facial expressions.

People sometimes smile even if
they're not happy

or scowl when they aren't angry.

Narrator: And there
are other problems.

Critics of ERT claim that the
image categorizing process

used to develop algorithms
is overly simplistic.

Something as complicated
as human emotion

can't be distilled down
to six basic feelings.

Badminton: Emotions are complex
and often interrelated.

There are many grey areas.

There are subtleties that no AI
is capable of detecting.

Well, not quite yet!

Narrator: Some are also
quick to point out

that people express emotions
in many different ways,

not just using
facial expressions.

Factors like body language
are also indicators

of how a person is feeling.

Alexander: Physical cues
such as crossed arms

or a slumped posture
can sometimes convey

someone's state of mind better
than the look on their face.

Narrator: But Ekman and his
supporters counter

that the research is sound and
stand by the assertion

that if a universal emotion is
triggered in a person,

then an involuntary facial
movement

naturally appears on their face.

Aitken: So the argument goes
that even if that person

tried to hide their feelings,

the basic,
reflex emotion would surface,

and if someone knew what to look
for, they could identify it.

Narrator: Still, there are
many skeptics who question

the scientific validity of ERT

and have problems with some of
the methodology.

Alexander: One of the issues
people have is that the

image datasets may be
made up of posed faces.

If someone is asked to
make a sad face,

it may look different
from how their face

actually looks when they're sad.

Aitken: It's a valid argument,
so the most recent systems

have started to draw on
images that are candid,

footage of people doing
mundane things like

driving their cars
or watching TV.

Narrator: There has also been
criticism of the forced-choice

answer method of labelling
pictures in datasets.

Because there are limited
options when asked

to ascribe an emotion
to a picture,

there's no room
for interpretation.

Badminton: Someone might look at
an image and think the person

is feeling guilt or shame,

but those feelings may not be on
the list of possible choices.

Narrator: And there
are cultural concerns.

People from different regions
of the world

convey emotions
in different ways.

Alexander: Many people use
smiles to show happiness

but for example,
in Japan some smiles

are simple expressions of
politeness, rather than joy.

So it can be fairly nuanced.

Aitken: But even if one culture
has a slightly different idea

of what a happy face looks like,

most people recognize
joy when they see it,

regardless of where
they're from.

Narrator: In order to mitigate
the effect of cultural nuances,

ERT companies are compiling
more diverse datasets.

Affectiva, one of the
leaders in the field,

boasts a collection of more
than 10 million images

of people's facial expressions
from 87 countries.

And they are always adjusting
their algorithms

to make them more accurate.

Badminton: What these companies
are now doing is including

an element of analysis
to their systems.

So that rather than just
identifying an emotion,

the AI is able to apply a
cultural context

when classifying it.

Narrator: Context is another
issue that critics of ERT

take umbrage with.

In 1972, Paul Ekman conducted
an experiment

to study the differences between
how Japanese

and American audiences reacted
to a horror film.

And found that Japanese people
showed less negative expressions

when there was an authority
figure in the room.

Alexander: Different cultures
have their own set of rules

about who can show
certain emotions to whom.

In this case, the Japanese
audience probably behaved

differently because they knew
there was someone there

who may have been judging them.

Narrator: And for people like
Sheikh Ahmed,

being judged by an Artificial
Intelligence system

while merely trying to find a
job would certainly

have an effect on one's
behaviour.

Aitken: Ahmed altered his
responses slightly

over the course of the
eight interviews that day.

I guess he thought that if he
gave the algorithm a variety

of answers, it might increase
his chances of a positive score.

Narrator: By the
end of the ordeal,

an exhausted Ahmed is drenched
in sweat, his mouth is dry

and he can't shake the feeling
that he hadn't made enough

eye contact with the camera or
said the right things.

Alexander:
Not enough eye contact?

You're shy
and have no confidence.

Too much eye contact,

and you're
aggressive and too intense.

Badminton: As difficult
as the process may be,

the reality is that ERT
in recruitment is only

going to become more common
as the technology advances.

Narrator: And ERT is
gradually creeping its way

into other fields as well.

The latest sector to feel
its touch is education.

True Light College, a secondary
school for girls in Hong Kong,

used ERT software to evaluate
students' faces

as they learned remotely
during the pandemic.

Alexander: The developers say
that the system helps teachers

make learning more
engaging and personal,

by reacting to a student's
expressions in real time.

It even sends them alerts if
they seem distracted or bored.

Narrator: The company behind the
software claims

that it is able to correctly
decipher a child's

emotional state about
85% of the time

and demand for the program
has increased dramatically,

with the number of schools using
it in Hong Kong

more than doubling
from 34 to 83.

Aitken: The whole thing seems
really invasive to me.

These are kids, after all.

Do we really need
to be monitoring

their faces as they learn?

Narrator: Elsewhere in China,
ERT is being used

for even more intrusive
purposes.

Cameras with emotion
recognition systems

have been installed in Xinjiang,

the region where an estimated
1 million mostly Uyghur Muslims

are being detained
in prison camps.

Chinese authorities believe that
their algorithms are able to

identify potential criminals by
determining their mental state.

Badminton: It's kind of the next
step in the evolution of ERT.

Some believe that not only can
the AI detect how a person is

feeling, but it's even able to
predict their future actions

and give an overall impression
of their personality.

Aitken: But there's no real
evidence that these systems

are even remotely accurate.

They're based on very
vague so-called science.

Narrator: In 2018, a
controversial study

out of Stanford University
in California

even went so far as to declare
that

facial analysis is capable of
identifying

a person's sexuality.

Using a dataset of over
35,000 images

taken from dating websites,

a machine-learning system
was able to differentiate

between pictures of gay
and straight people

with surprising accuracy.

Alexander: The program was
able to correctly categorize

81% of cases involving
images of men

and 74% of photographs of women.

When humans did the same test,
those numbers dropped

by 20% across both genders.

The researchers were actually
quite shocked at how easy it was

for the algorithm to
make the distinction.

Narrator: The authors of the
study concluded

that there is mounting
scientific proof

that there may be connections
between faces and psychology

that are impossible to detect
with the human eye,

but are identifiable
to machine learning systems.

Badminton: Several prominent
LGBT organizations

were not happy and demanded
that Stanford distance itself

from the research, calling
it dangerous and flawed.

Aitken: People were angry,
because it is without question

an international
human rights issue.

This kind of technology could
be used to expose people as gay

whether accurately
or inaccurately.

And in countries like
Saudi Arabia and Iran,

where homosexuality is
punished by execution,

that is very dangerous!

Narrator: While the
controversies around

emotion recognition
technology swirl,

the industry shows no sign
of slowing down.

Some estimates project
that it will reach

$37 billion US dollars by 2026,

up from $19.5 billion in 2020.

And it's not just
plucky startups

looking to get a
piece of the pie.

Tech industry titans like Apple,
Microsoft and Amazon are all

investing heavily in developing
their own ERT products.

Badminton: Obviously these
companies see something of value

in the technology,
but I think there will always be

lingering questions about
its scientific integrity.

Narrator: Meanwhile,
back in New York,

Sheikh Ahmed waits nervously

to finally find out
if he got a job.

♪ [show theme music]

♪♪

♪

Narrator: In a Baron County,
Wisconsin courtroom,

48-year-old Paul Zilly

is about to receive
his sentence.

The stakes are pretty high,

he'll either be sent to prison
or be given probation.

A significant factor guiding
the judge's decision

will be determining whether or
not he is likely to commit

another crime in the future.

Morgan: He had been arrested a
few months earlier

for stealing a lawn mower
and some other tools.

And he plead guilty
to all of the charges.

Narrator: Before his appearance
in court,

the county prosecutor
offered him a plea deal:

One year in jail and
follow-up supervision

to make sure that he
doesn't reoffend.

Aitken: His court appointed
attorney agrees to the terms,

saying that a long jail term
isn't in his client's

best interest because he's
not a career criminal...

In other words, the attorney
doesn't think

it's likely he will reoffend.

Narrator: Unfortunately,
it doesn't turn out

the way he expects,

Wisconsin judges are now looking
to a new tool

to help determine if criminals
will reoffend:

An artificial intelligence
risk-assessment program

that is designed to predict
future behavior

of convicted criminals.

Morgan: Wisconsin is
one of the first US states

to integrate it into their
criminal justice system.

Narrator: To Zilly's surprise,

it has rated him, "High Risk"

for committing violent
crime in the future

and 'Medium Risk' overall
as a potential reoffender.

Based on the algorithm's
prediction,

the judge overturns the
prosecution's plea deal

and sentences Zilly
to two years in county jail.

[gavel strikes]

Aitken: He is
completely shocked.

The judge doubled
his prison time.

He thought he was only
going to serve a year.

Narrator: Zilly is adamant
he won't reoffend,

but some people working within
the justice system are confident

that AI can accurately and
fairly predict

if someone will commit
a crime in the future.

Badminton: Over the
past few years

several tech companies
have used AI

in the development of
'Risk Assessment' software

that can be licensed by
various judicial systems.

Narrator: The software's
big selling point

is that it's able to mimic
the problem-solving

and decision-making capabilities
of the human mind.

Using machine-learning
algorithms,

it analyzes existing data to
detect patterns

and predict the likelihood that
crimes will occur in the future.

Morgan: It's kind of like how a
bookie determines the odds

for a sporting event or how
pollsters figure out

who might win an election.

Narrator: AI risk assessment
programs are now being used

by at least 16 different
European countries

and almost every US state.

Aitken: Legal agencies are
seriously understaffed,

Courts are deluged
with criminal cases,

and so there is a backlog of
cases waiting to be heard.

So often while
waiting for trial,

people have to wait in prison,

which only contributes
to overcrowding.

Narrator: The AI
programs are designed to help

alleviate these problems.

Punitive decisions are
difficult to make,

and made more so when
judges are flooded

with many complex cases
that require

a lot of knowledge and context.

The hope is that AI
will help judges,

making the process more accurate
and more efficient.

Badminton: It's not easy.
The reality is that they may

end up making a terrible mistake

by granting a dangerous criminal
parole or on the flip-side,

sentencing a person to prison
when probation would be better.

So essentially the software
is being used

to minimise these
kinds of errors.

Narrator: One of America's
leading risk assessment firms

claims that there are studies
proving that the technology

is more accurate than human
judges in predicting

a criminal's likelihood of
reoffending.

In 2020, researchers at
Stanford University

and UC Berkeley in California

discovered that when assessing
things as complex

as a criminal justice system,

AI is up to 30% more accurate

with its decisions than the
judges they surveyed.

Morgan: Critics challenge
these claims, they say

that there is not enough
evidence to prove that

these technology can actually
improve decision-making.

Narrator: In one example,
a 54 year old Florida man

with an extensive
criminal record

involving aggravated assault,

multiple thefts and
felony drug trafficking,

was arrested for shoplifting
and surprisingly,

the algorithm rated him
'low risk' for reoffending.

Aitken: Judging by
his criminal history,

you would probably
think the opposite.

But this could
indicate a problem

in how the software
assesses risk.

Narrator: It may also indicate
Paul Zilly is not actually

at 'High Risk' of reoffending
and was unfairly sentenced.

Badminton: The algorithms
look at police records

and court documents
to see if the individual

has any prior arrests
or convictions.

Those reports also present
other relevant information

to the algorithm,
like for example,

if the individual has
a history of substance abuse.

Narrator: It turns out,
Paul Zilly has a history

of drug abuse.
Before he was arrested,

he was struggling
with an addiction

to crystal methamphetamine.

He had told police he intended
to sell the items he stole

to fuel his drug habit.

Aitken: This definitely may
have contributed to him

being rated high risk
by the algorithm.

Drug addiction is
often related to crime,

as the desperate need
for a substance

leads to desperate measures.

Narrator: The algorithm also
analyses the responses to the

questionnaire Zilly filled out
while he was incarcerated.

It consists of 137 questions
that help determine

if a person is at risk of
reoffending.

Badminton: Some of the questions
are serious ethical quandaries,

for example,
"Does a hungry person

have the right to access food?"

Whereas others are based on
one's subjective opinion,

like "If people make me angry,
I can be dangerous."

It seems some of these questions
may not have a clear answer,

in which case,
why are they using them?

Narrator: According to
Zilly's risk assessment,

he scored poorly on the
questionnaire;

combined with his history of
drug abuse,

he was labelled 'High Risk'.

Morgan: This may help
explain why his sentence

was upped from one year to two.

In response, Zilly's
court-appointed attorney

filed an appeal,
trying to reduce the sentence.

Narrator: There is concern that
the questionnaire may also be

biased in what it specifically
asks of individuals,

like if they are employed,
where they live

and what the crime levels are
like in their neighborhood.

Critics say this could result
in the algorithms

making assessments that
are discriminatory.

Badminton: Since poorer
neighbourhoods often have

higher crime rates
than more wealthy ones,

the algorithm may assume
it's residents

are at a greater risk of
committing a crime

than if they were living
in a rich area.

Narrator: Civil rights activists
believe the technology

could unfairly flag
people of colour,

who statistically, live in
poorer neighbourhoods

with higher crime rates.

And there is compelling evidence
that it is already happening.

Aitken: Recently, a study of the
AI program that had provided

risk scores to offenders in
Broward County, Florida,

found the algorithm incorrectly
flagged black defendants

as future criminals at almost

double the rate
as white defendants.

And white defendants were
incorrectly assessed as low risk

to offend more often than
their black counterparts.

Narrator: Critics cite
the case of an 18-year-old

African-American woman who
was arrested for burglary

in Ft. Lauderdale, Florida.

Despite being a first
time offender,

the algorithm rated
her 'high risk'.

Morgan: Compare this
to the previous summer,

when an older white man from the
same area was arrested

and rated low risk,
despite having been

previously convicted
of armed robbery.

Narrator: This leads civil
rights activists to conclude

that risk assessment programs

may be perpetuating
existent biases,

further compounding
prejudice and racism

in the justice system.

But the companies that license
their software to several US

state justice systems claim that
a person's race

isn't a factor in the
algorithms' risk assessment.

AI advocates believe the
opposite, that the technology

actually makes the criminal
justice system

fairer for people of colour.

Aitken: They say that it cuts
the human, or biased factor out,

meaning that it should give a
more objective,

less-biased evaluation
of each person.

Badminton: But as we just
saw in Florida,

this isn't always the case.

The algorithm perpetuates
existing biases.

Narrator: At Paul Zilly's
appeal hearing,

his lawyer questions
Dr. Tim Brennan,

one of the creators
of the AI software

that assessed his client.

Morgan: Brennan testifies that
his software

wasn't designed to be used in
sentencing.

In fact he didn't want it
involved

in the criminial justice
system at all.

Its purpose was
to help reduce crime,

not to further punish
people like Paul Zilly.

Narrator: In light of
Brennan's testimony,

the judge reduces Zilly's
sentence to 18 months, admitting

that he may have put too much
faith in the algorithm.

Badminton: Here is a case
where the judicial system

relied far too heavily
on this technology,

leading the judge to make an
unfair sentencing decision.

And unfortunately, it's probably
safe to assume

that this isn't the only case
where this has happened.

Narrator: To attempt
to remedy this,

several civil rights lawyers,
UN officials and labor unions

are now lobbying for more
government regulation

of AI's use within
the legal system.

Aitken: It's a
civil rights issue.

Because the technology
can perpetuate biases,

the use of algorithmic tools
in courtrooms can lead to

violations of a person's right
to a fair sentencing.

Narrator: Despite
inherent problems,

several legal analysts believe
its potential for good

far outweighs the harm
it may cause.

Morgan:
The State of Virginia claims

that they've managed to cut down
on their prison populations

by 26% using these algorithms.

They say they'd been able to do
so by releasing people early

who are 'low risk' of
reoffending.

Aitken: Unburdening
the justice system,

while providing people with a
chance to rebuild their lives

outside of prison, is obviously
a benefit to everyone involved.

The question, as always, is

if we can live with the
negative consequences

of employing this technology.

Is it worth it if even one
person is sentenced unjustly?

Maybe not.

Narrator: The case of Paul Zilly
clearly illustrates

the inherent perils of allowing
algorithms to make decisions

regarding something as important
as a person's freedom.

And while it's still unknown
what future impact

AI will have on our
legal systems,

as more courtrooms adopt
this technology,

and barring a proper framework
regulating its use,

it's likely that injustices
will continue

and the controversy surrounding
it will further intensify.

♪

♪ [show theme music]

♪♪

♪

Narrator: At the Vellore
Institute of Technology

in southern India,

19-year-old Priyanjali Gupta

is a 2nd year
engineering student

specialising in data science.

In February 2021, she decides to
take the weekend off

from her studies and visit
her mother in New Delhi.

While there, Priyanjali
confides something to her.

Pringle: She's been thinking
about trying to develop

some kind of new technology
that can help people.

But she doesn't know what
specifically.

Narrator: One day, Priyanjali
has an epiphany of sorts

and realizes that
virtual assistants

which rely on voice commands
such as Alexa and Siri

are not accessible to people
who are deaf...

So she sets out to
create an application

that will be inclusive to people
with hearing disabilities,

using artificial intelligence.

Aitken: She wants
to invent an AI program

that can translate
visual 'sign language'

into English text
and do so in real-time.

Narrator: There are many
different forms of sign language

but Priyanjali's AI
application translates

the most commonly used,
American Sign Language, or ASL.

Which is used by
around 500,000 people

in the US and Canada.

Priyanjali is hopeful she can
get the software to work,

but it will be no easy feat.

It's a complicated
and technical process.

Aitken: The process is called,
'Deep Learning'.

It's where the AI is
trained to perform tasks

by analyzing
large amounts of data,

it's similar to
how human beings learn.

And it does so on a network
called a "neural network."

The more data the algorithms
are able to analyse,

the more they will "learn"
and the more accurate

they will become
in their analysis.

Narrator: In Priyanjali's case,
her machine will need to analyze

different sign language gestures

to be able to learn
what they mean.

It's in the developmental stages
but it could be promising.

Around 70 million people
around the world

use sign language
to communicate,

so it's crucial that
technologies like this exist.

Morgan: And it isn't just
limited to people

with hearing disabilities.

AI can help people who are blind
or have

other physical or cognitive
disabilities.

Pringle: The technology has the
potential to provide

more independence in their
day-to-day lives.

Narrator: There are already many
AI powered tools

available to the disabled.

People with visual impairments
can access talking keyboards,

and use various virtual
assistants like 'Siri'

and 'Alexa' to perform a web
search or write an email.

Pringle: There are also
AI powered applications

that can read out words
on a printed page,

a computer screen or smartphone.

It can even describe
what's on screen,

such as application icons,
photo images and videos.

Narrator: Artificial
intelligence is also having

an impact on helping
those for whom

communication can be
challenging.

Morgan: People with
certain brain injuries

or with conditions like
Parkinson's can have a hard time

speaking in ways that are easy
for others to understand.

But AI algorithms
can take what they are saying

and transform them into
audio or text files

that are easier to understand.

Narrator: The technology can
also assist people

who may be unable
to speak at all.

In Nebraska, Kaden Bowen is a
teenager with cerebral palsy,

it's a condition
which prevents him

from being able to walk or talk.

To help him communicate, he uses
a rudimentary speaking device

with buttons containing
preselected words or phrases.

But recently he and his father
began using Amazon Echo,

a virtual assistant that uses AI
in its voice-control system.

Aitken: Using his
speaking device,

Kaden can have the Echo perform
tasks he wants it to do,

like to call his family members
on their phones,

and ask them, for example,
to take him for a car ride.

It may seem small, but this
provides him with

an ability to communicate
that he didn't have before.

Narrator: One of the most
significant ways

that technology is improving
disabled people's lives

is in transportation.

Mobility is often one of their
most challenging issues.

But AI powered navigation tools
like 'Google Maps'

can help them attain
more autonomy.

Morgan: Apps like these utilize
GPS technology to make it

really easy to visualise the
route you need to take,

all while providing information
about accessibility,

like where ramps
or elevators are.

Narrator: The recent advancement
of self-driving cars

is also a potentially
significant development.

Aitken: People with disabilities
that prevent them

from being able to drive,
might be able to

use self-driving cars to
get around on their own,

providing them
with a degree of independence

they may not have had before.

Narrator: Despite AI's many
positive benefits,

there are experts who are
raising questions about

its potential limitations,
particularly

around the technology's
'financial accessibility'.

According to recent data,
roughly 26 percent

of US citizens with disabilities

are currently living in poverty,

nearly two and a half times
higher than people

who aren't disabled.

And it's more or less the
same for people

living in EU countries.

Morgan: They may be unable to
find work that pays

a decent wage, if they are
able to work at all.

They might not have financial
support from friends or family

and any government
disability funding

might not provide them
enough to live on.

Pringle: So owing to their
economic insecurity,

they may not have
the money to spend

on cutting edge technology
or software,

leaving them unable
to benefit from it.

Narrator: This situation could
be even more challenging

in developing nations where
poverty rates are higher

and the median income is lower.

Aitken: Problems with
access to technology

are sure to be
difficult to address,

because it is
systemic in nature.

Meaning that there are many
contributing factors

and reasons as to why it exists,

making it all the more
difficult to solve.

Morgan: But some are trying.

One company is setting up
an AI interface to help

people with disabilities find
employment opportunities.

It can browse job search results
and even set up interviews,

then can provide the individual
with an interactive

voice response, chatbots,
and voice assistants.

Narrator: And recently,
Microsoft invested $25 million

on a global AI
Accessibility initiative,

funding projects that
develop software

and technologies for
disabled people,

aiming to improve their
independence

and quality of life.

Pringle: It's worth remembering
that as the technology

becomes more available, its
costs will also come down.

And so that will increase
its accessibility

to people with disabilities.

Narrator: Perhaps there is some
hope in the fact that there are

so many young AI developers
like Priyanjali Gupta.

On her webcam, she records
herself doing

several basic sign language
gestures.

Aitken: The AI software will
then interpret the motions

and translate it into
readable English text.

Narrator: The project is still
in its initial phases

and faces some technical
limitations.

But maybe with time, it can
become a full fledged

on screen translator
of sign language.

Pringle: Thankfully, Gupta is
not the only one

developing AI programs
to help people

with hearing disabilities
or impairments.

Narrator: Several tech companies
are developing smartphone

applications that can use
its camera to lip-read.

There are also AI applications
that utilize

Automated Speech Recognition,
or ASR,

which can transcribe the
conversation

of a group of people
in real-time.

Morgan: Something like that
could help people with

hearing disabilities to be
included in a conversation

without even needing
to read lips.

Icing on the cake is that
these algorithms can add things

like punctuation and names of
the person who's speaking.

Narrator: Internet
accessibility is also

a significant issue
for disabled people.

While some websites are now
optimizing their platforms

to allow visually impaired
individuals

to adjust the font size
and colour,

to be more easily seen and read,

a recent study showed that
98% of the world's

top one million websites don't
offer full accessibility.

And there are serious concerns

about the ones that
are accessible,

specifically regarding
online privacy.

Morgan: Many of these
tools are cloud-based,

so it's possible that
information about

a person's disability could be
obtained by a third party.

In addition to being a huge
violation of privacy,

information like this falling
into the wrong hands

leaves the door open
to things like

discrimination or
social exclusion,

or even just online bullying.

Narrator: But many disabled
people are embracing technology,

believing its ability to
help them far outweighs

any potential problems
it may cause,

and for people like
Priyanjali Gupta,

this is all the
encouragement they need.

Morgan: Gupta is currently
able to get her AI technology

to adapt six different sign
language gestures into English.

"Yes", "No",
"Please", "Thank You",

"Hello" and "I Love You".

Narrator: Now in her 3rd year,
the 21-year-old

university student is
researching a new neural network

that will improve the video
analysis done by the AI.

And she's also trying to
secure additional funding

in order to make improvements.

Aitken: She sees her invention
as a small, but very important

step in helping people
struggling with disabilities.

Narrator: As technology
continues to be integrated

into the day-to-day lives of
people with disabilities,

there is real hope that it has
the potential to help them

live more independent and
fulfilling lives.

And as long as the Priyanjali
Guptas of the world

are out there using their
expertise to develop

new and innovative applications,

the future looks
more accessible than ever.