Original subtitles

♪

Narrator: The U.S. government suffers the most invasive

cyberattack in its history.

Nikolas Badminton: The hackers broke into email accounts

affiliated with the head of the Department of Homeland Security.

Ramona Pringle: You would think they'd have

all sorts of safeguards in place.

Narrator: Job hunters across the world are facing

stressful interviews conducted not by a person,

but by an algorithm that scrutinises

their every word and gesture.

K. Alexander: It analyzes their facial expressions;

how much eye contact they make, and even their tone of voice.

M. Aitken: They're essentially trying to impress a machine.

Narrator: In India, a research student is using new technology

to help deaf people communicate in a way

that they have never been able to do before.

Aitken: She wants to invent an AI program that can translate

visual 'sign language' into English text.

Anthony Morgan: And it isn't just limited to people

with hearing conditions. AI can help people

who are blind or have other disabilities.

Narrator: These are the stories of the future

that big data is bringing to our doorsteps.

The real world impact of predictions and surveillance.

The power of artificial intelligence

and autonomous machines.

For better or for worse,

these are the Secrets of Big Data.

Narrator: In late 2020, an employee at the Silicon Valley

headquarters of FireEye in California,

one of the most respected and successful cybersecurity firms

in the United States, is doing a routine systems check

when she notices something out of the ordinary.

Alexander: One of their employees seems to have

two phones registered to his network.

Narrator: While this may appear insignificant to an outsider,

FireEye's clientele includes some of the world's

biggest companies and top-level government institutions,

which makes them a constant target for cyber-espionage.

Badminton: Because of their high profile,

FireEye is always under threat of attack.

That's why anything unusual in their systems

is cause for concern.

Narrator: The employee that registered the second phone

is contacted and tells the security team

that he has no idea why there is another number

attached to his network.

Pringle: Alarm bells start to go off.

FireEye can only conclude that they've been compromised

and somebody has accessed their systems.

Narrator: The company immediately launches

an investigation.

After several weeks of analysis, FireEye discovers

that not only did someone breach their network,

but they also stole hacking applications

that the company employs to assess the safety

of its own clients' networks.

Alexander: This is very bad. The tools that they stole could

be used to stage sophisticated new attacks around the world.

Narrator: FireEye is able to trace the intrusion back

to something seemingly harmless,

a routine software update from a company called SolarWinds,

a leading provider of system management tools

for network and infrastructure monitoring.

Badminton: SolarWinds is a major player in the space,

with hundreds of thousands of customers around the world.

Narrator: The software, called Orion,

is a popular network management system.

To update it, users were prompted to log into

the SolarWinds' development website,

enter their password and then the new software would be

automatically integrated into their servers.

Alexander: On the surface, it's a pretty standard update.

Some bug fixes, performance improvements and such.

Narrator: But below the surface, it's anything but standard.

Someone has managed to insert a malicious code

into the Orion software update,

and unaware of this, some 18,000

SolarWinds customers downloaded the tainted product.

Once the update was completed, the perpetrators were able

to gain access to other companies and organisations

that these customers used and even worked for.

Including tech giants Intel,

Cisco and Microsoft.

Pringle: But what's more concerning is that a number

of US federal agencies are also compromised,

including the Treasury, Justice and Energy departments

and even the Pentagon.

Narrator: The SolarWinds hack is one of the largest and most

sophisticated cybersecurity breaches of the 21st century.

Authorities begin to investigate, trying to ascertain

who is behind this brazen attack

and how exactly they managed to execute it.

Badminton: The SolarWinds hack is what's called

a supply-chain attack.

Rather than trying to breach a company or institution directly,

hackers identify a third party vendor with weak cybersecurity

and use them to gain access.

Alexander: As there are many possibilities for

who that third party is, there are also

a few different types of supply chain attacks.

But one common trick is to breach businesses

that build websites.

Narrator: In a website builder attack,

hackers compromise companies who use

ready-made templates to create websites,

usually digital ad agencies or developers.

Once the business is breached,

the attackers manipulate the core script of the template,

redirecting victims to a corrupt domain.

Malware is then installed onto the systems of those

browsing legitimate websites.

Pringle: Builder attacks are very efficient

because instead of targeting a bunch of websites individually,

hackers can gain access to any site that uses the

doctored code, all by gaining access to just one company.

Badminton: What we're also seeing more and more of

are so-called "watering hole attacks",

where hackers single out a website that's visited often

by employees of a certain organisation

or even a whole sector like healthcare or defence.

Narrator: Once the target website of a watering hole

attack is compromised, the perpetrators distribute malware,

sometimes without the victims realising it.

But because users trust the site, it can also be hidden

in a file that they deliberately download,

unaware of the malicious content it contains.

In 2021, Google's Threat Advisory Group discovered

a watering hole attack that breached several media

and pro-democracy websites to target visitors

specifically from Hong Kong.

Cybersecurity experts suspect that the Chinese government

was behind the attack.

Pringle: Hackers are now using 'watering hole' sites

for cyber attacks against an array of victims

in many different sectors.

Narrator: But the most popular supply chain attack method

is third-party software interference,

as witnessed in the SolarWinds hack.

And as the investigation into the breach deepens,

the ingenuity and complexity of the operation

becomes evident to authorities.

Alexander: They discover that the hackers originally gained

access to SolarWinds over a year before the attack was exposed.

As a sort of trial run, they inserted a small snippet

of harmless code into a software update

to see what they could get away with it.

Badminton: Once that version of the software was published

and distributed with their code still intact,

they knew that a full-scale attack was possible.

Narrator: On the heels of this victory,

the attackers then do something strange,

vanish for five months.

Alexander: Presumably they were working on writing the code

for the main operation, because when they reappear,

they come equipped with a backdoor attack

the likes of which the world has never seen.

Narrator: Investigators are stunned when they discover

exactly how the perpetrators managed to introduce

the tainted code into the SolarWinds software update.

The first step was to embed code that informed them

whenever an employee on the development team

was preparing new software.

Pringle: These companies have a digital library,

and every time they engineer an update,

the developer has to check the code out

and then when they're done modifying it,

they check it back in.

Badminton: This creates a digital trail so it's easy

to track who has had access to the files

and when they were worked on.

Narrator: Once an update is complete,

what's called a build process is started,

which converts the code from human language

to computer language.

The finished software is then stamped with

what could be described as a digital seal,

which in most cases makes it impossible

to tamper with without someone being alerted.

Alexander: The hackers were able to study the SolarWinds

build process and sneak the code in

at the very last second so it went undetected.

Badminton: In real world terms, is like if someone slipped

a poison pill into a bottle of aspirin at the factory,

only a moment before the bottle was sealed shut.

Narrator: The resulting malicious software update

was then unknowingly sent out to SolarWinds customers,

giving the attackers total anonymous access

to any Orion user who installed the update

and had an internet connection.

Pringle: The code itself was sophisticated but brief,

only 3,500 encrypted characters long.

The best hackers are very economical in their programming,

the more concise the code, the harder it is to detect.

Narrator: As investigators unravel how the operation

was carried out, they begin to search for clues

as to who was behind it, but determining the identity

of the attackers is proving difficult.

Badminton: A lot of hackers inadvertently leave evidence

behind, some have coding tics that give them away based on

known previous attacks or they might even write something

in their native language, which can give away their nationality.

Narrator: But the SolarWinds code is so sophisticated,

that there are no clues to its origin.

Authorities can't find any evidence to pin down exactly

where the attack came from, but they have their suspicions.

In 2017, the most damaging and expensive cyberattack in history

was allegedly perpetrated by the Russian military.

The hack, called NotPetya, also used corrupted software

as a delivery method.

Alexander: NotPetya was originally a cyber-weapon used

against Ukraine by the Russians.

They breached tax software called M.E. Doc

that is widely used by Ukrainian businesses.

From there, it spread like wildfire around the world.

Narrator: Investigators discovered that NotPetya

had been specifically programmed to make it impossible

to recover any files once systems were infected.

It was designed to completely destroy

any computer that it infiltrated.

The malware targeted everything from energy companies,

the power grid and gas stations to airports,

banks and major corporations.

Pringle: The US government assessed that the

NotPetya attack ended up causing about $10 Billion USD

worth of damages world wide, making it the most expensive

and destructive cyberattack in history.

Narrator: US authorities begin to suspect that Russia

may be behind the SolarWinds hack as well, because both

used tainted software code as a launching point.

But the problem for investigators is

that the similarities end there.

Pringle: NotPetya was hell-bent on destroying everything

in its path, but the SolarWinds attack was done covertly.

They were also selective about what institutions to target.

Narrator: Breaches as large as the SolarWinds hack

can present an embarrassment of riches for attackers.

With so many options, it can be hard for them to narrow down

which companies or government agencies they want to access.

Badminton: What the attackers do is create a passive domain name

server system that not only identifies potential targets

by IP address, but also gives them a little bit

of information about each one.

Pringle: The hackers then choose

which targets are worthy of their attention.

They mostly go after tech companies

and high-profile branches of the government.

Narrator: The SolarWinds attackers manage to breach

about a dozen vital US government agencies.

Disturbingly, they even break into the Cybersecurity

and Infrastructure Security Agency, or CISA,

the office at the Department of Homeland Security

whose primary function is to defend government networks

from cyberattacks.

Alexander: For them to breach the very agency whose job it is

to defend against these kinds of attacks

is a major embarrassment for US authorities.

How does that even happen?

Narrator: According to the Department of Homeland Security,

their system only detects known threats and the SolarWinds

attack is unlike anything they've ever seen before.

On top of that issue, their processes don't involve

scanning software updates.

So even if the system could have identified the malicious code,

they never would have detected it,

because it was buried in the update.

Pringle: That seems like a pretty big oversight on the part

of the government, who you would think would have

all sorts of safeguards in place.

Narrator: As investigators dig deeper,

they discover something truly shocking.

The attackers had unfettered access to some of these systems

for an astonishing nine months before the hack was detected.

Badminton: Interestingly, the hackers didn't seem to disrupt

any systems or destroy any files.

They just kind of silently roamed around.

Which points to one thing - cyber espionage.

Narrator: Authorities now have the motive for the attack,

but are still unable to find any evidence

as to who perpetrated it.

But they can't help but come back to their prime suspect:

Russia.

More specifically, one of the most tenacious and cunning

hacking groups on the planet:

the Russian Intelligence-backed APT29,

also known as Cozy Bear.

Pringle: Cozy Bear has been responsible for some of the most

infamous hacks of US and NATO member countries

over the past several years.

They're the cream of the crop.

Narrator: In 2016, WikiLeaks released 20,000 emails

from the Democratic National Committee

that they acquired after Cozy Bear and another hacking team

believed to be tied to a separate branch

of the Russian intelligence service,

accessed the DNC's internal network.

Cozy Bear camped out in the system undetected

for over a year, actions suspiciously similar

to the SolarWinds hack.

And the resemblance doesn't stop there.

Badminton: Both of these attacks have common thread,

they use cutting edge digital tools,

and this suggests state funding.

They went after strategic information,

rather than financial gain;

and they chose targets of interest

to Russia's intelligence community.

Narrator: For its part, Russia denied any involvement

in the SolarWinds hack.

But the US was unconvinced

and imposed sanctions on them as punishment for the attack.

Pringle: The U.S. ultimately announced

that 10 Russian diplomats would be expelled from the country

and 32 entities and individuals would be blacklisted.

The sanctions also targeted six Russian tech firms

linked to intelligence services.

Narrator: The full extent of the damage caused

by the SolarWinds hack remains something of a mystery.

For the government agencies that were breached,

it's virtually impossible for them to know the sum total

of the information the Russians had access to.

Badminton: They do know for sure that the hackers broke into

email accounts affiliated with the

Head of the Department of Homeland Security

and also several others who work

in the department's cybersecurity division.

Narrator: For private companies, the impact is also murky.

Microsoft reported no evidence of stolen or leaked

customer data from the attack.

Alexander: It seems pretty clear that the US government

was the primary target of the attack.

The tech companies were probably just collateral damage.

Narrator: Up against this kind of formidable enemy,

American authorities face difficult questions.

Was the attack just the tip of the spear

in the escalating cyberwar between familiar

cold war adversaries?

Is another, more destructive hack waiting around the corner?

Most experts agree - it's not a matter of

if it happens again, but when?

♪ [show theme music]

♪♪

Narrator: 25-year-old New York native Sheikh Ahmed

is on the hunt for a job as a bank teller.

He has applied for many positions across the city,

and today he gets the news that he has been selected

for not just one, but eight interviews.

However, these would be no ordinary meetings with

Human Resources representatives or branch managers.

These are HireVue assessments,

a state of the art recruiting tool

that uses artificial intelligence

to assess a candidate's worthiness

with no prospective employer present.

Badminton: The Hirevue system uses a person's phone

or computer camera to scrutinize the smallest details

of their answers to a standard set of questions.

Alexander: It analyzes their facial expressions;

how much eye contact they make,

what words they use and even their tone of voice.

HireVue claims that by using these metrics,

they can determine how enthusiastic a person is

about getting the job.

Narrator: This information is then used to automatically

produce an employability score,

which is ranked against other candidates.

The HireVue algorithm is part of a burgeoning field

of artificial intelligence called ERT,

Emotion Recognition Technology.

Aitken: ERT basically tries to identify how someone is feeling

based on their facial expressions

and other physical clues.

Narrator: These systems rely on two factors - computer vision,

to accurately recognize facial movements,

and machine learning to analyze and decipher them.

The algorithms reference huge image databases of human faces

that are classified by emotion,

and then try to match them to the subject.

Badminton: Six basic feelings are used:

fear, anger, joy, sadness,

disgust, and surprise.

Narrator: Pioneering American Psychologist Doctor Paul Ekman

was the first to categorize these as the fundamental

human emotions back in the 1960s.

Ekman is considered to be the founding father of ERT,

and his early research still echoes in today's

sophisticated artificial intelligence systems.

Alexander: He believed that these were the

universal feelings that all humans shared,

regardless of gender, culture,

location or situation.

Narrator: In 1978, he published

the the Facial Action Coding System or FACS.

Alexander: The FACS system categorized around 40 unique

muscle movements of the face and called the elements of each

expression an action unit.

Narrator: For the most part, FACS was a resounding success,

but there were issues.

The central problem was that it was very time consuming to use,

it took up to 100 hours to teach users the procedures,

and an hour to evaluate only one minute of film footage.

But a promising new technology that might help overcome

these obstacles was on the horizon,

computer vision.

Aitken: In the early '90s, researchers realized

that in order to take advantage of advancing technology,

they needed a database of standardized images

to work with.

Badminton: At this point, the US government stepped in

and financed a program to compile facial pictures.

They saw the potential for ERT as a security application.

Narrator: By the end of the 1990s,

machine-learning scientists began to collect

and classify these archives,

resulting in robust image datasets that provide

the foundation for much of today's AI based research.

And emotion recognition technology

is quickly becoming big business.

Alexander: One early provider of ERT services

was a startup called Affectiva.

Their technology was sold to businesses

as a market research product,

analyzing real-time emotional reactions to ads

and new products in focus groups.

Narrator: ERT has since expanded into many other areas

of business, particularly recruitment,

where companies like HireVue claim

that they can streamline the hiring process

by using their systems to weed out unworthy applicants

quickly and accurately.

Badminton: A process that used to take weeks

now only takes a few days.

It's way cheaper and faster than if humans were involved.

Narrator: In fact, ERT is now so prevalent in human resources,

that there are online guides with tips for candidates

on how to best present themselves to the camera.

For job seekers like Sheikh Ahmed, the process can be

an intimidating and distressing experience.

Aitken: They're essentially trying to impress a machine.

It really is kind of strange.

Narrator: Ahmed has spent countless hours studying guides

on how to speak and comfort himself,

but on the day of the interviews

he frets over something seemingly trivial,

how to position the camera.

Alexander: A high angle might make him seem weak and small,

whereas a low angle might make him appear too dominant.

Narrator: And there are other factors

fuelling Ahmed's anxiety,

namely that random sounds might harm his score.

He turns off the air conditioning system,

and tucks himself into the corner of his father's

soundproof music studio, far away from the normally

pleasant chirping of the family's pet bird.

Badminton: Because the software analyses

the sound of people's voices, any outside interference

could have an impact on his evaluation.

Narrator: Ahmed settles into a gruelling day

and confronts the unsettling reality of facing an algorithm

that judges every involuntary gesture that he makes

and every word that he utters.

To critics of emotion recognition technology,

and there are many, this is problematic.

Most wonder if artificial intelligence can really

interpret something as complex and nuanced as human behaviour.

Aitken: Some argue that it's impossible to know definitively

what a person is feeling simply by reading

their facial expressions.

People sometimes smile even if they're not happy

or scowl when they aren't angry.

Narrator: And there are other problems.

Critics of ERT claim that the image categorizing process

used to develop algorithms is overly simplistic.

Something as complicated as human emotion

can't be distilled down to six basic feelings.

Badminton: Emotions are complex and often interrelated.

There are many grey areas.

There are subtleties that no AI is capable of detecting.

Well, not quite yet!

Narrator: Some are also quick to point out

that people express emotions in many different ways,

not just using facial expressions.

Factors like body language are also indicators

of how a person is feeling.

Alexander: Physical cues such as crossed arms

or a slumped posture can sometimes convey

someone's state of mind better than the look on their face.

Narrator: But Ekman and his supporters counter

that the research is sound and stand by the assertion

that if a universal emotion is triggered in a person,

then an involuntary facial movement

naturally appears on their face.

Aitken: So the argument goes that even if that person

tried to hide their feelings,

the basic, reflex emotion would surface,

and if someone knew what to look for, they could identify it.

Narrator: Still, there are many skeptics who question

the scientific validity of ERT

and have problems with some of the methodology.

Alexander: One of the issues people have is that the

image datasets may be made up of posed faces.

If someone is asked to make a sad face,

it may look different from how their face

actually looks when they're sad.

Aitken: It's a valid argument, so the most recent systems

have started to draw on images that are candid,

footage of people doing mundane things like

driving their cars or watching TV.

Narrator: There has also been criticism of the forced-choice

answer method of labelling pictures in datasets.

Because there are limited options when asked

to ascribe an emotion to a picture,

there's no room for interpretation.

Badminton: Someone might look at an image and think the person

is feeling guilt or shame,

but those feelings may not be on the list of possible choices.

Narrator: And there are cultural concerns.

People from different regions of the world

convey emotions in different ways.

Alexander: Many people use smiles to show happiness

but for example, in Japan some smiles

are simple expressions of politeness, rather than joy.

So it can be fairly nuanced.

Aitken: But even if one culture has a slightly different idea

of what a happy face looks like,

most people recognize joy when they see it,

regardless of where they're from.

Narrator: In order to mitigate the effect of cultural nuances,

ERT companies are compiling more diverse datasets.

Affectiva, one of the leaders in the field,

boasts a collection of more than 10 million images

of people's facial expressions from 87 countries.

And they are always adjusting their algorithms

to make them more accurate.

Badminton: What these companies are now doing is including

an element of analysis to their systems.

So that rather than just identifying an emotion,

the AI is able to apply a cultural context

when classifying it.

Narrator: Context is another issue that critics of ERT

take umbrage with.

In 1972, Paul Ekman conducted an experiment

to study the differences between how Japanese

and American audiences reacted to a horror film.

And found that Japanese people showed less negative expressions

when there was an authority figure in the room.

Alexander: Different cultures have their own set of rules

about who can show certain emotions to whom.

In this case, the Japanese audience probably behaved

differently because they knew there was someone there

who may have been judging them.

Narrator: And for people like Sheikh Ahmed,

being judged by an Artificial Intelligence system

while merely trying to find a job would certainly

have an effect on one's behaviour.

Aitken: Ahmed altered his responses slightly

over the course of the eight interviews that day.

I guess he thought that if he gave the algorithm a variety

of answers, it might increase his chances of a positive score.

Narrator: By the end of the ordeal,

an exhausted Ahmed is drenched in sweat, his mouth is dry

and he can't shake the feeling that he hadn't made enough

eye contact with the camera or said the right things.

Alexander: Not enough eye contact?

You're shy and have no confidence.

Too much eye contact,

and you're aggressive and too intense.

Badminton: As difficult as the process may be,

the reality is that ERT in recruitment is only

going to become more common as the technology advances.

Narrator: And ERT is gradually creeping its way

into other fields as well.

The latest sector to feel its touch is education.

True Light College, a secondary school for girls in Hong Kong,

used ERT software to evaluate students' faces

as they learned remotely during the pandemic.

Alexander: The developers say that the system helps teachers

make learning more engaging and personal,

by reacting to a student's expressions in real time.

It even sends them alerts if they seem distracted or bored.

Narrator: The company behind the software claims

that it is able to correctly decipher a child's

emotional state about 85% of the time

and demand for the program has increased dramatically,

with the number of schools using it in Hong Kong

more than doubling from 34 to 83.

Aitken: The whole thing seems really invasive to me.

These are kids, after all.

Do we really need to be monitoring

their faces as they learn?

Narrator: Elsewhere in China, ERT is being used

for even more intrusive purposes.

Cameras with emotion recognition systems

have been installed in Xinjiang,

the region where an estimated 1 million mostly Uyghur Muslims

are being detained in prison camps.

Chinese authorities believe that their algorithms are able to

identify potential criminals by determining their mental state.

Badminton: It's kind of the next step in the evolution of ERT.

Some believe that not only can the AI detect how a person is

feeling, but it's even able to predict their future actions

and give an overall impression of their personality.

Aitken: But there's no real evidence that these systems

are even remotely accurate.

They're based on very vague so-called science.

Narrator: In 2018, a controversial study

out of Stanford University in California

even went so far as to declare that

facial analysis is capable of identifying

a person's sexuality.

Using a dataset of over 35,000 images

taken from dating websites,

a machine-learning system was able to differentiate

between pictures of gay and straight people

with surprising accuracy.

Alexander: The program was able to correctly categorize

81% of cases involving images of men

and 74% of photographs of women.

When humans did the same test, those numbers dropped

by 20% across both genders.

The researchers were actually quite shocked at how easy it was

for the algorithm to make the distinction.

Narrator: The authors of the study concluded

that there is mounting scientific proof

that there may be connections between faces and psychology

that are impossible to detect with the human eye,

but are identifiable to machine learning systems.

Badminton: Several prominent LGBT organizations

were not happy and demanded that Stanford distance itself

from the research, calling it dangerous and flawed.

Aitken: People were angry, because it is without question

an international human rights issue.

This kind of technology could be used to expose people as gay

whether accurately or inaccurately.

And in countries like Saudi Arabia and Iran,

where homosexuality is punished by execution,

that is very dangerous!

Narrator: While the controversies around

emotion recognition technology swirl,

the industry shows no sign of slowing down.

Some estimates project that it will reach

$37 billion US dollars by 2026,

up from $19.5 billion in 2020.

And it's not just plucky startups

looking to get a piece of the pie.

Tech industry titans like Apple, Microsoft and Amazon are all

investing heavily in developing their own ERT products.

Badminton: Obviously these companies see something of value

in the technology, but I think there will always be

lingering questions about its scientific integrity.

Narrator: Meanwhile, back in New York,

Sheikh Ahmed waits nervously

to finally find out if he got a job.

♪ [show theme music]

♪♪

♪

Narrator: In a Baron County, Wisconsin courtroom,

48-year-old Paul Zilly

is about to receive his sentence.

The stakes are pretty high,

he'll either be sent to prison or be given probation.

A significant factor guiding the judge's decision

will be determining whether or not he is likely to commit

another crime in the future.

Morgan: He had been arrested a few months earlier

for stealing a lawn mower and some other tools.

And he plead guilty to all of the charges.

Narrator: Before his appearance in court,

the county prosecutor offered him a plea deal:

One year in jail and follow-up supervision

to make sure that he doesn't reoffend.

Aitken: His court appointed attorney agrees to the terms,

saying that a long jail term isn't in his client's

best interest because he's not a career criminal...

In other words, the attorney doesn't think

it's likely he will reoffend.

Narrator: Unfortunately, it doesn't turn out

the way he expects,

Wisconsin judges are now looking to a new tool

to help determine if criminals will reoffend:

An artificial intelligence risk-assessment program

that is designed to predict future behavior

of convicted criminals.

Morgan: Wisconsin is one of the first US states

to integrate it into their criminal justice system.

Narrator: To Zilly's surprise,

it has rated him, "High Risk"

for committing violent crime in the future

and 'Medium Risk' overall as a potential reoffender.

Based on the algorithm's prediction,

the judge overturns the prosecution's plea deal

and sentences Zilly to two years in county jail.

[gavel strikes]

Aitken: He is completely shocked.

The judge doubled his prison time.

He thought he was only going to serve a year.

Narrator: Zilly is adamant he won't reoffend,

but some people working within the justice system are confident

that AI can accurately and fairly predict

if someone will commit a crime in the future.

Badminton: Over the past few years

several tech companies have used AI

in the development of 'Risk Assessment' software

that can be licensed by various judicial systems.

Narrator: The software's big selling point

is that it's able to mimic the problem-solving

and decision-making capabilities of the human mind.

Using machine-learning algorithms,

it analyzes existing data to detect patterns

and predict the likelihood that crimes will occur in the future.

Morgan: It's kind of like how a bookie determines the odds

for a sporting event or how pollsters figure out

who might win an election.

Narrator: AI risk assessment programs are now being used

by at least 16 different European countries

and almost every US state.

Aitken: Legal agencies are seriously understaffed,

Courts are deluged with criminal cases,

and so there is a backlog of cases waiting to be heard.

So often while waiting for trial,

people have to wait in prison,

which only contributes to overcrowding.

Narrator: The AI programs are designed to help

alleviate these problems.

Punitive decisions are difficult to make,

and made more so when judges are flooded

with many complex cases that require

a lot of knowledge and context.

The hope is that AI will help judges,

making the process more accurate and more efficient.

Badminton: It's not easy. The reality is that they may

end up making a terrible mistake

by granting a dangerous criminal parole or on the flip-side,

sentencing a person to prison when probation would be better.

So essentially the software is being used

to minimise these kinds of errors.

Narrator: One of America's leading risk assessment firms

claims that there are studies proving that the technology

is more accurate than human judges in predicting

a criminal's likelihood of reoffending.

In 2020, researchers at Stanford University

and UC Berkeley in California

discovered that when assessing things as complex

as a criminal justice system,

AI is up to 30% more accurate

with its decisions than the judges they surveyed.

Morgan: Critics challenge these claims, they say

that there is not enough evidence to prove that

these technology can actually improve decision-making.

Narrator: In one example, a 54 year old Florida man

with an extensive criminal record

involving aggravated assault,

multiple thefts and felony drug trafficking,

was arrested for shoplifting and surprisingly,

the algorithm rated him 'low risk' for reoffending.

Aitken: Judging by his criminal history,

you would probably think the opposite.

But this could indicate a problem

in how the software assesses risk.

Narrator: It may also indicate Paul Zilly is not actually

at 'High Risk' of reoffending and was unfairly sentenced.

Badminton: The algorithms look at police records

and court documents to see if the individual

has any prior arrests or convictions.

Those reports also present other relevant information

to the algorithm, like for example,

if the individual has a history of substance abuse.

Narrator: It turns out, Paul Zilly has a history

of drug abuse. Before he was arrested,

he was struggling with an addiction

to crystal methamphetamine.

He had told police he intended to sell the items he stole

to fuel his drug habit.

Aitken: This definitely may have contributed to him

being rated high risk by the algorithm.

Drug addiction is often related to crime,

as the desperate need for a substance

leads to desperate measures.

Narrator: The algorithm also analyses the responses to the

questionnaire Zilly filled out while he was incarcerated.

It consists of 137 questions that help determine

if a person is at risk of reoffending.

Badminton: Some of the questions are serious ethical quandaries,

for example, "Does a hungry person

have the right to access food?"

Whereas others are based on one's subjective opinion,

like "If people make me angry, I can be dangerous."

It seems some of these questions may not have a clear answer,

in which case, why are they using them?

Narrator: According to Zilly's risk assessment,

he scored poorly on the questionnaire;

combined with his history of drug abuse,

he was labelled 'High Risk'.

Morgan: This may help explain why his sentence

was upped from one year to two.

In response, Zilly's court-appointed attorney

filed an appeal, trying to reduce the sentence.

Narrator: There is concern that the questionnaire may also be

biased in what it specifically asks of individuals,

like if they are employed, where they live

and what the crime levels are like in their neighborhood.

Critics say this could result in the algorithms

making assessments that are discriminatory.

Badminton: Since poorer neighbourhoods often have

higher crime rates than more wealthy ones,

the algorithm may assume it's residents

are at a greater risk of committing a crime

than if they were living in a rich area.

Narrator: Civil rights activists believe the technology

could unfairly flag people of colour,

who statistically, live in poorer neighbourhoods

with higher crime rates.

And there is compelling evidence that it is already happening.

Aitken: Recently, a study of the AI program that had provided

risk scores to offenders in Broward County, Florida,

found the algorithm incorrectly flagged black defendants

as future criminals at almost

double the rate as white defendants.

And white defendants were incorrectly assessed as low risk

to offend more often than their black counterparts.

Narrator: Critics cite the case of an 18-year-old

African-American woman who was arrested for burglary

in Ft. Lauderdale, Florida.

Despite being a first time offender,

the algorithm rated her 'high risk'.

Morgan: Compare this to the previous summer,

when an older white man from the same area was arrested

and rated low risk, despite having been

previously convicted of armed robbery.

Narrator: This leads civil rights activists to conclude

that risk assessment programs

may be perpetuating existent biases,

further compounding prejudice and racism

in the justice system.

But the companies that license their software to several US

state justice systems claim that a person's race

isn't a factor in the algorithms' risk assessment.

AI advocates believe the opposite, that the technology

actually makes the criminal justice system

fairer for people of colour.

Aitken: They say that it cuts the human, or biased factor out,

meaning that it should give a more objective,

less-biased evaluation of each person.

Badminton: But as we just saw in Florida,

this isn't always the case.

The algorithm perpetuates existing biases.

Narrator: At Paul Zilly's appeal hearing,

his lawyer questions Dr. Tim Brennan,

one of the creators of the AI software

that assessed his client.

Morgan: Brennan testifies that his software

wasn't designed to be used in sentencing.

In fact he didn't want it involved

in the criminial justice system at all.

Its purpose was to help reduce crime,

not to further punish people like Paul Zilly.

Narrator: In light of Brennan's testimony,

the judge reduces Zilly's sentence to 18 months, admitting

that he may have put too much faith in the algorithm.

Badminton: Here is a case where the judicial system

relied far too heavily on this technology,

leading the judge to make an unfair sentencing decision.

And unfortunately, it's probably safe to assume

that this isn't the only case where this has happened.

Narrator: To attempt to remedy this,

several civil rights lawyers, UN officials and labor unions

are now lobbying for more government regulation

of AI's use within the legal system.

Aitken: It's a civil rights issue.

Because the technology can perpetuate biases,

the use of algorithmic tools in courtrooms can lead to

violations of a person's right to a fair sentencing.

Narrator: Despite inherent problems,

several legal analysts believe its potential for good

far outweighs the harm it may cause.

Morgan: The State of Virginia claims

that they've managed to cut down on their prison populations

by 26% using these algorithms.

They say they'd been able to do so by releasing people early

who are 'low risk' of reoffending.

Aitken: Unburdening the justice system,

while providing people with a chance to rebuild their lives

outside of prison, is obviously a benefit to everyone involved.

The question, as always, is

if we can live with the negative consequences

of employing this technology.

Is it worth it if even one person is sentenced unjustly?

Maybe not.

Narrator: The case of Paul Zilly clearly illustrates

the inherent perils of allowing algorithms to make decisions

regarding something as important as a person's freedom.

And while it's still unknown what future impact

AI will have on our legal systems,

as more courtrooms adopt this technology,

and barring a proper framework regulating its use,

it's likely that injustices will continue

and the controversy surrounding it will further intensify.

♪

♪ [show theme music]

♪♪

♪

Narrator: At the Vellore Institute of Technology

in southern India,

19-year-old Priyanjali Gupta

is a 2nd year engineering student

specialising in data science.

In February 2021, she decides to take the weekend off

from her studies and visit her mother in New Delhi.

While there, Priyanjali confides something to her.

Pringle: She's been thinking about trying to develop

some kind of new technology that can help people.

But she doesn't know what specifically.

Narrator: One day, Priyanjali has an epiphany of sorts

and realizes that virtual assistants

which rely on voice commands such as Alexa and Siri

are not accessible to people who are deaf...

So she sets out to create an application

that will be inclusive to people with hearing disabilities,

using artificial intelligence.

Aitken: She wants to invent an AI program

that can translate visual 'sign language'

into English text and do so in real-time.

Narrator: There are many different forms of sign language

but Priyanjali's AI application translates

the most commonly used, American Sign Language, or ASL.

Which is used by around 500,000 people

in the US and Canada.

Priyanjali is hopeful she can get the software to work,

but it will be no easy feat.

It's a complicated and technical process.

Aitken: The process is called, 'Deep Learning'.

It's where the AI is trained to perform tasks

by analyzing large amounts of data,

it's similar to how human beings learn.

And it does so on a network called a "neural network."

The more data the algorithms are able to analyse,

the more they will "learn" and the more accurate

they will become in their analysis.

Narrator: In Priyanjali's case, her machine will need to analyze

different sign language gestures

to be able to learn what they mean.

It's in the developmental stages but it could be promising.

Around 70 million people around the world

use sign language to communicate,

so it's crucial that technologies like this exist.

Morgan: And it isn't just limited to people

with hearing disabilities.

AI can help people who are blind or have

other physical or cognitive disabilities.

Pringle: The technology has the potential to provide

more independence in their day-to-day lives.

Narrator: There are already many AI powered tools

available to the disabled.

People with visual impairments can access talking keyboards,

and use various virtual assistants like 'Siri'

and 'Alexa' to perform a web search or write an email.

Pringle: There are also AI powered applications

that can read out words on a printed page,

a computer screen or smartphone.

It can even describe what's on screen,

such as application icons, photo images and videos.

Narrator: Artificial intelligence is also having

an impact on helping those for whom

communication can be challenging.

Morgan: People with certain brain injuries

or with conditions like Parkinson's can have a hard time

speaking in ways that are easy for others to understand.

But AI algorithms can take what they are saying

and transform them into audio or text files

that are easier to understand.

Narrator: The technology can also assist people

who may be unable to speak at all.

In Nebraska, Kaden Bowen is a teenager with cerebral palsy,

it's a condition which prevents him

from being able to walk or talk.

To help him communicate, he uses a rudimentary speaking device

with buttons containing preselected words or phrases.

But recently he and his father began using Amazon Echo,

a virtual assistant that uses AI in its voice-control system.

Aitken: Using his speaking device,

Kaden can have the Echo perform tasks he wants it to do,

like to call his family members on their phones,

and ask them, for example, to take him for a car ride.

It may seem small, but this provides him with

an ability to communicate that he didn't have before.

Narrator: One of the most significant ways

that technology is improving disabled people's lives

is in transportation.

Mobility is often one of their most challenging issues.

But AI powered navigation tools like 'Google Maps'

can help them attain more autonomy.

Morgan: Apps like these utilize GPS technology to make it

really easy to visualise the route you need to take,

all while providing information about accessibility,

like where ramps or elevators are.

Narrator: The recent advancement of self-driving cars

is also a potentially significant development.

Aitken: People with disabilities that prevent them

from being able to drive, might be able to

use self-driving cars to get around on their own,

providing them with a degree of independence

they may not have had before.

Narrator: Despite AI's many positive benefits,

there are experts who are raising questions about

its potential limitations, particularly

around the technology's 'financial accessibility'.

According to recent data, roughly 26 percent

of US citizens with disabilities

are currently living in poverty,

nearly two and a half times higher than people

who aren't disabled.

And it's more or less the same for people

living in EU countries.

Morgan: They may be unable to find work that pays

a decent wage, if they are able to work at all.

They might not have financial support from friends or family

and any government disability funding

might not provide them enough to live on.

Pringle: So owing to their economic insecurity,

they may not have the money to spend

on cutting edge technology or software,

leaving them unable to benefit from it.

Narrator: This situation could be even more challenging

in developing nations where poverty rates are higher

and the median income is lower.

Aitken: Problems with access to technology

are sure to be difficult to address,

because it is systemic in nature.

Meaning that there are many contributing factors

and reasons as to why it exists,

making it all the more difficult to solve.

Morgan: But some are trying.

One company is setting up an AI interface to help

people with disabilities find employment opportunities.

It can browse job search results and even set up interviews,

then can provide the individual with an interactive

voice response, chatbots, and voice assistants.

Narrator: And recently, Microsoft invested $25 million

on a global AI Accessibility initiative,

funding projects that develop software

and technologies for disabled people,

aiming to improve their independence

and quality of life.

Pringle: It's worth remembering that as the technology

becomes more available, its costs will also come down.

And so that will increase its accessibility

to people with disabilities.

Narrator: Perhaps there is some hope in the fact that there are

so many young AI developers like Priyanjali Gupta.

On her webcam, she records herself doing

several basic sign language gestures.

Aitken: The AI software will then interpret the motions

and translate it into readable English text.

Narrator: The project is still in its initial phases

and faces some technical limitations.

But maybe with time, it can become a full fledged

on screen translator of sign language.

Pringle: Thankfully, Gupta is not the only one

developing AI programs to help people

with hearing disabilities or impairments.

Narrator: Several tech companies are developing smartphone

applications that can use its camera to lip-read.

There are also AI applications that utilize

Automated Speech Recognition, or ASR,

which can transcribe the conversation

of a group of people in real-time.

Morgan: Something like that could help people with

hearing disabilities to be included in a conversation

without even needing to read lips.

Icing on the cake is that these algorithms can add things

like punctuation and names of the person who's speaking.

Narrator: Internet accessibility is also

a significant issue for disabled people.

While some websites are now optimizing their platforms

to allow visually impaired individuals

to adjust the font size and colour,

to be more easily seen and read,

a recent study showed that 98% of the world's

top one million websites don't offer full accessibility.

And there are serious concerns

about the ones that are accessible,

specifically regarding online privacy.

Morgan: Many of these tools are cloud-based,

so it's possible that information about

a person's disability could be obtained by a third party.

In addition to being a huge violation of privacy,

information like this falling into the wrong hands

leaves the door open to things like

discrimination or social exclusion,

or even just online bullying.

Narrator: But many disabled people are embracing technology,

believing its ability to help them far outweighs

any potential problems it may cause,

and for people like Priyanjali Gupta,

this is all the encouragement they need.

Morgan: Gupta is currently able to get her AI technology

to adapt six different sign language gestures into English.

"Yes", "No", "Please", "Thank You",

"Hello" and "I Love You".

Narrator: Now in her 3rd year, the 21-year-old

university student is researching a new neural network

that will improve the video analysis done by the AI.

And she's also trying to secure additional funding

in order to make improvements.

Aitken: She sees her invention as a small, but very important

step in helping people struggling with disabilities.

Narrator: As technology continues to be integrated

into the day-to-day lives of people with disabilities,

there is real hope that it has the potential to help them

live more independent and fulfilling lives.

And as long as the Priyanjali Guptas of the world

are out there using their expertise to develop

new and innovative applications,

the future looks more accessible than ever.