All language subtitles for cyberwar.s01e15.720p.hdtv.x264-w4f_Track02

BEN: Scientists design machines
that think for themselves...

I guess I have an aversion
to the word "drone".

It implies something mindless.

...with the potential
to become weaponized...

(Popping)

The military would be crazy
not to take these technologies

and utilize them in
ways that would help us

perform better on
the battlefield.

...or even hacked.

There will always
be vulnerabilities.

You can't design them
out of your system

just with a sheer force
of will and intelligence.

But how will it change
the way we fight wars?

It is a fantasy to think that
you can delegate the decision

over life and death to a robot.

♪

50 years ago, it was pure
science fiction to imagine

a U.S. soldier sitting
in a bunker in Nevada

could remotely pilot a plane
flying thousands of miles away,

and then drop a hellfire
missile on an enemy target.

But this is reality in 2016.

Today, unmanned aerial
vehicles, or UAVs,

are a vital part of warfare.

They're in the stockpiles of
more than two dozen nations,

and target and kill
thousands around the world.

The next generation
of drone technology

is even more sophisticated.

I found it here, in
this idyllic little field

just outside of
Budapest, Hungary.

When you look at a
flock of starlings in the sky,

their motion is based
on very simple rules.

The whole pattern, their
motion is some kind of

meta level intelligence or
collective intelligence,

and the same kind of thing
applies to drones as well.

When I see the
whole flock flying,

it's always magical for
me after so many years.

This is Gabor Vasarhelyi.

He's part of a team developing
technology they hope will be

used in agriculture
or search and rescue.

They're working
on drone swarms,

which is to say a flock of
autonomous drones in the sky,

moving and making decisions as
a posse of like-minded robots.

(Beeping noise)

Unlike the typical drones
used today, there's no

human operator directing
their movement individually.

(Buzzing)

Oh, now you're hearing them
kinda creep up, eh?

(Buzzing)

These machines are given
a set of instructions...

BEN: Holy ----.

...then they figure out
how to execute them as a team,

solving problems in real time.

(Shrugging)

That beacon in my hands
makes the drones track

whoever's holding it,
wherever they go.

Notice how they coordinate
and keep their distance

from one another automatically.

Ah!

How are they communicating
with one another?

The drones have a local
communication network.

Every single drone just sends
out some info about itself,

and all the drones that are
close enough to receive this,

they receive and
integrate into the decisions.

Why do you think the
military's interested

in this kind of technology?

If you have one big aircraft
and someone shoots it,

then it's a lot of damage and
then your whole man is gone.

When you have 100 drones instead
and someone shoots at it,

then one drone is taken away
and the rest can do the same.

That's how mosquitos
work, for example.

Like there are many small
mosquitos, everybody is just

sipping a little bit of blood,
but the whole species goes on.

Air Forces of the future
may be stoked on the idea

of drone swarms, but
what about the navies?

Here at a NATO research facility
in Italy, scientists are

essentially developing
autonomous unmanned subs.

But signals travel
more slowly underwater,

and that makes it harder to
create a submarine swarm.

One of the things
we're trying to do is

create the internet
for underwater robots.

So we have the internet on land,

and now we're getting
the Internet of Things.

Well, we're doing Internet of
Things for underwater things.

So, how is the
internet underwater?

It sucks!

(Laughing)

John Potter is the scientist in
charge of strategic development.

He hopes that the drones
they're developing here

will be used to map the oceans.

So when it comes
to autonomous...

or when it comes to, let's
say, drones in the water,

what is coming over the horizon?

I guess I have an aversion
to the word "drone".

It implies something mindless.

From the outset,
there was never an option

to go the "drone" route,
of having a dumb device.

Because once you put it
in the sea, and it dives

and it's gone more than
100 metres away from you,

you don't know what it's up to.

And if you ever want
to see it come back...

(Laughing)

...you want to have something
a little smarter than that.

So what would you
call them, unmanned...?

So these vehicles are a
step forward in autonomy

with respect to
what's currently...

typically in operation
at the moment.

They're able to adaptively
change what they are doing.

So you can put vehicles in
and have them work as a team.

And if one vehicle sees
something and it knows it has

a team member with maybe
a high resolution camera

or imaging system that's
just over here somewhere,

and it knows how far away
it is, it says, "Hey, Bill.

Come and take a look at this,
I think I found a so-and-so.

You wanna check that out,
see what you think of it?"

And I guess that's one of the
other things people are going to

wonder, is like... is something
like an underwater system

like that, could that be
weaponized at some point?

That's not something we
are really working on here.

How these autonomous abilities
eventually translate into

operational systems depends
on lots of other folk

and organizations within
NATO and elsewhere.

It is something
that's being considered.

But I would like to see all the
stakeholders - which basically

means all of society - become
informed, well informed,

and think carefully about this.

What is it that we actually
want from autonomous systems?

And that really hit me.

The people developing these
systems aren't the ones

who will decide how they're
used in military conflict.

But whether or not
you're comfortable

with autonomous drone
swarms, they're coming.

I'm in Portugal, heading out
to a NATO research ship

in the Atlantic to get
a first-hand view

of underwater drone testing.

I'm gonna have to climb
that puppy right there,

like Blackbeard or something.

(Grunting)

♪

This is more or less our
control centre where we keep

the picture of everything that
we have deployed at sea.

This here is the
position of the ship,

this is where we are now.

You'll see here these green
points is our field of

deployed assets for the
current tests we're doing.

This is basically
your ocean lab.

It's our ocean lab, exactly.

That's a good way to put it,
it's our ocean lab.

Joao Alves is the Coordinator
for Underwater Communication.

He and his crew spend weeks at
sea experimenting with things

like submarine communication
and anti-sub warfare tech.

"Time Bandit."

So these are the
anti-submarine warfare ones?

These are the ones that we
employ for our multistatic

anti-submarine warfare
missions, yes indeed.

Now I know these machines
right now are just...

they're doing nothing more than
reconnaissance or surveillance

or even just mapping, or
anti-submarine warfare

in terms of detecting
different possible enemy craft,

but do you think at any point

these things could
be weaponized?

Do you think
that's a possibility?

We, as a science-based
research centre,

are interested in
developing the autonomy,

the capabilities
of these machines.

Then, I mean, it's totally out
of our scope, the usage of...

We are very excited on
the examples you just gave,

on the improving
the reconnaissance,

improving the mapping
capabilities...

other than that, I mean, we...

You're not touching the rest?

No, not at all.

I mean, not even our interest.

Joao says he's not interested
in weaponizing these drones.

But like other people working
on the same autonomous machines,

he doesn't decide how
they're ultimately used.

That's up to the military.

BEN: Unlike the carpet
bombings of the past,

drone strikes are
supposed to be precise,

limiting civilian casualties.

That said, it's tough to
know exactly how many people

around the world were killed
by drones in the past decade.

Estimates range
from the thousands

to the tens of thousands, and
it's even harder to figure out

how many of those killed
or injured were civilians.

♪

But it's not just the
numbers that alarm some critics.

Have you seen a US
president go on TV to say,

"Tonight, I ordered
airstrikes in Libya"?

Or Pakistan or Yemen?

They don't do that anymore.

Because something about
drone technology and other

weapons technology has enabled
US presidents and politicians

to basically shift
what the norm is.

Naureen Shah is the director
of Amnesty International USA's

Security and
Human Rights Program.

She's a vocal opponent
of the US military's

targeted killing campaigns,

and worries about the autonomous
technologies in development.

So the idea that this could
reduce civilian casualties

to you is impossible?

Our concern is it actually would
increase civilian casualties,

increase the risk of
civilian casualties.

I of course agree that if we can
keep people out of harm's way,

that's vital.

But if you don't have
governments having to weigh

the costs to their own citizens,
when they decide to go to war,

then you're really making
it so that their stakes

are a lot lower.

And that could mean that
the US and other governments

are just engaging a lot more in
warfare than they did before,

just not calling it warfare,
and we already see that.

The US is using lethal force
right now in Libya, Syria,

Iraq, Afghanistan,
Yemen, Pakistan, Somalia.

There's something kind of
invisible about the use of

autonomous weapons, or robots.

Something that enables
policy makers to think, well,

we could go... we could use
lethal force in a surgical way,

in a limited way, and we
wouldn't even have to have

a big public debate about it.

We're supposedly
just at war all the time.

But at what point is this
technology just inevitable?

One of the things
that is so difficult

is that technology is
fascinating for all of us.

And it's so fascinating
that it creates

almost a kind of glee.

It's a fixation that
our generation has

on the possibilities
provided by technology.

It can make us better
than our own selves,

that somehow a robot is more
selfless than a human being is,

doesn't have the prejudices
of a human being, and that

can solve the fundamental
gruesomeness of war.

It is a fantasy to think that
you can delegate the decision

over life and death to a robot
and somehow that inherently

makes it more humane and
more precise and more lawful.

Because there's nothing
that would give us any reason

to believe that a robot
has human empathy,

that it has the ability to
make a judgment about who is

a civilian and who is not a
civilian, and whether or not in

those particular circumstances a
civilian really poses a threat.

But some scientists completely
disagree with Naureen,

and think that it is in fact
possible to program a robot

to act in a more humane,
precise and lawful way

than a human soldier.

And here on the Georgia Tech
campus, they tried to prove it.

♪

So if you did something bad,
and you felt guilty about it,

you would be less
likely to do it again.

We want the robot to
experience... to behave in...

A similar way, right.

The robots... None of these
robots feel emotions, okay?

They don't feel anything.

They're not sentient.

But people can perceive
them as feeling things.

Ron Arkin is a
roboticist and professor.

He's developing technology

designed to program
ethics into robots.

There are issues associated
with human war fighters

who occasionally are
careless, make mistakes,

and in some cases
commit atrocities.

So you really think that
you can program a robot to be

a better and more efficient
killer than a human being,

and without making the same kind
of mess ups that we might see,

say in friendly fire
or civilian casualties?

I'm not interested in making
them better and more efficient

killers; I'm interested in
making them better protectors

of non-combatants and better
protectors of civilians while

they are conducting missions
than human war fighters are.

So my goal is to make them
better adhere to international

humanitarian law as embodied
in the Geneva conventions

and the rules of engagement.

You cannot shoot
in a no-kill zone,

you cannot shoot people
that have surrendered,

you cannot carry out
some summary executions.

These systems should
have the right to

refuse an order as well, too.

- Really?
- Yeah.

This is not just a
decision of when to fire,

it's also a decision
of when not to fire.

So that if someone tells it
to attack something that is

against international
humanitarian law,

and its programming tells
it that this is against

international humanitarian
law, it should not engage

that particular target.

So will these weapon systems,

could they be like the
Tesla self-driving car?

You know, you won't get
into as many accidents.

Yes, and that's the argument
that self-driving cars use.

It's the same sort of thing.

They say that human beings
are the most dangerous things

on the road because
we get angry, we drink,

we are distracted.

It'd be better to have
the robots driving us there.

Look at 9/11.

There was no reason an
aircraft should've crashed

into those buildings.

It's an easy task for a
control system, now and then,

to be able to change its
altitude when it recognizes

it's in a collision course and
avoid that particular object.

We chose not to do that.

We choose to trust human beings
over and over and over again,

but that's not
always the best solution.

BEN: If there's one thing I've
learned over the last few years,

it's that it's possible
to hack virtually everything

running on code - from a nuclear
enrichment facility to an SUV.

But what about the
drone swarms of the future?

Here in Texas, there's a team
of researchers that demonstrated

how military drones
used today are hackable.

Todd Humphreys is the director
of the Radionavigation Lab.

So this is our arena.

BEN: Nets for the drones?

TODD: The nets' for the FAA.

So there's a
future, you think,

wherein you could see
a ton of these things?

Dinner table-sized drones
possibly, like in a swarm?

Oh yeah.

- Attacking a target?
- That's right.

And they have no regard
for their own life, right?

So these are suicide drones.

They pick their target,
they go directly at it,

and the kinds of close-in
weapons systems and...

large-scale weapon systems
that our US destroyers

and the Navy have
today or other ships,

they'll be no match against
16 of these at once.

Right, and it all
starts here in Austin.

Well, we don't intend
to do development

of war machines here.

But I will say that our
somewhat whimsical games

that we'll be playing,
they are going to engage

our operators and
our drones in scenarios

that are applicable
to all sorts of fields.

One of those so-called whimsical
games he and his students

showed me was a drone version
of Capture the Flag.

(Popping)

But that's not all
they've been up to.

Back in 2011, when the Iranians
claimed to have hacked

a US military drone, Todd
and his team proved that

it was indeed possible,
and demonstrated how

a simple method called
spoofing could have been used

to jack US military hardware.

Every drone has
a few vital links.

One of them is to
its ground controller,

and one of them is to overhead
satellites for navigation.

Spoofing attacks one
of those vital links.

It basically falsifies a GPS
signal, makes a forged signal,

sends it over to the drone
and convinces the drone that

it's in a different place or
at a different time, you know,

because GPS gives us
both time and position.

And you've actually proven
this is possible?

We've demonstrated it, yeah.

So we've done this,
we've done it with a drone,

we've done it with
a 210-foot super yacht.

So this is back in
2011 it was possible.

- Yeah.
- How about now?

Have drones kind of
caught up to this?

Have engineers realized
that, you know,

when you put this
autonomous thing in the air,

essentially it can be
overtaken by a hostile actor?

I'd like to be able
to say yes, but no.

The FAA has... has charged a
tiger team to looking into this,

and they came back after
two years of study and have

put together a set of
proposals that would make

commercial airliners
more resilient to spoofing,

more... have better
defenses against spoofing.

They've also looked at even
smaller unmanned aircraft.

But things move slowly in the
world of commercial airliners.

And as far as smaller unmanned
aircraft, I think those of us

who are just playing around
with the small toys and such

aren't really thinking so much
about security at this point.

So we know that hacking
drones are possible, and yet

people are starting to think
about creating drone swarms.

Mm-hmm.

What if your drone
swarm... you send one off

and you're thinking you're
gonna destroy your enemy,

and all of a sudden it
turns right back around

and it comes at you
because it's been hacked?

Right.

I mean, do you have to prepare
it to be able to destroy

your own drone swarm?

Absolutely.

I believe you have to have a
killswitch for your own swarm,

your own resources,
your assets, and that

that must be an
ironclad kill switch.

The only way to think about
security is as an arms race.

There will always
be vulnerabilities,

and you can't somehow design
them out of your system

just with sheer force
of will and intelligence.

♪

BEN: I'm at the
Pentagon in Washington

to meet with Robert Work.

He's basically the number two
at the Department of Defense,

which is why he gets the
Blackhawk helicopter treatment.

We're heading out to
the US Army Research Lab

at the Aberdeen Proving
Ground in Maryland.

Work is leading the Third
Offset Strategy, an initiative

aimed at building up the
military's tech capabilities

to counter big-time adversaries
like Russia or China.

The strategy calls for the US
military to focus on robotics,

miniaturization, 3D printing,
and autonomous systems.

So why is the US government
so interested in autonomous

weapons systems and autonomous
machinery for the military?

Autonomy and
artificial intelligence

is changing our lives every day.

And the military would be crazy
not to take these technologies

and utilize them in ways that
would help us perform better

on the battlefield.

I mean, would it be crazy
because other countries

are going to do it too?

We know that other great
powers like Russia and China

are investing a lot of
money in autonomy and AI,

and they think differently
about it than we do.

You know, we think
about autonomy and AI

of enabling the
human to be better.

Authoritarian regimes sometimes
think about taking the human

out of the equation and
allowing the machine

to make the decision, and we
think that's very dangerous.

That's not our
conception at all.

It's more like Iron Man,

where you would use the
machine as an exoskeleton

to make the human stronger,
allow the human to do more;

an autonomous intelligence
that's a part of the machine

to help the human make
better decisions.

Would taking humans
out of the loop

give your adversary
an advantage?

This is a question just
like cyber vulnerability

that keeps us up at night.

Would a network that is working
at machine speed all the time

be able to beat a network

in which machines and
humans work together?

Um, and in certain
instances like...

as I said, cyber warfare,
electronic warfare,

machines will
always beat humans.

I mean, that will always happen.

This is a competition, and we
think that the way this will go

for the next multiple decades is

AI and autonomy will
help the human.

And you will never try to
make it go all automatic.

But we have to watch,
and we have to be careful,

and make sure that
that doesn't happen.

But not everyone thinks about
the Third Offset Strategy

in such an optimistic way.

In fact, some people think
it's only creating a new

arms race for robotic war tools
that could escalate quickly.

Naureen Shah is
one of those people.

What would you say to the DOD
policy-makers who are actually

looking into researching
autonomous weapons systems?

I would say ban
these weapons systems,

because you yourselves know
what the consequences would be

if other governments had them.

Look at this issue not
from the perspective just of

the US government, but from
how to keep communities

all around the world safe from
unlawful use of lethal force.

If the US government is
concerned about what it sees as

bad actors having
these weapons systems,

then it shouldn't develop them.

Once you start to manufacture
these weapons systems and

authorize arms sales
to all these countries,

this technology is
going to proliferate,

and it will unfortunately
spiral out of control.

We know that you can do it.

We know that the
technology is tantalizing,

but you know that if
you start down this road,

you're going to a
very, very dangerous place,

and you shouldn't go there.

At this point, it
might be too late.

These war toys are on the way.

And the thing is, almost every
developer of autonomous machines

I met is separated from the
people who actually decide

how their creations
will enter war.

So while researchers toss around
ideas about humane battles

or friendly killer robots,
you get the feeling

conflict isn't going to
be any less horrifying

using autonomous bots.

Whether it's stick and stones,
guns, missiles, or drones,

people will always go to war,
and it's always brutal.

Even if a robot is
fighting for you.