Original subtitles

BEN: Scientists design machines that think for themselves...

I guess I have an aversion to the word "drone".

It implies something mindless.

...with the potential to become weaponized...

(Popping)

The military would be crazy not to take these technologies

and utilize them in ways that would help us

perform better on the battlefield.

...or even hacked.

There will always be vulnerabilities.

You can't design them out of your system

just with a sheer force of will and intelligence.

But how will it change the way we fight wars?

It is a fantasy to think that you can delegate the decision

over life and death to a robot.

♪

50 years ago, it was pure science fiction to imagine

a U.S. soldier sitting in a bunker in Nevada

could remotely pilot a plane flying thousands of miles away,

and then drop a hellfire missile on an enemy target.

But this is reality in 2016.

Today, unmanned aerial vehicles, or UAVs,

are a vital part of warfare.

They're in the stockpiles of more than two dozen nations,

and target and kill thousands around the world.

The next generation of drone technology

is even more sophisticated.

I found it here, in this idyllic little field

just outside of Budapest, Hungary.

When you look at a flock of starlings in the sky,

their motion is based on very simple rules.

The whole pattern, their motion is some kind of

meta level intelligence or collective intelligence,

and the same kind of thing applies to drones as well.

When I see the whole flock flying,

it's always magical for me after so many years.

This is Gabor Vasarhelyi.

He's part of a team developing technology they hope will be

used in agriculture or search and rescue.

They're working on drone swarms,

which is to say a flock of autonomous drones in the sky,

moving and making decisions as a posse of like-minded robots.

(Beeping noise)

Unlike the typical drones used today, there's no

human operator directing their movement individually.

(Buzzing)

Oh, now you're hearing them kinda creep up, eh?

(Buzzing)

These machines are given a set of instructions...

BEN: Holy ----.

...then they figure out how to execute them as a team,

solving problems in real time.

(Shrugging)

That beacon in my hands makes the drones track

whoever's holding it, wherever they go.

Notice how they coordinate and keep their distance

from one another automatically.

Ah!

How are they communicating with one another?

The drones have a local communication network.

Every single drone just sends out some info about itself,

and all the drones that are close enough to receive this,

they receive and integrate into the decisions.

Why do you think the military's interested

in this kind of technology?

If you have one big aircraft and someone shoots it,

then it's a lot of damage and then your whole man is gone.

When you have 100 drones instead and someone shoots at it,

then one drone is taken away and the rest can do the same.

That's how mosquitos work, for example.

Like there are many small mosquitos, everybody is just

sipping a little bit of blood, but the whole species goes on.

Air Forces of the future may be stoked on the idea

of drone swarms, but what about the navies?

Here at a NATO research facility in Italy, scientists are

essentially developing autonomous unmanned subs.

But signals travel more slowly underwater,

and that makes it harder to create a submarine swarm.

One of the things we're trying to do is

create the internet for underwater robots.

So we have the internet on land,

and now we're getting the Internet of Things.

Well, we're doing Internet of Things for underwater things.

So, how is the internet underwater?

It sucks!

(Laughing)

John Potter is the scientist in charge of strategic development.

He hopes that the drones they're developing here

will be used to map the oceans.

So when it comes to autonomous...

or when it comes to, let's say, drones in the water,

what is coming over the horizon?

I guess I have an aversion to the word "drone".

It implies something mindless.

From the outset, there was never an option

to go the "drone" route, of having a dumb device.

Because once you put it in the sea, and it dives

and it's gone more than 100 metres away from you,

you don't know what it's up to.

And if you ever want to see it come back...

(Laughing)

...you want to have something a little smarter than that.

So what would you call them, unmanned...?

So these vehicles are a step forward in autonomy

with respect to what's currently...

typically in operation at the moment.

They're able to adaptively change what they are doing.

So you can put vehicles in and have them work as a team.

And if one vehicle sees something and it knows it has

a team member with maybe a high resolution camera

or imaging system that's just over here somewhere,

and it knows how far away it is, it says, "Hey, Bill.

Come and take a look at this, I think I found a so-and-so.

You wanna check that out, see what you think of it?"

And I guess that's one of the other things people are going to

wonder, is like... is something like an underwater system

like that, could that be weaponized at some point?

That's not something we are really working on here.

How these autonomous abilities eventually translate into

operational systems depends on lots of other folk

and organizations within NATO and elsewhere.

It is something that's being considered.

But I would like to see all the stakeholders - which basically

means all of society - become informed, well informed,

and think carefully about this.

What is it that we actually want from autonomous systems?

And that really hit me.

The people developing these systems aren't the ones

who will decide how they're used in military conflict.

But whether or not you're comfortable

with autonomous drone swarms, they're coming.

I'm in Portugal, heading out to a NATO research ship

in the Atlantic to get a first-hand view

of underwater drone testing.

I'm gonna have to climb that puppy right there,

like Blackbeard or something.

(Grunting)

♪

This is more or less our control centre where we keep

the picture of everything that we have deployed at sea.

This here is the position of the ship,

this is where we are now.

You'll see here these green points is our field of

deployed assets for the current tests we're doing.

This is basically your ocean lab.

It's our ocean lab, exactly.

That's a good way to put it, it's our ocean lab.

Joao Alves is the Coordinator for Underwater Communication.

He and his crew spend weeks at sea experimenting with things

like submarine communication and anti-sub warfare tech.

"Time Bandit."

So these are the anti-submarine warfare ones?

These are the ones that we employ for our multistatic

anti-submarine warfare missions, yes indeed.

Now I know these machines right now are just...

they're doing nothing more than reconnaissance or surveillance

or even just mapping, or anti-submarine warfare

in terms of detecting different possible enemy craft,

but do you think at any point

these things could be weaponized?

Do you think that's a possibility?

We, as a science-based research centre,

are interested in developing the autonomy,

the capabilities of these machines.

Then, I mean, it's totally out of our scope, the usage of...

We are very excited on the examples you just gave,

on the improving the reconnaissance,

improving the mapping capabilities...

other than that, I mean, we...

You're not touching the rest?

No, not at all.

I mean, not even our interest.

Joao says he's not interested in weaponizing these drones.

But like other people working on the same autonomous machines,

he doesn't decide how they're ultimately used.

That's up to the military.

BEN: Unlike the carpet bombings of the past,

drone strikes are supposed to be precise,

limiting civilian casualties.

That said, it's tough to know exactly how many people

around the world were killed by drones in the past decade.

Estimates range from the thousands

to the tens of thousands, and it's even harder to figure out

how many of those killed or injured were civilians.

♪

But it's not just the numbers that alarm some critics.

Have you seen a US president go on TV to say,

"Tonight, I ordered airstrikes in Libya"?

Or Pakistan or Yemen?

They don't do that anymore.

Because something about drone technology and other

weapons technology has enabled US presidents and politicians

to basically shift what the norm is.

Naureen Shah is the director of Amnesty International USA's

Security and Human Rights Program.

She's a vocal opponent of the US military's

targeted killing campaigns,

and worries about the autonomous technologies in development.

So the idea that this could reduce civilian casualties

to you is impossible?

Our concern is it actually would increase civilian casualties,

increase the risk of civilian casualties.

I of course agree that if we can keep people out of harm's way,

that's vital.

But if you don't have governments having to weigh

the costs to their own citizens, when they decide to go to war,

then you're really making it so that their stakes

are a lot lower.

And that could mean that the US and other governments

are just engaging a lot more in warfare than they did before,

just not calling it warfare, and we already see that.

The US is using lethal force right now in Libya, Syria,

Iraq, Afghanistan, Yemen, Pakistan, Somalia.

There's something kind of invisible about the use of

autonomous weapons, or robots.

Something that enables policy makers to think, well,

we could go... we could use lethal force in a surgical way,

in a limited way, and we wouldn't even have to have

a big public debate about it.

We're supposedly just at war all the time.

But at what point is this technology just inevitable?

One of the things that is so difficult

is that technology is fascinating for all of us.

And it's so fascinating that it creates

almost a kind of glee.

It's a fixation that our generation has

on the possibilities provided by technology.

It can make us better than our own selves,

that somehow a robot is more selfless than a human being is,

doesn't have the prejudices of a human being, and that

can solve the fundamental gruesomeness of war.

It is a fantasy to think that you can delegate the decision

over life and death to a robot and somehow that inherently

makes it more humane and more precise and more lawful.

Because there's nothing that would give us any reason

to believe that a robot has human empathy,

that it has the ability to make a judgment about who is

a civilian and who is not a civilian, and whether or not in

those particular circumstances a civilian really poses a threat.

But some scientists completely disagree with Naureen,

and think that it is in fact possible to program a robot

to act in a more humane, precise and lawful way

than a human soldier.

And here on the Georgia Tech campus, they tried to prove it.

♪

So if you did something bad, and you felt guilty about it,

you would be less likely to do it again.

We want the robot to experience... to behave in...

A similar way, right.

The robots... None of these robots feel emotions, okay?

They don't feel anything.

They're not sentient.

But people can perceive them as feeling things.

Ron Arkin is a roboticist and professor.

He's developing technology

designed to program ethics into robots.

There are issues associated with human war fighters

who occasionally are careless, make mistakes,

and in some cases commit atrocities.

So you really think that you can program a robot to be

a better and more efficient killer than a human being,

and without making the same kind of mess ups that we might see,

say in friendly fire or civilian casualties?

I'm not interested in making them better and more efficient

killers; I'm interested in making them better protectors

of non-combatants and better protectors of civilians while

they are conducting missions than human war fighters are.

So my goal is to make them better adhere to international

humanitarian law as embodied in the Geneva conventions

and the rules of engagement.

You cannot shoot in a no-kill zone,

you cannot shoot people that have surrendered,

you cannot carry out some summary executions.

These systems should have the right to

refuse an order as well, too.

- Really? - Yeah.

This is not just a decision of when to fire,

it's also a decision of when not to fire.

So that if someone tells it to attack something that is

against international humanitarian law,

and its programming tells it that this is against

international humanitarian law, it should not engage

that particular target.

So will these weapon systems,

could they be like the Tesla self-driving car?

You know, you won't get into as many accidents.

Yes, and that's the argument that self-driving cars use.

It's the same sort of thing.

They say that human beings are the most dangerous things

on the road because we get angry, we drink,

we are distracted.

It'd be better to have the robots driving us there.

Look at 9/11.

There was no reason an aircraft should've crashed

into those buildings.

It's an easy task for a control system, now and then,

to be able to change its altitude when it recognizes

it's in a collision course and avoid that particular object.

We chose not to do that.

We choose to trust human beings over and over and over again,

but that's not always the best solution.

BEN: If there's one thing I've learned over the last few years,

it's that it's possible to hack virtually everything

running on code - from a nuclear enrichment facility to an SUV.

But what about the drone swarms of the future?

Here in Texas, there's a team of researchers that demonstrated

how military drones used today are hackable.

Todd Humphreys is the director of the Radionavigation Lab.

So this is our arena.

BEN: Nets for the drones?

TODD: The nets' for the FAA.

So there's a future, you think,

wherein you could see a ton of these things?

Dinner table-sized drones possibly, like in a swarm?

Oh yeah.

- Attacking a target? - That's right.

And they have no regard for their own life, right?

So these are suicide drones.

They pick their target, they go directly at it,

and the kinds of close-in weapons systems and...

large-scale weapon systems that our US destroyers

and the Navy have today or other ships,

they'll be no match against 16 of these at once.

Right, and it all starts here in Austin.

Well, we don't intend to do development

of war machines here.

But I will say that our somewhat whimsical games

that we'll be playing, they are going to engage

our operators and our drones in scenarios

that are applicable to all sorts of fields.

One of those so-called whimsical games he and his students

showed me was a drone version of Capture the Flag.

(Popping)

But that's not all they've been up to.

Back in 2011, when the Iranians claimed to have hacked

a US military drone, Todd and his team proved that

it was indeed possible, and demonstrated how

a simple method called spoofing could have been used

to jack US military hardware.

Every drone has a few vital links.

One of them is to its ground controller,

and one of them is to overhead satellites for navigation.

Spoofing attacks one of those vital links.

It basically falsifies a GPS signal, makes a forged signal,

sends it over to the drone and convinces the drone that

it's in a different place or at a different time, you know,

because GPS gives us both time and position.

And you've actually proven this is possible?

We've demonstrated it, yeah.

So we've done this, we've done it with a drone,

we've done it with a 210-foot super yacht.

So this is back in 2011 it was possible.

- Yeah. - How about now?

Have drones kind of caught up to this?

Have engineers realized that, you know,

when you put this autonomous thing in the air,

essentially it can be overtaken by a hostile actor?

I'd like to be able to say yes, but no.

The FAA has... has charged a tiger team to looking into this,

and they came back after two years of study and have

put together a set of proposals that would make

commercial airliners more resilient to spoofing,

more... have better defenses against spoofing.

They've also looked at even smaller unmanned aircraft.

But things move slowly in the world of commercial airliners.

And as far as smaller unmanned aircraft, I think those of us

who are just playing around with the small toys and such

aren't really thinking so much about security at this point.

So we know that hacking drones are possible, and yet

people are starting to think about creating drone swarms.

Mm-hmm.

What if your drone swarm... you send one off

and you're thinking you're gonna destroy your enemy,

and all of a sudden it turns right back around

and it comes at you because it's been hacked?

Right.

I mean, do you have to prepare it to be able to destroy

your own drone swarm?

Absolutely.

I believe you have to have a killswitch for your own swarm,

your own resources, your assets, and that

that must be an ironclad kill switch.

The only way to think about security is as an arms race.

There will always be vulnerabilities,

and you can't somehow design them out of your system

just with sheer force of will and intelligence.

♪

BEN: I'm at the Pentagon in Washington

to meet with Robert Work.

He's basically the number two at the Department of Defense,

which is why he gets the Blackhawk helicopter treatment.

We're heading out to the US Army Research Lab

at the Aberdeen Proving Ground in Maryland.

Work is leading the Third Offset Strategy, an initiative

aimed at building up the military's tech capabilities

to counter big-time adversaries like Russia or China.

The strategy calls for the US military to focus on robotics,

miniaturization, 3D printing, and autonomous systems.

So why is the US government so interested in autonomous

weapons systems and autonomous machinery for the military?

Autonomy and artificial intelligence

is changing our lives every day.

And the military would be crazy not to take these technologies

and utilize them in ways that would help us perform better

on the battlefield.

I mean, would it be crazy because other countries

are going to do it too?

We know that other great powers like Russia and China

are investing a lot of money in autonomy and AI,

and they think differently about it than we do.

You know, we think about autonomy and AI

of enabling the human to be better.

Authoritarian regimes sometimes think about taking the human

out of the equation and allowing the machine

to make the decision, and we think that's very dangerous.

That's not our conception at all.

It's more like Iron Man,

where you would use the machine as an exoskeleton

to make the human stronger, allow the human to do more;

an autonomous intelligence that's a part of the machine

to help the human make better decisions.

Would taking humans out of the loop

give your adversary an advantage?

This is a question just like cyber vulnerability

that keeps us up at night.

Would a network that is working at machine speed all the time

be able to beat a network

in which machines and humans work together?

Um, and in certain instances like...

as I said, cyber warfare, electronic warfare,

machines will always beat humans.

I mean, that will always happen.

This is a competition, and we think that the way this will go

for the next multiple decades is

AI and autonomy will help the human.

And you will never try to make it go all automatic.

But we have to watch, and we have to be careful,

and make sure that that doesn't happen.

But not everyone thinks about the Third Offset Strategy

in such an optimistic way.

In fact, some people think it's only creating a new

arms race for robotic war tools that could escalate quickly.

Naureen Shah is one of those people.

What would you say to the DOD policy-makers who are actually

looking into researching autonomous weapons systems?

I would say ban these weapons systems,

because you yourselves know what the consequences would be

if other governments had them.

Look at this issue not from the perspective just of

the US government, but from how to keep communities

all around the world safe from unlawful use of lethal force.

If the US government is concerned about what it sees as

bad actors having these weapons systems,

then it shouldn't develop them.

Once you start to manufacture these weapons systems and

authorize arms sales to all these countries,

this technology is going to proliferate,

and it will unfortunately spiral out of control.

We know that you can do it.

We know that the technology is tantalizing,

but you know that if you start down this road,

you're going to a very, very dangerous place,

and you shouldn't go there.

At this point, it might be too late.

These war toys are on the way.

And the thing is, almost every developer of autonomous machines

I met is separated from the people who actually decide

how their creations will enter war.

So while researchers toss around ideas about humane battles

or friendly killer robots, you get the feeling

conflict isn't going to be any less horrifying

using autonomous bots.

Whether it's stick and stones, guns, missiles, or drones,

people will always go to war, and it's always brutal.

Even if a robot is fighting for you.