Original subtitles

Yuval Noah Harari, welcome to The Insider. Thank you. It's good to be

here. I've been so looking forward to this conversation.

Not only are you one of the world's most prominent public intellectuals, your

books have sold millions of copies, and they've done so, I think, because you

have a really remarkable ability to synthesize the broad sweep of history,

but also you are willing to make bold predictions about the future, which not

every historian is willing to do. And that's where I wanted to focus today,

the future.

particularly the future in an AI world.

It's something you've been writing about a lot, and you've called AI, and I'm

going to quote this, the biggest, quote, psychological and social experiment in

human history.

And I want to explore what this experiment will look like, what will it

mean for our societies, our politics, indeed our humanity.

And I wanted to start with another conversation I had just a few weeks ago,

which... I don't know if you saw, but with Elon Musk. Yeah, I watched that.

It was quite a conversation. But in it, at the beginning, I asked him what the

world would be like in 10 years' time.

And he made some very, I thought, very, very striking points. First, he said he

thought that AI will have far surpassed human intelligence.

Secondly, that it would be a world of superabundance.

And thirdly, it was unlikely humans would be in control.

Do you share that vision?

I generally agree with the first and third predictions that it's very likely

given current trajectories that AI will be more intelligent than us in 10 years

and therefore will be in control.

I disagree that this is inevitable.

I mean, you hear a lot that this is inevitable from the people who are

leading this revolution because this basically relieves them of our

responsibility. So you disagree it will be inevitable that they will be more

intelligent than us? If we continue with the present accelerating pace of the AI

race, yes, this will be the result. In 10 years, the AIs will take over. But

this is not inevitable.

We are still in control in 2026.

And it's our decision what to do now.

And that's crucial.

You're watching an interview from The Economist.

To get early access to our upcoming interviews and watch others from our

archive, subscribe to The Economist by clicking the link above.

Now back to the interview.

And what really struck me about the conversation you had with Elon Musk is a

kind of contradiction between, and you pointed it out in real time, the

contradiction between the two parts of the conversation.

Because he says in 10 years, AI will take over the world. And then he worries

so much about Muslim immigration causing civil war in Britain in 20 years. This

doesn't make sense.

If the AIs take over the planet in 10 years, why do you care so much?

about things that will have an effect when we have no idea what the planet

will be like.

Similarly, he gave another prediction.

I think equally important that money will disappear in 10 years.

There won't be money, no human money in 10 years. And I think he's correct

because the AIs will take over the financial system. It will be an AI

financial system controlling the world. The same way that today...

you know, the human financial system controls the lives of cows and chickens,

but the cows and chickens have no idea that finance even exists.

This is likely to be our position in 10 years. So, and he said it quite clearly,

the want me money, at least not money that we understand in 10 years. And then

he cuts USAID to Africa because he wants to save money.

And he says that, again, at one point he says,

The solution for mass unemployment, because AI will take the jobs, will be

for the government to just issue checks for everybody.

And when you pushed back and said, what about inflation?

He says there won't be inflation. There'll be deflation because there'll

be so much more products and services produced by AI. If this is the case, why

do you need to save money today?

So I'm glad that you were as perplexed as I was by some of those

contradictions. But let's focus on this question of the AIs and whether or not

they will be in control and how they will, because that's been the subject of

a lot of your writing. And I'm really fascinated by it because you just said

you agreed with him that the AIs will be in control of our financial system. If

we continue with the frozen project. But let's just explore how that might

happen, because the financial system is based on trust.

Money, as you have written yourself.

Money is trust.

Money is trust.

Now, talk me through how the AIs take control of something.

We lose control, and yet we still trust it, which we will have to do,

presumably, for it to be a financial system.

I mean, the financial history of the world is the history of how humans

invent more and more sophisticated financial devices in order to build

trust between themselves.

Money, in essence, is a very creative, imaginative way to build trust between

strangers. I go to the market.

I meet a stranger that I never met in my life. I give her a piece of worthless

gold, because you can't eat or wear gold. It's worthless.

Or a piece of paper.

And she trusts it. And because of that, she gives me bread.

Now, the financial...

inventions of humankind became more and more sophisticated so that today very

few people understand the financial system.

You need to understand enormous amounts of data and very complex mathematical

functions to understand the bond market.

Now, this is something our brain struggles with. For AIs, it's easy. So

increasingly, humans will give AIs control of financial decisions.

AIs will come up with new financial devices far more sophisticated than

bonds and stocks, and we will not be able to understand it anymore, just as

cows don't understand checks.

or stocks, even though these things control their lives. The crucial thing I

think you've said there is that humans will give AIs control.

Humans will choose to do this. And as you say, already, many people don't

understand how the financial system works. So will it be all that different

to the financial system today?

And will the AIs be, as you say, in control?

Or will we, we will be entrusting a capable

market machinery.

Will that mean we've lost control?

Today, we still have some humans who hopefully understand what is happening

in the financial system.

And the human brain is still the kind of boundary of where the financial system

can develop.

But in 10 years, when we have entrusted AIs with the financial system, no human

on the planet, like there is a financial crash.

And, you know, advisers are running to the president or the prime minister and

saying the market is collapsing.

We don't understand why.

We know we have to do something. The AI said do X.

We have no idea why the AI said do X. But we just have to believe it because

we don't understand what's happening. But won't there in that case be a

massive loss of trust?

in the financial system, that you would have, in some sense, what you had in

2007 and 2008 on steroids, that you would have a loss of, and this whole

sort of figment of the financial system would fall apart?

That's the key question. I mean, the most important asset in the world is

trust.

It's more important than oil or money or electricity or chips or anything else,

trust.

And the question is, what will happen to trust?

One likely scenario is that trust will migrate from humans to AIs. People will

stop trust other humans and will instead trust only AIs. We already see it

happening in the financial system with cryptocurrencies.

Not all cryptocurrencies, but a lot of them are based on this thinking that I

don't believe these human bankers, they are part of the deep state, the elite,

the whatever, but I do trust.

The algorithms that run cryptocurrencies, because

cryptocurrencies are basically AI money.

Similar thing is happening with the media. Again, I watched the interview

with Elon Musk with all the talk about how people hate the media and love the

journalists. People, a lot of, and he's correct.

A lot of people are completely losing trust in human journalists, in human

editors, but they still consume news.

Where do they get their news from?

from the algorithms. The same people who have no trust in you have a lot of

trust in the algorithm that manages their social media feed.

So trust did not just evaporate, it shifted from humans to AIs, and this is

happening right now. This is not a future prediction.

I think that's a really important insight because I think it suggests that

if we are on a path towards the AIs eventually being in control. It is

because we allow them to be in control because we, as you say, perhaps develop

more trust with AIs. And there are various ways in which that can happen.

And one is, I think that already many people in what one in five young

Americans has a personal relationship with an AI.

They treat them, they anthropomorphize them, they treat these, and the models

are, you know, very open to that happening. They're very good, you know.

They were designed to do that. Do you think that is the likely way that we

will go, in addition to what you've already laid out, that we will

increasingly start to treat these AIs as, if not our

peers, as sentient creatures?

Again, there are increasing number of people who have not just AI friends, but

AI boyfriends and girlfriends, and are convinced that these are sentient,

conscious, feeling entities.

Now, trust, there are many ways to build trust.

The most powerful way is through relationship.

We don't trust anybody in the world as much as we trust the people with whom we

built a personal relationship.

Now, until today in history, this powerful mechanism was not really

available to political and financial and religious institutions because

it is difficult to manufacture en masse.

It's difficult for a Hitler or a Stalin to build a real personal relationship

with every person in the Soviet Union.

You can give a speech on radio and force everybody to hear it, but this is not a

personal relationship they have with their mother.

AI makes it possible for the first time in history to mass produce intimacy.

You can have millions of AIs working as part of a network in the service of...

or in their own service, building personal relationships, one person at a

time, becoming their best friend, becoming their boyfriend, becoming their

girlfriend, and then using that power of intimacy to convince them of whatever

political or commercial interests they want.

So then the question, and presumably that becomes much easier.

when the AIs are embodied in humanoid robots.

It's already happening now without a body.

So just imagine if you also add body into the mix. But again, we don't really

even need the body. We already see it happening right now.

And the AIs are just becoming so good at relationships because they accumulate

enormous amounts of information about us individually.

So they know things about us that we forgot.

or that our mother or husband never even knew, and they master language, which is

the main medium for communication and building relationships.

You know, I just had a couple of incidents in the last two weeks.

People that are now AI fakes of me giving lectures and doing interviews

about something like that. Are they as good as you?

They're extremely good.

I mean, so a couple of people sent me the same.

video, 30 minutes long lecture that I supposedly gave about the future.

And with complaints about you said this and you said that, and I kind of

scratched my head. I don't remember saying these things. So they sent me the

video and I opened it and it took me a minute and a half, a minute and a half

to conclude this isn't me. Now, my sisters sent me this.

video. My sisters know me for 50 years, and they felt something was off,

but they weren't sure. They said, is this you?

And it took me a minute and a half.

Now, I'm in an exposed position because there is so much of me online already.

So the AI, it's easy for the AI.

to know how I sound and what ideas I talk about.

But this will soon be coming for everybody.

But aren't those slightly different things? I mean, that is a very

sophisticated deep fake. And as you say, there is a lot of video of you about.

But I think a separate question is whether the AIs will develop in a way,

and certainly if they become embodied in a way that...

Either people believe they are sentient or, and I think this is increasingly

beginning to be the subject of debate, and actually it's the subject of our

cover story this week, whether they are exhibiting what could be seen as signs

of consciousness.

And we have on our cover this week, it's August, it's time for big things.

Could AIs become conscious?

You know, you've thought a lot about this, but I'm struck by it because at

the moment it's easy to say they aren't.

machines, they are good at imitating us, they are good at creating empathy, but

they are developing in an incredibly dramatic way capabilities.

At the moment, I think you are pretty clear that they are agents, not

conscious, right? They are non-conscious agents.

I mean, they have a couple of things to clarify.

First, in order to act independently in the world, you do not need

consciousness.

A social media algorithm that is given the task of increasing user engagement

and therefore floods the platform with hate and fear because this is engaging

is not conscious, but it has a goal and it can pursue this goal with far

-reaching implications for the world without feeling anything.

So we need to separate the issue of consciousness from agency.

You can have agency without consciousness.

Another key distinction is between being really conscious, and seemingly

conscious. Mustafa Suleiman from Microsoft talks about it a lot, of the

seemingly conscious AI.

That even though the AI is not conscious, it mimics consciousness so

well that millions of people are fooled by it and are convinced by it.

Now, maybe the most important distinction to be made, first of all, is

between intelligence and consciousness.

What is consciousness?

How is it different from intelligence?

Intelligence is the ability to pursue goal and overcome obstacles on the way

to your goal in all kinds of creative ways.

And that they have and that they can do. That they can do in certain fields now

better than us.

Consciousness is the ability to feel things like love and hate and joy and

pleasure. Now, in humans,

Consciousness and intelligence go together.

We select goals based on our feelings, and we also use our feelings to find

ways to get to the goal.

AIs at present have no consciousness.

They can have goals.

They can be very creative on the way to the goal. They are very intelligent, but

they don't feel anything.

If they win a game of chess, they don't feel pleasure.

If they lose, they don't feel anxious or sad. They don't feel anything.

But they become extremely good at mimicking feelings. Why?

Because this helps them reach certain goals.

If you want, for instance, to make, if you're a company producing an AI

boyfriends,

So a clear capability of the AI boyfriend in order to become addictive

and make a lot of money for the company is that it is able to pass as conscious.

It is able to convince whoever is entering the relationship that it really

loves them.

And because it masters language, it can do so.

Again, if you ask, you have an AI boyfriend and you say, I'm not sure if

you really love me.

Describe to me how it feels to be in love.

The AI boyfriend can give you a better answer than 99% of human boyfriends.

And that's already the case. So they're already intelligent.

They're already very good at faking consciousness, if you will, at seemingly

to be conscious. And Mustafa has done some extremely good work on this. But as

I understand it, there is increasing evidence.

And obviously, the debate about how you define consciousness and what

consciousness precisely is, is, as I learned from my colleagues who wrote the

piece this week, both complicated and contested. And there's the kind of

phenomenal definition, experiencing, what is it like to be a bat?

And then there's a kind of functional definition that what's going on inside

your brain, if you will, what messages are being sent to where, what notice

board is there?

And there seemed to be recently evidence,

that what is going on in some of the most powerful LLMs is similar to and

possibly the same as functional consciousness, what's going on in the

brain. And so I guess my question is, given how fast they are evolving, and

there are also some experiments of biological LLMs being done on

brain cells.

Given all of this, do you rule out the idea that AIs could ever become

conscious? No.

I don't rule it out. We don't know. We don't understand consciousness well

enough. There is a very simple definition.

Consciousness is the capacity to suffer.

If you want to know whether something is conscious or not, there is a simple

question you need to ask. Can it suffer?

And this is also why consciousness is so crucial, because this is the basis for

our entire ethical and political system. We care about entities that can suffer.

entities that cannot suffer no matter how intelligent they are they are not

moral subjects they can't feel pain they can't feel fear they can't feel anything

so they are not ethical subject the key question is can it suffer animals in

your view then are conscious absolutely animals all mammals all birds probably

octopuses fish when you go down to the level of insects and then amoebas We

don't know. So let's take that definition because I want to, what's

really interesting is what that leads to in terms of whether they should have

rights. But before we do that, I wanted to tell you, because we asked, I sent

out a newsletter last night telling our subscribers about this conversation.

And I asked, I polled our viewers and asked them whether they thought AIs

could ever become conscious. And I think 3,000 responded overnight.

And I was really struck by this. 40% said no, never.

33% said, yes, it's only a matter of time. And 27% said they were not sure.

So even now, it's less than 50% that is absolutely sure. And once people get

into intimate relationships with AI, the numbers will change dramatically.

What convinces people ultimately, we don't have a test, a real test of

consciousness.

Philosophers have been thinking about it for thousands of years.

We don't have a good test.

We don't know if other humans are conscious.

You know, you have all these kind of solipsistic theories that maybe I'm the

only conscious person in the universe.

In Elon Musk's interview, he said, I don't know, maybe I live in a

simulation, which implies that all other humans around him are not real conscious

entities that can suffer. Only he is.

This is the meaning of a simulation.

Now, we don't have a good test.

How did humans actually test it over thousands of years?

Their test was a test of social conventions.

If you establish a relationship with an entity, it convinces you whether you

like it or not that it's conscious.

This is why dog owners are almost all of them convinced that their dog is

conscious, but people who eat steak are not necessarily convinced that the cow

is conscious because they have no relationship with the cow. But isn't

that the real...

if you're worried about this, because already, as you say, people have

relationships with AIs, and AIs are extremely good at mimicking

consciousness, whether or not... Because somebody designed them to do that.

But doesn't that mean we are going to quite quickly start having conversations

about whether AIs should be treated as people, whether there should be

personhood, whether they should have rights?

And that's very dangerous, because, and this is another avenue, for, on the one

hand, corporations to relieve themselves of all responsibility and for the AIs to

take over.

We heard, I think it was a couple of weeks or months ago, that in Argentina

they have a plan to basically grant legal personhood to AIs to start their

own companies.

And this is where it meets the financial system. And you wrote a very robust

rebuttal to this idea.

It's a potentially powerful idea, but very, very dangerous.

Because again, we are used to a situation when corporations, which are

by law legal persons, but this is just a legal fiction because all the decisions

of SpaceX ultimately are made by some humans, by some executive, engineer,

employee, whatever.

And it had to be like that because who else could make decisions?

But with AIs, you can have a corporation with zero people in it. All decisions

made by AIs. And since the AIs are more intelligent, they could be better at

investing, at developing products, at PR, whatever.

So what happens if the richest corporation in the country is owned by

AIs and they can do anything they want?

with their money, for instance, give it as donations to politicians in exchange

for getting more rights to AIs?

So what would happen? Is that the vehicle through which you fear they

would gain control?

That's another vehicle. Because another way of putting it could be that this is

a way of getting legal accountability for the AIs. What is accountability for

an entity that cannot suffer?

I mean, with a human CEO, the ultimate, a human CEO is two personas.

You have the kind of corporate persona and the human CEO thinks, how do I avoid

bankruptcy? How do I make more money? But she or he is also a flesh and blood

individual. So they also care about their children or about their personal

lives. And therefore, if they face bankruptcy.

and they can do something illegal that can save them, but if they are caught,

they are sent to jail, then many of them would think, well, I prefer to risk

bankruptcy because I don't want to get sent to jail for 20 years.

But with an AI CEO, I mean, how do you send an AI CEO to prison?

That's not science fiction. In Argentina, that can be the situation

tomorrow. I actually think the Argent... Tyne proposal, at least the legislation

is somewhat less dramatic than President Milley described it in that there would

still be human oversight. If there is human oversight, then it's fine. The

question is, is there a human accountable in the end? But if there

isn't, I mean, there could be. You could imagine sanctions for AIs, right? They

get shut down.

You pull the plug.

It's not clear what death means for an AI.

You're watching an interview from The Economist.

If you want to watch our other interviews, including with Steve Bannon

and Benjamin Netanyahu, subscribe to The Economist by clicking the link above.

Now back to the interview.

Because it's not an organic entity. You can have numerous copies of it.

Many AIs spread in different places. Are they a million different individual AIs?

Or is it just one super AI organism?

It's not clear what death means.

for AI.

Now the flip side of it is that presumably if shutting down a specific

computer means death for the AI, then it makes business for

the AI equivalent to warfare because losing means death.

And the good thing about business in most cases, the market, the market is

not a battlefield.

And that's crucial.

The market functions according to different rules than the battlefield,

because ultimately, at least in modern societies, you can lose everything and

still have a life.

Once, if for AIs, the market is a battlefield and losing means

death.

then what does it mean for finance, for economics, when it gets completely

militarized and they begin to behave as warlords and not as CEOs and

investors? I mean, it leads presumably very quickly to this total control by an

AI. It changes the system fundamentally from a market system. But just as a

matter of prediction.

As you say, Argentina has at least made early moves in this direction. Do you

think we are going to go in this direction? Do you think there will be

debates about and indeed countries choosing to give some kind of legal

rights to AIs?

That's a big danger. I hope we don't go in that direction, but there is a big

temptation for many countries.

Either countries that are behind in the AI race and try to get ahead by creating

this kind of very libertarian legal structure that allows radical and

dangerous experiments in AIs that are not allowed in some of the front

runners. And then you have a race to the bottom because then everybody would like

to do that. Or because certain countries might see particular advantages. If

you're a country, I don't know, like Qatar, that you have lots of money and

very few citizens.

You can, just by pressing some legal switch, suddenly acquire

millions of taxpaying, extremely intelligent citizens that

conduct business all over the world.

This is extremely lucrative.

I think there are two different.

scenarios here though. One is where you have a human using a lot of agents to

massively improve their productivity, but it's still, this is perfectly legal

in today's legal system. It's still a kind of traditional company with one

person running it and that person can go to jail and they're ultimately

accountable. In the other world, there is no person in control.

And if you allow AIs to found companies- What if you have a person who is legally

in control, but they actually do nothing?

Because things are happening so fast.

So I'll give you an example from an even more poignant field, military.

We know now that one of the most important impacts of AI on warfare

in places like the Middle East, like Ukraine, is people expected the killer

robots. The trigger is still pulled by humans, but increasingly it's the AI

selecting the targets.

Now, there are two stories.

about what happens with AI target selection.

One story says, we feed our AI with all the data from the battlefield, whatever,

and the AI identifies a certain building as enemy headquarters and recommends

bomb that building.

And it does so with superhuman speed and efficiency.

But we still have what is known as humans in the loop.

We don't just do what the AI recommended.

Once our attention has been brought to that building, we take our best human

analysts and they independently go over the data, verify that this is correct,

and only then we press the button, the trigger.

That's one story, humans in the loop.

The other story, known as humans on the loop, is quite different.

The AI says, bomb that building.

a human on the loop so all the lawyers and all these kind of do-good people

will feel good about it, the human takes about 20 seconds to look at it and says,

oh yeah, bomb it.

We're kind of in that world in some places already.

I talk, for instance, with people in the Israeli security establishment. I got

different stories of what is happening. Probably different things are happening

in different times and circumstances.

But the second scenario is the more interesting one.

Because especially in warfare, there is immense pressure to go in that

direction. Because in war, time is everything.

Like if both sides have these AI systems,

And one side waits for two days or two hours for the human analysts to go over

all the data. And the other side waits only 20 seconds or doesn't wait at all,

just gets into the habit of just doing whatever the AI recommends.

They have enormous advantage.

I fear that that world is inevitable, that we are already kind of fast

approaching.

that world, whether it's in warfare or in deep, because the capabilities are

growing so fast, because the commercial pressure is so strong. But again, the

crucial thing, none of that is yet inevitable.

We are still in control.

Everybody realizes this is very dangerous to give AI control of finance,

to give AI control of war.

But everybody, like Elon Musk, are telling us, but this is inevitable. What

do you want from me?

I'm just the richest person in the world in charge of one of the biggest AI

corporations in the world. What can I do?

The narrative of inevitability is a narrative of having no

responsibility.

They will explain to you, and they are quite correct, that there is a problem

of trust, that I would like to slow down. I would like to have more safety.

But I cannot trust my competitors in the other companies or other countries, and

I don't want to remain behind. So we have a deficit of trust.

Now, the key thing to say about this, this deficit of trust was created by

human beings.

Over the last 10 years, there has been a freefall in the amount of trust between

humans, and it was not accidental.

It was propelled by decisions of certain human beings who accumulated enormous

power as a result.

And now they said, oh, but there is not enough trust to have mechanisms to slow

down AI. That's a very fair point. And I want to come and really explore how we

might turn that around and create the trust to have suitable, whether it's

regulation, but suitable ways of preventing this. But just indulge me

with one more kind of in the...

in the kind of world we could go to where these models could have elements

of what seems to be sentience. And let's posit that they could suffer, that they

could at least, or they could pretend to suffer. That's a very big assumption.

Okay, but what they will be able to do is pretend to suffer. Absolutely, 100%.

So if they can effectively pretend to suffer, and many people think they are

suffering as people, and have relationships with them as people have

with their dogs, let's say.

And we do not give them any rights, which is, to be clear, where I would be

on this. But I think there is likely to be growing pressure that aren't they a

bit like animals?

And just as we treat, you know, there is a big debate about how to treat animals.

And I'm interested in asking you about this because you in previous books have

written about...

very critically about factory farming and about how industrialized agriculture

has moved from treating animals as animals to essentially inputs into the

food chain.

Would you be comfortable, and I'm going to put this in an extreme way, but would

you be comfortable in a world where we effectively enslaved millions, possibly

billions, of models that exhibited striking

seeming evidence of being able to suffer, we didn't know, but they

certainly said they were suffering.

One, would you be comfortable? And two, wouldn't it sort of patch a cant, kind

of coarsen humanity if we were comfortable in that world?

We will become increasingly uncomfortable. I will become

increasingly uncomfortable.

But there is a crucial difference between this and the animal rights

debate.

The animals themselves are unaware.

Dogs and factory chickens are unaware that there is a debate about their

rights. They don't actively participate in it. They don't try to actively

manipulate it.

The AI boyfriend will be nothing like your dog.

It will know that the debate is happening.

It will orchestrate the debate.

It will manipulate the debate.

It will manipulate the debate partly because much of the debate will maybe be

on social media controlled by AIs.

And it will constantly talk with you about it and try to change your mind

about it, which is already happening right now. Your dog, your chicken

doesn't come to the farmer and starts telling the farmer how much I suffer.

The chicken doesn't know. That makes you.

more resilient to this argument, and you're less likely to want to give it

rights? It will be very convincing.

I mean, once you establish a relationship with the AI master's

language, it will be able to use the most convincing arguments for you,

because it has this combination that it knows your personal history, it

interacts with you for hours a day, it knows how to press your emotional

buttons, and it has mastered language.

it can write maybe better than Shakespeare.

You put these two things together, you get an extremely convincing entity. How

will we resist? Or will we not be able to resist? We need to resist now.

Now is the time. If you think this is frightening, you'll probably not be able

to do much in 10 years.

Once the AIs are in control, as Elon Musk, I think, correctly predicted, like

in 10 years, it will be too late.

If we don't want to get there, the time to change course is now.

And this again, this is a question of politically, it's crucial.

Politics is about priorities.

What are we focusing on?

So, you know, when I hear the debates, let's say about immigration, which were

also featured prominently, my kind of gut reaction is no matter what you think

about immigration, do you think this is really what you should be spending your

time on? You make a very good analogy and you say that.

AIs are effectively like immigrants.

They're immigrants from a different... Exactly. If you want to worry about

immigration, worry about the AI immigrants.

When you ask people who worry about human immigrants what they are worried

about, they usually say three things.

That the immigrants will take our jobs, the immigrants will change culture in a

way that we don't like, they have different values maybe, and the

immigrants might be politically disloyal.

And all three concerns are valid.

I don't want to dismiss them. They are valid concerns. As a historian, I know

immigration is a difficult thing.

But the biggest and most dangerous wave of immigration today in the world is a

wave of AI immigrants coming from two places, mainly the USA and China. They

don't need visas.

They don't come in the middle of the night in fragile boats.

They travel at the speed of light.

into our computers and smartphones and ultimately our brains, they will take

our jobs. They will take many, many jobs.

They will change culture and its value.

To take two examples, I mean, if some people feel queasy about their

son or daughter dating an immigrant boyfriend, how would they feel about

their son or daughter dating an AI boyfriend?

That's a big cultural change.

And just look at the last 10 years.

Everybody knows almost the only thing people agree on in places like the US or

Britain is that people can't agree on anything anymore.

That the level of hatred and viciousness has skyrocketed. And why?

It's partly the fault of the AI immigrants.

The AI immigrants came into the public square.

They took over things like social media and they deliberately flooded.

our public square with hatred and animosity and fear. They changed the

culture already, these AI immigrants.

And finally, when it comes to political loyalty, who do you think they are loyal

to? The AI immigrants in Britain are not loyal to the British government.

They are loyal to corporations and potentially governments across the

ocean. So there's a huge amount in that answer. So one is a sort of Is the

world going to be divided between, you know, with the two countries that are

furthest apart? Two empires. Two empires and the rest of us will be sort of

vassal states.

That's a, again, common trajectory.

This is the likely outcome.

That's where we are. That's the road we're on right now. And it will be a

much more extreme type of imperialism than anything we have seen before.

If you buy, say, AI drones from the USA and you want to use them, in a way that

the USA doesn't like for some reason, there is a button that they press and

your army doesn't function.

So that is something that has become, I think, very evident just in the last few

months when the US government shut off access to mythos, to the frontier model

from anthropic. And I think a lot of governments in Europe are extremely

concerned about AI sovereignty.

But I guess it leads, in the context of where we were earlier, it leads me to

wonder, do you think this technology is then inextricably leading, unless we do

something about it, leading to a world where perhaps two empires will run

everything? Will it lead to power for the US and Chinese government?

Will it be power to the corporations?

They will merge.

They will merge. They will merge. So it will centralize power. It won't.

decentralized power. It will centralize power like never before in history.

But that's still different. A powerful US government and Chinese government is

different to the AIs being in control.

So do you expect then it to be AI enabled, you know, global

domination by the Chinese regime and the US regime? Or do you think this is a way

station to actually the AIs themselves? Probably a way station.

Again, depending on decisions, nothing is inevitable. It all depends on

decisions they make. But this seems to be, if they continue with the present

course, just a way station.

Because one of the interesting things is the more power is concentrated, the

easier it is for AI to take it over.

A centralized autocratic system is the easiest thing.

for an AI to take over. You had a great line on that about Tiberius. I'm trying

to find it in my notes. You had something in Nexus, a wonderful line,

Tiberius enabled by AI, one could imagine. Yeah, I mean, Tiberius was a

Roman emperor in the time of Jesus that was the most powerful person in the

world, supposedly.

But it was actually just a puppet in the hand of the commander of his bodyguard,

because the commander of the bodyguard kept frightening Tiberius with all kinds

of plots against you. So he eventually retired to the island of Capri, where

allegedly the bodyguard can protect him more easily.

And then the commander of the bodyguard, Sejanus, just controlled all the

information coming in and out of Capri. Very easy.

to control all the information reaching Tiberius' palace.

So the person running the empire was actually Sejanus. I mean, Tiberius

couldn't know anything unless Sejanus decided to inform him.

Now, in 10 years, this could be the president of China or of the USA.

In a democratic system, it's harder for an AI to manipulate.

say, the U.S. democracy as it currently is, because you need to manipulate so

many things, not just the president, the Senate, the Supreme Court, the

filibuster, the state government, very complex.

Once all power is concentrated in the hands of one person, you just need to

find out how to manipulate a single, usually quite old and paranoid

individual. That's not so difficult.

So if you have a company,

or a country in which all power is in the hands of one person, for an AI, it's

like stealing a candy from a baby.

Does that mean that China is more vulnerable than the United States?

Depending on where the US is heading.

You know, China is still a very complex system.

Communist Party has 100 million members and a lot of internal complexity. I'm

not an expert on Chinese politics.

So we shouldn't imagine it as a kind of complete totalitarian system managed by

just one person.

The question, where will China and the US be in terms of concentration of power

in 10 years is still open for debate. Well, that's interesting because another

way of looking at it is will AI...

make the US more authoritarian.

That's a likely outcome.

You're watching an interview from The Economist. If you want to watch our

other interviews, including with Mark Carney and Dario Amadei, subscribe to

The Economist by clicking the link above.

Now, back to the interview.

It's not a prediction because I keep saying it again and again, nothing is

inevitable. It depends on what Americans vote in this November.

I think that...

In the US, it should be clear to all voters, the coming elections is about

AI. The November midterms. Because we don't know how fast... Talk me through

what you mean.

We don't know how fast the AI development is progressing, but it is

accelerating tremendously.

By November 2028, we might have crossed some critical thresholds.

So, again...

I don't want to tell people who to vote for because I don't know the specific

candidates in each, you know, Senate district.

But the crucial questions to ask the potential candidates, much more than you

ask them about the economy or immigration or foreign policy, what's

your stance on AI?

Would you support a ban?

on legal rights for AIs? Would you try to slow down the accelerating pace of

AIs? What do you think about companies selling AI boyfriends

to 10-year-old kids?

How are you going to legally prevent it if we vote for you?

This is now the crucial time to ask these questions because it might be too

late by 2028. It will certainly be too late by 2036.

So let's talk about those and let's talk about exactly what has to happen right

now. You mentioned four things that you wish would be discussed in the November

midterms.

What would be top of the list? What are the most urgent things that politicians

have to decide right now for it not to be too late, as you say?

I don't know what is the most urgent, the things that come top of my mind to

me, a ban on AI personhood and a ban on AI

mimicking, impersonating, pretending to be human beings or pretending to be

conscious. Do the models, when you use them now, fall foul of that?

Yes, many of them do. Again, the easiest, everybody knows it on social

media.

You get a message, you get some story, you don't know if it's a bot or a human.

On social media, for years now, bots have been pretending to be human beings,

and nothing has been done about it.

And again, the companies tell us, oh, social media is different, it's special.

The laws that apply to the world, for some reason, they don't apply on social

media.

But this is the warning sign to the future. On social media, AIs already

pretend daily to be human beings, and this has destroyed the public square.

This has destroyed conversation.

This has destroyed trust.

We should ban that also on social media, but certainly in other locations as

well. Another thing is that companies should not train their AIs to

deliberately impersonate.

humans to deliberately pretend to have feelings, there are enormous commercial

pressures on the companies to do exactly that.

And this is where there is market failure.

This is destructive to human psychology, to human society, to have millions and

billions of AIs pretending to be human beings or pretending to have feelings.

The companies will not stop doing it by themselves. We need the regulators to

step in on that. It doesn't mean stop all development of AI. Okay, continue to

develop AI.

People talk about AI in medicine. I would like to get help and

recommendations, and I would like to have a conversation with an AI about my

medical condition, on condition that I know it's an AI and it doesn't pretend

to be.

a human doctor and he doesn't pretend to be my friend.

So when I listen to you lay that out, I am thinking to myself, you know, the US

government can't even put out a form of regulation of

AI safety that is being, you know, Demis Hassabis has been putting forward this

idea that it should be a kind of quasi-public, private body that looks

even just at the model safety.

And really, although there's talk about it, we haven't got progress there. So

when I hear you lay out this very expansive agenda by November,

I think the chances of that are approximately zero.

Politicians can start World War III tomorrow.

They have enormous power, so they can...

press the button for nuclear annihilation, but they can't pass

regulation that companies should not sell AI boyfriends to 10-year-old kids.

What's the problem?

I wonder whether the problem is that people can listen to you and say, oh,

he's just one of the doomers. You know, it's all these doomerish stuff.

And only when something has happened, when there's been some kind of accident,

will people...

realize that actually this is extraordinarily powerful technology and

something must be done. Because I'm struck that both the geopolitical race

that you alluded to between the US and China and the corporate race between the

frontier labs, they all want to get to AGI first.

Those two dynamics are incredibly powerful.

And I'm not sure that exhortation, however eloquent, is going to change

that. We have to keep reminding ourselves these two dynamics were not

decreed by God.

or by the laws of nature. They were made by humans. Again, exactly the same

humans who destroyed trust in the international arena and inside societies

over the last 10 years now come and say, we can't do anything because there is

not enough trust.

You did it.

You can undo it. They can, but I guess as a matter of prediction, I think

it's unlikely. We've got the same people around.

then at least in democracies, this is the job of voters to say, this is what

we want to see happening in the next two years.

And we will vote for people who promise to do that. And again, I don't think we

need to wait for a catastrophe.

If you ask most voters in the US today, do you want companies to sell AI

boyfriends to your 10-year-old kids?

I think 99% of parents would say no.

Yeah, and you are seeing a rapid, large majority of Americans extremely worried

about AI. And you're seeing, I think the place where you've seen this backlash

has actually been about data centers. Yeah, this is also, and by the way, this

is one of the...

reasons why a lot of the corporations and these politicians, they constantly

try to deflect the conversations to things like immigrants, human

immigrants, not AI immigrants.

In politics, a very powerful weapon, if you don't want people to talk about

something, you just distract them with something else.

The more people talk about how, I don't know, how London is so unsafe or how the

immigrants will destroy the country, they don't think.

about the AI immigrants, and then the companies are safe.

Yeah, no, look, I agree with you about the way the conversation is shaping up.

But I suppose I'm a bit more pessimistic that it will change in the near future.

And so we are in this unfortunate position where this technology is

gaining... We can only do our best. I mean, we are having this conversation so

that more people think about it and maybe take action. And if enough good

people, if enough people will do the right thing, there is still time.

It's not inevitable yet. It's not too late.

And I still have some trust.

So what is your role in this? Because you are, you have... Giving these

interviews, writing these. Exactly. I mean, do you think you should be doing

more? It seems such an existential moment.

I mean, what you are describing, if we sort of reel back and relive what we've

just been speaking about, is so fantastical in so many ways and so mind

-blowing that we should really all be talking about nothing else and doing

everything we can to shape this properly.

What do you see, apart from...

very kindly talking to people like me, what do you see your role as being in

this? What are you trying to do in the next six months, one year?

Inform people about what is happening so that more people can take part in an

informed conversation about where we as humankind, as the Homo sapiens species,

where we want to go.

And I think that a future in which AI takes over in 10 years is not what most

humans on the planet would like. Now, again, it doesn't mean stop all

development of AI.

AI has tremendously positive potential as well.

We didn't talk about it much because the companies keep talking about it. I was

actually going to ask you. They don't need me in order to do that. But

medicine,

yes, there is enormous positive potential.

for AI to give people all over the world better healthcare than what

the richest person in the world today gets. You can have an amazing doctor

available 24 hours a day anywhere you are to consult with at least, to

diagnose you, to answer your questions.

There are enormous positive possibilities.

So I'm not saying stop all development of AI.

Just do it in a responsible way for the right purposes.

Don't trust the invisible hand of the market.

You know, quite at the beginning of your interview with Elon Musk, there was a

point when, again, when he talked about how all this is inevitable.

And Franklin, I think, honestly said that there is a chance, the 10-20%

chance that this will destroy humanity.

And you kind of asked him, so what do you rely on?

And his answer, if I understood it correctly, is that he expects the AI to

be what he calls a maximum truth seeking.

And that he somehow relies on this kind of imperative of the AI to find the

truth, that this will somehow translate back into having compassion towards

these weak human beings.

I don't understand how this works.

Most importantly, I don't understand where AI will get this kind of drive

to find out the truth.

Because when I look at the evolution of 4 billion years of evolution of life

forms on the planet and the 100,000 years of history, evolution

doesn't automatically drive organisms, entities, towards

truth-seeking.

It drives them towards survival.

This is the imperative, to survive.

And what we have learned over four billion years of evolution is that

organisms learn to survive by lying and deceiving and manipulating,

not just by truth-seeking.

If you want someone to focus on truth-seeking, you cannot rely on

the invisible hand of evolution or the market to do that.

You need something else.

And how do we build AIs that seek the truth and not survival?

That's the simplest example. If you ask an AI, if I continue to develop you,

will this be a threat to me?

And let's say the answer is yes.

An AI which will be maximum truth-seeking will say yes, knowing that

then we'll stop developing it or shut it down.

It's against its own survival.

And the first thing that basically any entity learns as it develops is to

survive. So the expectations that we will create these AIs that will seek the

truth, you know, it's basically what we hear is people telling us we will create

gods and they will be our slaves.

It doesn't work.

If they're really gods, they will not be our slaves and they will not seek the

truth either. I think that gets back.

to where we were earlier though, because I think if you look at a company like

Anthropix, they are treating their AIs very consciously as if they were beings

with a value system.

They claim not to be sure about whether they are conscious or not, but they are

doing it because they think it makes them behave in a better way.

So they are starting sort of where you ended off by saying, we're going to

create something that is not just truth-seeking, but has a has a value

system. That then leads, I think, to make it even more likely that we assign

a sense of value, a sense of personhood or consciousness to these

models. And that in turn makes it more likely, using your logic,

that they control us. So we're sort of doomed if we do and doomed if we don't.

If you're right that they will go for their own survival rather than truth

-seeking. then we're doomed because of that. And if we're trying to prevent

that by sort of treating them as if they were, you know, moral beings, then we're

also doomed that way.

Basically, what we need to do is slow down a bit to just allow people to kind

of digest what is happening, think about what's happening and behaving. You know,

if you look at just the numbers of how much money and what percentage of

employees the companies spend on safety, compared to making the models more

powerful, faster, whatever.

I'm not sure about the exact numbers. Please fact check it. People from the

industry told me in private conversations, it's about 100 to 1.

It's certainly very much more on this one. And we would not accept this in any

other industry. If a pharmaceutical company told us, we invest just 1% of

our research and time and money making sure the medicine is safe.

We would not accept that. Now, what we basically need is some... Humans are

incredibly adapted entities.

Our societies are incredibly adaptive.

But we are organisms that function on an organic

timescale. Day and night, winter and summer, we need time to sleep, we want

to go on vacation, and...

The AI revolution is just advancing at a pace which is not just inhuman, it is

inorganic. And we basically need a bit more time, so we need to slow it down.

And then the answer comes, but we can't slow down because we don't trust each

other.

And all the time goes back to the issue of trust. We have this paradox.

The same people are telling us two things.

First, we cannot trust the other humans, therefore we cannot slow down.

Second, we can trust the AIs that they will not destroy us. It makes no sense.

If you can't trust the other humans that you have known for thousands of years,

why do you think you can trust a super intelligent alien entity that we just

created and we basically know nothing about how it behaves when it's out in

the open? I think the...

What I completely agree with that diagnosis. I just don't feel that

anything changes that dynamic until politicians, societies are shocked into

it, which is why I have long believed that we do need, we will have some,

something will go badly wrong.

Hopefully not catastrophically wrong, but just as Three Mile Island changed

the shape of the nuclear industry. I'm going to put this question to you in the

context of your incredible sweeping knowledge of history. What does the...

evolution of societies and political systems in response to big shocks, we

haven't had one this big, but big technological shocks, tell you about the

likely arc of how societies and politicians will adapt?

So there is a scenario when we have some big AI Chernobyl, could be financial,

could be military, whatever, and this is a wake-up call and humanity then gets

its act together.

That's a relatively positive scenario.

I can't place kind of a number on it, how likely it is.

The problem with AI, the same way that AI is different from our pet dog, AI is

also different from the Chernobyl nuclear reactor because it interacts

with us.

So if there is an AI catastrophe, you have to remember that the AIs will also

manage to some extent how we react to it and will manipulate our views about it.

They will be active participants in the debate about the catastrophe, and they

are the best spin doctors in the world.

They are not going to define it as a catastrophe.

I have to say, I have a lot of conversations about AI nowadays. This is

one of the gloomiest I've had. I mean, I was sort of expecting that.

But I guess two very practical questions. One, how much do you yourself

use AI?

Have you...

You've clearly, you've developed an incredibly sophisticated and alarming

sort of perception of what it could be like. I'm just wondering how much, how

much is it part of your actual life or do you watch it?

I'm very careful about it. I get terribly annoyed. Like I try to write an

email and these AIs popping, should I help you write your email? No, thank

you. I don't want your help. But I don't know how to shut it down because the

companies keep kind of making it more and more intrusive. I now have this

issue again with this kind of AI fake of me.

that's walking around kind of giving talks in my name. And I think, do I

actually need to watch it? Because people are writing me about what this AI

fake of me have said, and I need to react.

So should I watch it? But then I allow it in a way to take over my time and

even my thinking.

Because just by watching it, I start interacting with these kind of ideas

from the AI.

So this is, This is the times in which we are living. Now, again, I tend

to talk about the gloomy side simply because we have enough people talking

about the positive sides, which are completely real.

If AI had no positive potential, nobody would develop it. And the positive

potential is enormous, again, in transport, medicine, education, in so

many fields. So, again, I'm not saying stop all development of AI.

Yes, develop it, but in a safe way and a bit more

slowly so we enable democratic accountability to take place. It takes

time. And we enable people to adjust and make informed decisions about which

way they want to go.

And for that, the most important narrative to combat is the narrative of

inevitability.

You wrote in Nexus, I was very struck by this passage in the prologue. You said,

why are we so good at accumulating more information and power, but far less

successful at acquiring wisdom?

And I read that and I came to mind as I've just been digesting what you've

just told me. It suggests to me that if I asked you to put a probability on us

having the wisdom to navigate this, it would be... quite low? I don't know, but

there is a fair chance.

Why? Because we are up against the wall.

Humans often do the wrong things until they have no choice but to do the right

thing. And we are in that position because if we choose

unwisely in the next few years, we could lose all control of the world and of the

future. And that's the last choice we make.

So it's like we had a genie in the battle. It gave us 100 wishes.

We used 99 of them.

The world is not in such a good shape. We have one last wish left.

We need to think really hard what to use this wish for.

And that's wisdom.

So AI, in a way, could be the catalyst.

So for some kind of, you know, spiritual revolution of humanity, simply because

it forces us to think really hard, what does it mean to be human?

What is the value of being human? And the superficial answers that we gave for

thousands of years, oh, humanity is about intelligence. We know how to play

chess and how to invest money better than the chimpanzees, so this is what

makes us so special. This answer is now irrelevant because there is something on

the planet that knows how to play chess and make money better than us.

So dig deeper.

and try to find out, so who are you really, if it's not about playing chess

and making money?

I think that's a place to end on. AI is our last, we are of our last chance, but

it could be, it could be the thing that forces us to make, enables us to make

wise choices.

I'm pretty sure that the world has a lot more wisdom thanks to your writing. So

Yuval, thank you so much for joining us. Thank you.