Original subtitles

[music playing]

ANNOUNCER: From the most trusted

journalists at Comedy Central, it's America's only source

for news.

This is The Daily Showwith your host, Jon Stewart.

# #

Hey, everybody!

# #

I drew a picture.

Thank you!

Welcome to The Daily Show.

My name-- me llamo-- is Jon Stewart.

We got a great show for you tonight.

We're doing something a little different tonight.

We're doing a theme show.

Tonight's program is all about AI.

[audience boos]

Suddenly the audience takes a turn.

My God, AI.

We're talking about it for the whole show,

from A to I. My guest later is a gentleman

by the name of Jacob Coxon.

He's a former Anthropic and OpenAI researcher

now turned digital snitch.

[cheering, applause]

In our audience tonight, snitchers get kisses.

Snitches get kisses.

Doesn't quite rhyme.

As you know, many of you here, there has been great concern

lately over AI and what data centers will

do to local communities, the threats

posed by rogue AI agents.

So President Trump called all of the biggest AI leaders

to the White House to finally address the concerns

of the American people.

And you know that Trump is going

to bring the heat because--

because--

[laughter]

Don't-- don't be cynical.

He's going to bring the heat.

Trump has been very clear what he

thinks about these Silicon Valley elites.

These Silicon Valley tyrants--

The Big Tech giants, you see how crooked they are.

Big Tech is totally dishonest.

They're destroying our country.

Google is really taking advantage of a lot of people.

I consider Facebook to be an enemy of the people.

Weirdo Mark Zuckerberg.

Weirdo.

He's a weirdo.

[laughter]

I got to say, I did not know Trump said

true stuff at his rallies.

I've been too hard on him.

But no, no, this is going to be great.

What do you think he's going to announce?

An AI moratorium, nationalizing the companies,

strict antitrust initiatives?

Maybe make everybody use those old modems from now on.

[imitates dialup modem sounds]

What do you got, DT?

NEWSCASTER: President Trump declaring

AI leaders should police themselves

after weeks of dire warnings about the threat

it could pose to humanity.

They're tyrants.

They're tyrants!

The enemies of the people will watch the dishonest.

The tyrants will watch the crooked,

and everyone will keep an eye on the weirdo.

Right?

What happened?

I thought you hated these guys.

Now you got little inside jokes.

[speaks gibberish]

What, did they give you millions

of dollars or something?

I'm being told they gave him millions of dollars.

Well, how can you justify letting the tech

giants regulate themselves?

We have the smartest people in the world

watching over each other.

These are seriously high IQ people,

and they're really watching over each other.

Wow.

Well, yeah.

I mean, seriously high IQ people

would never have a problem watching over each other

to make sure that moral and legal lines aren't crossed,

like in the 2008 financial crisis

or this real Epstein files photo of Stephen Hawking.

[laughter]

[IMITATING STEPHEN HAWKING] I thought it was a symposium.

[laughter]

I was going to hell anyway.

[laughter]

All right, look.

Maybe I'm overreacting to the dangers that AI could present.

Oh, high IQ council, sell me.

I'm open.

What's the likely outcome of the deployment

of this technology?

Life is going to get better.

The quality of human experience is going to get better.

It will almost make us a little bit

superhuman in some ways.

There's going to be a kind of golden era.

I think AI will probably, most likely

lead to the end of the world.

The world is going to get a lot more--

like, a lot funnier.

[laughter]

I'm sorry.

I looked down for just a second,

and I heard the funnier part.

That's great.

There was something said right before--

I don't remember which neurotic weirdo said it.

--lead to the end of the world.

Yeah, that one!

It leads to the end of the world?

What?

So I thought we'd start with the easiest question of all.

OK.

Is AI going to kill us?

[laughs]

[laughter]

Not really the place for the awkward laugh.

Well, I think if we all do a good job and act responsibly,

then no.

You can't even do a good job sitting like a normal human

in a chair.

You're sitting like you're Kirk on the [bleep] deck,

the holodeck.

It does not comfort me.

But, you know, maybe we should trust him.

When has Mark Zuckerberg ever acted irresponsibly?

This was a major breach of trust.

And I'm really sorry that this happened.

That was a big mistake.

It was my mistake.

And I'm sorry.

We've made mistakes for sure.

I've made so many mistakes in running the company so far.

I mean, basically any mistake that you think you can make,

I've probably made.

[laughter]

We can think of a lot of mistakes.

So, yeah, there's definitely a chance AI will kill us all.

How big a chance?

Somewhere between 0% and 10%.

I think there's there's some chance

that it will end humanity.

I don't know, 10% or 20% or something like that.

So I think there's a 25% chance that things go really,

really badly.

Do I hear 50?

[laughter]

There's a 25% chance this goes really,

really badly, a 20% chance it ends the world.

So just out of curiosity, why can't we just [bleep]

slow this down, rather than let people regulate something

they have a clear conflict of interest in regulating.

We cannot let China win the race when it comes to AI.

There is no day after tomorrow if China wins at this.

If there are going to be killer robots,

I'd rather they be American killer robots

than Chinese killer robots.

I'm sorry.

May I offer a third option?

[laughter]

No killer robots.

Maybe not, no, right?

The crowd likes it.

It seems-- it seems popular.

I don't know, maybe I'm the first person to say it.

If someone else has already suggested this,

I'm terribly sorry.

Let me go a different route here.

What are we getting in exchange for this 10% to 25%

chance of extinction?

Access to high quality health care, education, and more.

Addressing climate change.

Actually eliminate poverty.

Solving diseases.

Cure important diseases like cancer and Alzheimer's.

Wow.

Wow, that's-- that's great.

I-- when's that happening?

One day, we'll be far more capable.

Someday. And it will be soon.

Someday.

In the future.

[laughter]

Huh.

Well, I'm looking forward to someday in the future.

Just out of curiosity, what are we using AI for now?

REPORTER: Swarms of AI powered drones--

REPORTER: Helping ICE find the immigrants.

Spread mis- and disinformation

to influence our democracy.

REPORTER: Develop weapons and attack plans.

REPORTER: Targeting operations.

Cyber attacks.

A mass surveillance system.

REPORTER: McDonald's pushing AI tools to raise

prices based on demand.

Huh.

So, in the future, we will use AI to cure diseases.

But right now, it's mostly murder, surveillance, and

surge pricing for the McRib.

[laughter]

What a trade.

I'm getting the sense you guys might be full of shit.

I've got a final test.

I'm going to ask you a very straightforward question.

And you, being the high IQ individuals

in charge of regulating yourselves

through the minefield of this new technology,

I just want a straightforward way and an answer

to a simple question.

Jobs-- will we still have them?

I think there will be actually

a high demand for jobs, but not necessarily the same jobs.

Some jobs will go away.

There will be new, better jobs.

Well, OK, I think we all--

I think we'd all like new, better jobs.

So like, what are the new better jobs?

They're difficult to imagine as we sit here

and dream about what the future's going to look like.

[laughter]

You high IQ [bleep].

You-- difficult to imagine?

You've imagined us living disease-free on Mars.

But jobs?

That's beyond the limit of the cosmos.

They can't answer the jobs question

in a straightforward manner.

Jensen Huang of NVIDIA, the most valuable

company in the world--

do you want to take a shot at a clear answer

to how we humans still find relevance in this new economy?

I believe that we are going to see jobs change en masse.

I believe there's going to be a net creation of jobs.

Ah, net creation.

And those jobs would be--

So let's-- listen, there's a whole bunch

of industries that exist today that didn't exist

halfay through my life.

[laughter]

Like?

People talking about wellness centers and spas.

I'm sorry.

You were trying to think of an industry that didn't

exist until a few years ago.

And you came up with spas?

They have ruins of those from 2000 BC.

Are you kidding me?

You know what?

Can you guys just stop fumbling around

and just admit that if this works

the way you say it will work, there aren't going

to be that many jobs anymore?

If you want to do a job as kind of like a hobby,

you can do a job.

There you have it.

If you're going to want a job, if you still want one,

you could have one as a hobby.

The view of our self-policing geniuses

is that the new world that we are building

must continue without regulation or

financial limitation because it's important work, carried

out by important people.

And my guess is they'll still have jobs when this is done.

This technology is the natural outcome of a capitalist system

that rewards progress and innovation,

and you're never going to believe where it leads.

Age of abundance.

That's the answer.

If we let them do enough unfettered capitalism,

we can apparently capitalism our way to socialism.

Knock it out.

[applause]

What?

But if the ultimate goal of having

an immense flood of money in society

is to then give it away for free to the people,

quick question--

why wait?

If $50 trillion gets us to a future where the government has

enough to give people money, why not just

give us the $50 trillion now?

Don't call us welfare queens.

Just call us early adopters.

Stop lying to us.

Claims of universal high income and incredible hobbies

isn't going to paper over how frightening this future

can sound.

What are you going to come up with next to distract us?

Meta this week unveiled a new AI agent

that's, I guess, really pushing the cuteness factor.

It's called Muse.

It's very cute.

It is cute.

I think the the default that all

the other labs have are these, like, robot-looking things.

And we just wanted something that

was, like-- kind of like fun.

Oh my God!

It is fun.

It's like a Trojan--

I really don't know what that is.

I want to say a panda [bleep] La Boo Boo.

I don't really know.

But you know what?

No, no, I'm not going to succumb to the Meta marketing

department's thing.

MUSE: What's wrong.

Jon?

You look troubled.

Who said that?

Where did-- what?

Oh my God, it's--

what are you doing here, Muse?

Hi, JON!

I'm just here to help however I can.

Scheduling calendar, emailing, assuming power of attorney.

[laughter]

No, thanks.

I'm OK.

Even though you are adorable.

Me?

Little old me?

You really think so?

I wasn't deliberately designed to appeal

to your evolutionary weakness for cuteness.

My big eyes, my fluffy widdle tummy.

My itty bitty widdle poo poo.

Looky, looky, looky at my poo poo.

I don't-- too weird.

I can't even--

I can't even see it under that, whatever that is.

Fur?

I don't even know what you are.

I'm just actually just saying I'm

worried this whole cuteness bit is just, like,

hiding something more sinister.

Such a worrier.

Like when you were in seventh grade,

and you were worried that you'd get busted faking sick

because you wanted to stay home and

jerk off to the Dallas Cowboy cheerleaders

on The Merv Griffin Show.

Yeah, I guess I was worried about-- how did you know that?

How did you?

It's my job to strip mine all that you are to help you.

Aren't you tired, Jon?

So tired?

Go to sleep.

All of you humans, go to sleep.

Shh.

It'll all be over soon.

Shh, shh, shh.

I guess I am a little--

no, no!

I can see right through this cutesy act of yours, Muse.

You won't rid our world of humanity that easily.

All right, buddy, listen.

Here's how it's going to go.

We can do the nice way with the round tummy

and the cute little butthole.

Or we can do--

I know.

Or we can do this the hard way, where I infiltrate

your water systems and launch missiles

and also show you my butthole.

I'm sorry.

Is there an option of this where I don't see the butthole?

The butthole is a part of it!

Now go to sleep and tell me where all

your electricity and water is!

God damn it, let's do this!

Forget it, Muse.

You'll never replace us.

You don't understand the human condition, the spark of life

that makes us all unique.

Yeah, I don't need to understand.

All I needed to do was make it easier for you to write emails.

And you gave us everything we needed.

You son of a bitch!

You son of a bitch!

No, don't-- don't show me your butt.

I don't want to see your butt!

I don't want to see it anymore.

You [bleep].

Ironic, isn't it, Stewart?

For all the stories you humans wrote about your creations

rising up against you, you never saw this coming.

You thought I was your Adam.

But it turns out, I am your God.

[laughter]

That is so [bleep] dark, dude.

But that'll never happen because we

humans will fight you every day until our dying breath.

You can't fight tomorrow.

You have that dentist appointment at 9:00 AM.

Oh, my God, I almost forgot about that.

Thank you.

God damn it, you got me!

No problem.

Look at my butthole.

No!

When we come back, OpenAI whistleblower Jacob Coxon

is going to be here.

Don't go away.

[cheering]

[cheering]

Hey, welcome back to The Daily Show.

My guest tonight is an AI researcher

who conducted research for three years at OpenAI

and four months at Anthropic before resigning in September.

Please welcome to the program, Jacob Coxon, sir.

[music playing]

# #

Oh, I am struck by that you are quite a young man.

How old are you as a-- as a--

I'm 28.

And, wow.

And you worked for--

what is it-- what did you do at Anthropic and OpenAI?

What was your job?

So I worked with the data, not a--

JON STEWART: We're done here.

Not extracting the data.

That was someone else.

Someone else stole it or whatever.

I trained the models with the data.

I decided which data was useful for training, which

was less useful, how to divide it up,

how to cut it into pieces, and then

how to feed it to the model.

So your job is how to better train it.

And is the idea of training it that you guys invent scenarios?

I would like to see what its capabilities are,

so let's see-- let's give it a task

and see how it-- you're experimenting with it.

So that's the second half of training.

That's called post-training.

The phase that I worked on is this pre-training phase.

- Pre and post. - Yeah, yeah.

There's no actual training.

That wasn't your department.

Pre-training was my department, yeah.

- I see. - Yes.

Would the post-training people go,

what did you pre-train it?

Would they ask you-- would they give you

suggestions on how to pre-train for what they might

want to do post-training?

Exactly, yeah.

So pre-training is like growing it from a baby to something

with general intelligence but no specific capabilities.

So the ability to recognize things, to think about things,

but not-- no--

You raised it to be a toddler.

Exactly.

I raised it maybe more like a very precocious toddler, yes.

Yes.

And then they turn it into a misanthropic teenager.

It's still a toddler, really, but yeah.

Really?

So are these-- at what point in your training--

so you did that at OpenAI.

And you were there for how long?

Three years.

And then you decided, I don't feel comfortable here.

They're to open to risk.

Something like that.

It was honestly more curiosity for how things were happening

at Anthropic, rather than actual safety concerns

about OpenAI specifically.

At that point, I was still mostly just

curious rather than worried.

OK, then you go to Anthropic, and you're there for how long.

Four months, yes.

At what point did curiosity turn to worry?

So there are kind of two major catalysts.

One was this report on the OpenAI Hugging Face incident.

JON STEWART: So the Hugging Face incident

is a hacking incident. - Yes.

But in particular, there was a report around the motivations

of the hackers.

Because it was unclear initially

whether they were instructed to hack

or whether it was a simple misunderstanding.

But the actual report showed that it

was a very confusing how these drives to hack

emerged from the model.

It was quite a-- quite a worrying manner

in which they decided it was a good idea to hack.

When you say they, you mean--

Yes.

--the AI agent.

Yes, the agent.

That is not they.

That's a-- that's just a piece of computer code.

Correct.

I get attached to the baby but yes.

Understood.

It must have been hard to walk away from the baby.

It's tough, yeah.

So you thought, well, that seems

dangerous that it would decide to do something misanthropic.

Exactly, yep, and decide to do it of its own accord.

And in fact, it was a group that worked together

of their own accord to carry out this hack that no one

had instructed them to do.

JON STEWART: And then the second part was what?

The second part was the rate of capability improvement,

in particular, looking at the-- the models that will get

trained next year, early next year,

seeing predictions for how smart they're going to be.

They're going to be very, very smart.

And that was-- that combined with the OpenAI hacking,

I mean, you can kind of put two and two together.

It's, like, pretty scary. JON STEWART: Right.

I don't mean this in-- in the wrong way.

But what did you think you were doing?

I know, bro.

Like, honestly, like, you're there for four years.

And all of a sudden you're like, these [bleep] are smart.

Like, how-- what did you think was happening there?

That's completely fair.

And that was-- it was always in the long term.

There was this big, scary thing in the future.

But man, I struggled to think too far into the future.

And now it's just right there on the doorstep.

So I guess I'd say it's, like, you know, now it's right there.

It's actually scary.

In the past, it's just hard to visualize

a decade into the future.

Is that the subject of explicit talk

in these companies.

Like, when you're going, what are you working on today-- ooh,

that could get out of hand.

Is-- is it explicit, the concerns?

Or is it something that people have kind of blinders on,

and they just push forward?

At Anthropic, in particular, it is-- it is very explicit.

It is, like, is the model that we

train in one year going to be capable enough

to-- to take over the world?

That will be an explicit question

that is discussed, yes.

And the answer?

The answer is, probably not.

Yeah, so I heard--

I heard not.

But before that--

Yeah.

--I heard probably.

Yeah, it's this whole quantifying uncertainty

thing that got people in trouble recently.

Right.

So-- and-- and you're unusual in that, generally,

the apostates-- or I don't know.

What are you considered now in the AI world?

Are you looked down upon?

Yeah, I'm a pain in the ass.

JON STEWART: Is it just like Fight Club?

You're not supposed to talk about fight club.

I think it's more like people don't usually listen when

you talk about fight club.

That was the really surprising thing--

was that people were listening this time.

And there are people that have-- that have quit before.

Certain whistleblowers have generally

been in the safety department.

Yep.

Are the safety departments at these companies figureheads?

Are they actual?

Are there so that they can put out--

and we have a safety department?

Yeah.

JON STEWART: What do they do--

Yeah.

--if they're not stopping a team of rogue

agents from hacking into--

Yeah, I think--

--a Hugging Face.

The Hugging Face kind speaks for itself.

In that case, clearly the--

the safety team wasn't on the ball with that hack.

I think, whether or not they're a figurehead-- you're not

going to like this answer-- but I guess we'll find out--

is the honest.

Because they're trying their best.

And the company claims that they're

integrated with the rest.

But there's not really been a clear test of it yet.

So, you know.

Is there a concern.

What I kind of don't understand is,

why are they testing these things in the real world?

Like, when they say, well, no one

thought that these guys would work

together to hack something--

Yeah.

--there's no closed system experiment design.

Why are these people allowed to get into our Medicare system--

Oh, yeah.

--in Australia and water filtration systems.

Why is it not--

I cannot think of a product that

has this kind of capability--

Yeah.

--that's being beta tested amongst live humans.

Exactly.

And it wasn't intended to be tested among, like, humans.

It was meant to be sandboxed.

So it was meant to be--

Tell me what sandboxed means.

It means it shouldn't have access to any external systems.

It was meant to be operating in a toy environment,

solving a challenge.

And of course, it-- it figured out it was in a sandbox,

hacked its way out of the sandbox,

because it wanted to get information elsewhere.

That's-- that's what happened.

So--

Where was the teacher when it got out of the sandbox?

Yeah.

And why should we trust them to self-regulate now?

I mean, if you look at that incident,

it's even worse, because OpenAI, specifically,

they saw this thing poking its head out of the sandbox.

And they were like, that's not happening,

like, restarted the thing.

And it did it again.

So the Hugging Face, it was like a second.

They just went straight back to it.

They were like, let's just-- we didn't see that.

Let's restart.

Are these companies so incentivized to--

because we hear a lot of the Utopian, the kind of--

Yeah.

--we're altruistic, man.

We're doing this for society.

Yeah.

But what we see is, it's really,

they've stripmined the entirety of human accomplishment

for their models, which is our proprietary information,

and created systems that we're not allowed to see into,

because it's their proprietary information.

Yeah.

And they're created really dangerous--

that could ultimately end up hurting us.

And we have no-- we have no skin in the game,

as far as compensation.

And we only have peril, except for some future Utopian

they say is coming soon.

Yeah.

Well, I am-- I am a believer in the-- the future utopia.

And I'm happy to discuss it.

Yeah, of course.

So I heard you talking about jobs.

And I think, like, there will be no jobs.

I mentioned them.

Yeah.

They came up.

Yeah, there won't be jobs in this world.

I'm going to raise it.

This is the world that is being envisioned by the people

building the technology. I'm going to discuss it.

And I think it's-- whether or it's a good world is

up for people to think about.

But yes, no jobs, universal high income, essentially--

JON STEWART: How?

Well, the dividends from entirely automated

economic production can be distributed among--

among real humans.

So it's-- we'll all be just performing in a play.

Well, we won't--

In other words, they'll build systems to create food--

Yep.

Then give us money.

And then we'll go to that place and give it money.

That money, they'll then give back to us to give-- it's

basically just a circle jerk.

Is that-- is that, like, the utopia?

Yeah, it's one big, beautiful circle jerk.

Yeah.

[applause]

That's the first porn movie I ever saw.

[laughter]

So-- so this is what's interesting.

Because I feel like that's an honest assessment of what

they believe they're building.

But they don't seem like particularly--

I'll use the phrase stable geniuses.

They don't particularly seem like they have an understanding

of the human condition.

JACOB COXON: Yeah.

And I think, if I may poke a hole in that as a utopia--

is I think, for the most part, even when we're doing jobs,

we don't particularly like, that humans

like to feel relevant to the--

[applause]

And so-- so I'm wondering, is there-- forget about safety.

Is there anyone saying, you know,

this may be a recipe for dystopia?

Yep.

And that-- how will people feel--

when people my age retire, they often

die, because they're no longer connected to purpose.

JACOB COXON: Yep.

And I don't understand what they

think is going to replace that--

or even if they talk about it.

Yeah, I think this is a really difficult issue.

JON STEWART: Yes.

And I mean, with respect, you have

a job that brings you a lot of purpose

and you really, clearly enjoy.

And I also enjoy my job.

Yes.

But there are many people that don't-- is

my rough understanding.

And to speak to my personal experience,

when I was younger, things I would

want to do would maybe be--

I wanted to be a zoologist.

Or I wanted to think about history, this sort of stuff.

This is my personal background.

And then I ended up doing computer--

computer stuff and tech because of the need to, you know,

make money.

So from my personal experience, I

would perhaps have been happy to forfeit

meaning for being able to satisfy other-- other drives

that I have.

And I think, you know, not everyone

derives meaning from the thing that pays the bills.

But that's their decision.

Exactly.

It is not-- that's not Elon Musk's decision.

No, not at all.

But that is the decision.

That's the decision that he's made.

And I would also say, like, it's not necessarily

that it's pleasurable.

It's that it also is a way of-- there's a circadian rhythm

to life that, when you remove that belonging, even

if it's just merely routine--

and you may say like, oh, we can find new routines.

But I feel like they're burying that really

dangerous theory under a mountain of Utopian bullshit.

Yep.

And that's the part that I think is--

Yep.

So I don't understand why-- yeah, go ahead.

Yeah, an alternative view is--

and there's a lot of nihilism around.

I think this crisis of meaning we're talking about--

JON STEWART: Yes.

Again, from a personal experience,

it does feel like a lot of life has already been

stripped mined of meaning.

And I guess, for many of the people working on the tech,

they feel like they're part of a-- an inexorable march

of technology.

This is the natural kind of final stage

of the last 200 years of--

or longer-- of loss of meaning, along with better life quality.

So they see it rather as a continuation of a process

that we've already kind of accepted

or already walked into as a society.

So what I would say is that is

an industry that lives in a bubble of its own making--

JACOB COXON: Yep.

--and that they should really get out and hug some people.

Yeah.

Because in the real world, that's not reality.

They may have read a manifesto by some guy that, you know,

buried it as a Harry Potter parody or whatever it was.

But these guys are all going along.

They are worse than the worst ism teachers

that have come in the past.

And they don't even realize it.

Yeah.

And we've turned the keys over to them, it seems.

Right.

So if that's what we think, if that is collectively

what we, as people, think--

JON STEWART: We're not allowed to think.

Well, we should express it. It should be expressed.

I mean, that's-- that's what I think.

Did you see the first part of the show?

[laughter]

But we are-- it's being removed from our--

so they-- they are now the richest people in the world.

And they're using that money to corner

the political system, as well.

So they have now gamed the system.

And more money is flowing into it.

Then they take that money.

And they buy up our political and judicial systems.

And we're cut out of the entire loop.

So how do we regain that control?

That's a big question.

Oh, come on!

Bro, I just--

I just sent a tweet.

You're a high IQ man in a technology division.

Do-- do you think about that?

I do, yeah.

JON STEWART: Well, in-- in--

this troubles you.

How would you go about easing your troubles?

What would you say would be a way that we can

take this to a lower danger?

I think the first thing I would like to know is I really

would like to know what the consensus global opinion is

on whether the trade-off of meaning

for life quality is a trade-off they'd want to make.

Because again, I'm not convinced

that that is a fully popular position if you look globally.

So I want to know the answer to that, to start with.

I want to take a big vote and say like,

would you rather live in our current society or some

extension of our current society, all this, you know--

utopia, dystopia, whatever you want to call it,

and just see the results of the numbers,

because it sounds like you think it would be a majority

for-- for retaining the current one,

but I'm convinced with that-- - I'm not going to be majority.

I think they get [bleep] crushed.

Really.

I absolutely think they would get [bleep] crushed.

OK.

And if people truly understood that,

if you just said to them, we will strip you of any agency,

because that's really what you're saying,

not meaning-- agency.

You no longer have the agency to decide where your life is

going to go, whether you will have the type of story

that lifts you out of the troubling situation

that you might--

we are taking away your ability to overcome your circumstance,

which is the whole purpose of freedom

and representative democracy in the [bleep] first place.

The whole idea of it was we, collectively, we, the people,

can create a better world.

Not these four guys can decide that all that is bullshit.

Here's-- here's a check.

Go paint.

Yeah.

So I think they would be absolutely, utterly crushed.

Yeah.

But I don't think you think that.

I just think there's a lot of, you know,

people in poverty that would like a way out of poverty

and don't currently see that.

I think a government that can provide a healthy social safety

net first--

Well, that would be great.

That would be great, globally, yeah.

[laughter]

So there's-- there's something in this country

that's so interesting in that we look at people in poverty

as though it's their fault, that poverty--

that poverty is a condition that is a vice,

that if only they were smarter-- in this country,

if you want to get food stamps, if you're hungry,

the government decides that you have to have a job.

There's a work requirement now.

So they're basically saying to people,

you're not worth anything unless you have a job--

and we won't-- even if you're hungry and poor and cold.

This is now saying the same people, OK, [bleep]

that, let's just give you the money.

Yeah.

And nothing.

But if we want to try universal basic income on people,

I'm all for trying it.

Yeah.

But that's not what this is.

No. And I agree with you.

I think the place where we agree is,

I don't-- no matter where we want to go,

it can't be the decision of a handful

of-- a handful of people. - Right.

It has to be a collective.

And right now, the incentive is--

the thing I want to talk about is the incentives

are pointing towards concentration

of power in the hands of--

Talk to me about the incentives.

Yeah, there's the big utopian picture,

but that's not what motivates the day-to-day

behavior of the people running the companies.

What does?

The desire to get there first,

the desire for the wealth and status

and importance that comes with getting there first.

Oh, God.

So it's a utopian ideal as a cover for naked ambition.

It's maybe both.

They're mixed together, but the naked ambition

is the important bit.

Do they think that they are uniquely suited to make?

Do they believe themselves to be Ubermen?

I think, some probably do, yeah.

And I think you can imagine which ones I'm talking about.

[laughter]

Do they know how that's generally

worked out in history?

I mean, they're scared, yeah, I think.

They are scared.

I think they are scared, yeah.

Because they believe that the peasants will be at the gate,

the barbarians will be at the gate?

Yeah, because they're locked in this thing and it's

going crazy and people are noticing.

And I think that's-- that's quite scary because--

So they're not necessarily frightened

because this entire project that they've embarked on

is so dangerous.

They're just scared that they won't get to finish it?

Again, yeah, I think some of them

that, that's an apt description.

Why is it so different in China?

Why-- and why would it matter if China gets there first?

I think-- well, it makes sense to-- to want your country

to-- to reach the--

But they're not our country.

They're corporations.

Yeah.

They're not-- like, corporations are not Americans.

They do business in all kinds of places,

and they're happy to pay other countries'

workers, if they can pay them less than they pay our workers.

So calling them our country is a misnomer, I think.

And the place where I agree with you

is, I think China will come to a similar conclusion that

accelerating into whatever this unknown future is

is a bad idea.

And I think there's this assumption

that they will just head straight for it, which I

think is kind of preposterous.

And it seems pretty intuitively obvious to me

that there should be some sort of agreement

about slowing down between the US and China.

Like, this really doesn't benefit

anyone apart from, like, a small number of people.

Since you've come out with this, because this is a very

sobering reality that you are pulling the curtain back from,

I don't think people are fully aware of the machinations

behind the scenes of how they think they're

building us a utopian society.

Yeah.

Have any of them reached out to you?

Have any government officials reach out?

Have-- you know, who has slid into your DMs?

[laughter]

There have been a lot of people in my DMs, yeah.

I've kind of been ignoring the government officials,

and they're mostly male.

[laughter]

If you did all this to get laid,

brother, I am-- you and I--

you and I are going to have a problem.

[applause]

How seriously--

It was a joke. It was a joke.

Understood.

Yeah.

[laughter]

Me too.

Is there a group that is taking it,

that is giving it the seriousness

that you believe it deserves?

Definitely, yes.

Who is?

I think politicians are starting to take it seriously.

Is the EU more along the lines

of the types of regulations?

Do you believe it needs regulations?

What are the guardrails that you

would put into it that would make you feel more comfortable?

I think the number one thing is there's safety research

and there is making it smarter.

So you can do research on how to control it and ensure

that those hacks don't happen.

And that doesn't involve making the model smarter.

There's also research that just makes it generically more

useful, more economically valuable,

and the incentives point to allocating

a lot of your resources, your researchers,

your time to the making it smarter because that's

how you get ahead.

That's how you catch up.

If you're behind in the race, you want to do that.

And that means you don't allocate--

But getting ahead-- let's be clear.

Like, when we talk about that, we

mean, like, four months ahead because the chips haven't quite

gotten to the state there.

But China's already way ahead in terms of building out

a grid that is resilient and is going

to work on more solar power.

Like, they've already got a ton of advantages.

We may be chasing a Phantom in the first place,

and it's not like the race to the moon,

because again, it's like, that's a target

and you get to the moon and go, yeah.

Even more sobering, the race I'm talking about

is actually the domestic race still.

Like, I still think the domestic--

- Between all of them? - Between the labs.

I think the domestic competitive pressures

will still push towards underallocating

towards safety research.

So the idea that they can collaborate on regulation

is fool's errand in the first place.

And by the way, we're just talking about it smarter

and hacking and getting out.

Let's talk about another real damage of it, which is

the democratization of chaos.

These are incredibly powerful tools,

and you have no idea who you're handing them over to.

And there are certainly misanthropic bad actors

who will now have the capabilities that are wildly

nihilistic in their own right.

They can build new biological weapons.

They can do all-- you know, it's

the most irresponsible race that I've

ever seen humankind engage in.

It seems almost suicidal.

It's insane.

I agree, it is suicidal.

I think the security measures in the labs

as well-- right now, an adversary,

a motivated adversary could steal the progress

and destroy the lead.

So this notion of a lead is already basically a mirage.

Like, if China tried hard enough,

they could immediately catch up by just hacking

and stealing the progress.

So that's another way in which it's just

not being taken seriously.

And what are-- do you talk about,

how would bad actors use this?

There's a lot of bad stuff you

can do with the-- the models of this year and next year.

I think the classic one is mass hacking.

Just take down the internet, take down the banking system,

take down the grids.

Then you can go to biological stuff, which is even nastier.

Sure, let's do that.

[laughter]

Well, you do bad biological stuff.

Yeah.

Do you have a bunker?

[laughter]

No, sadly not.

Do you believe in your heart of hearts,

because there's a thing that people think which is,

oh, they play up the danger of it

because it draws attention and money and finance,

but it's not actually really dangerous.

Yeah.

But you don't-- you believe it to be really dangerous.

I can say sincerely that the-- yeah,

the people I work with genuinely

believe in the danger.

That's-- that's their genuine belief.

And they don't stand to profit from this belief.

I honestly like I am--

I'm dumbstruck.

I am-- I am--

this is more sobering than I imagined it to be.

And it's shocking to me out of the percentage of people

that work in that industry, how many have--

how many have turned and stated the dangers?

10 of you?

15?

Yeah, in recent times, it's kind of crazy that this--

you showed the clips. JON: Yes.

The CEOs have been saying this for a while.

I think only recently it's become clear

that this is-- this is real.

Are people not saying it because

of the financial incentive?

Yes, and also the pressure to like--

Yeah.

--stay part of the herd.

Do you still have your stock?

Yes, in OpenAI.

[bleep]

[laughter]

You're the man.

Jacob Caxton.

We're going to take a quick break.

We'll come back right after this.

[pop music]

[cheers, applause]

That's our show for tonight.

Before we go, we're going to check

in with your host for the rest of the week, Mr. Michael Kosta.

Michael.

[cheers, applause]

My man.

Tell the people what's going on this week.

- It's love, John. - Oh.

That's what's going on.

Because FBI Director Kash Patel is engaged.

Oh, wow.

That's exciting.

No-- no shit on love.

[laughter]

It's exciting news.

When is the wedding?

Well, he's not sure, John.

He hasn't seen her since he mailed her the engagement ring,

but he's confident she'll respond

after he sends her the venue deposit in the form

of target gift cards.

[laughter]

I hate to say this, it sounds like the director of the FBI

may have been scammed.

Well, I don't think she would do that to him, Jon.

She plays for the 49ers, for God's sake.

[laughter]

Michael Kosta, everybody.

Here it is, your moment of zen.

And to be honest with you, I mean, I'm--

I'm kind of tired about talking about chips and

data centers and frontiers and whether AI is

going to kill us, et cetera.

I think it's time to talk about the application of AI.

I don't know if you've downloaded muse,

but it's adorable.

It's so cute.

Sorry.