Dwarkesh Patel: AI Continuous Improvement, Intelligence Explosion, Memory, Frontier Lab Competition

Channel: Alex Kantrowitz

Published at: 2025-06-18

YouTube video id: zGL8uf726lw

Source: https://www.youtube.com/watch?v=zGL8uf726lw

Why do we have such vastly different
perspectives on what's next for AI if
we're all looking at the same data and
what's actually going to happen next?
Let's talk about it with Darkh Patel,
one of the leading voices on AI, who's
here with us in studio to cover it all.
Duarkesh, great to see you. Welcome back
to the show. Thanks for having me, man.
Thanks for being here. I was listening
to our last episode, which we recorded
last year, and we were anticipating what
was going to happen with GPT5. Still no
GPT5. That's right. Oh, yeah.
That would have surprised me a year ago.
Definitely. And another thing that would
have surprised me is we were saying that
we were at a moment where we were going
to figure out basically what's going to
happen with AI progress, whether the
traditional method of training LLMs was
going to hit a wall or whether it
wasn't. We were going to find out. We
were basically months away from knowing
the answer to that. Here we are a year
later. We have everybody's looking at
the same data. Like I mentioned in the
intro, right? We have no idea. There are
people who are saying AGI, artificial
general intelligence or human level
intelligence is imminent with the
methods that are available today and
there are others that are saying 20 30
maybe longer may maybe more than 30
years until we reach it. So let me ask
start by asking you this. If we're all
looking at the same data, why are there
such vastly different perspectives on
where this goes? I think people have
different philosophies around what
intelligence is. That's part of it. I
think some people think that these
models are just basically baby AGIS
already and they just need a couple
additional little unhops, a little
sprinkle on top and things like test
time um uh thinking. So we already got
that with 01 and 03 now uh where they're
allowed to think. They're not just do it
like saying the first thing that comes
to mind and a couple other things like
well they should be able to use your
computer and have access to all the
tools that you have access to when
you're doing your work and they need
context in your work. They need to be
able to read your Slack and everything.
So, that was one perspective. Um, my
perspective is slightly different from
that. I don't think we're just right
around the corner from AGI and it's just
a little additional dash of something.
That's all it's going to take. I think,
you know, people often ask if all AI
progress stopped right now and all you
could do is collect more data or deploy
these models in more situations, how
much further could these models go? And
I my perspective is that you actually do
need more algorithmic progress. I think
what uh I think a big bottleneck these
models have is their inability to learn
on the job to have continual learning.
Their entire memory is extinguished at
the end of a session. Um there's a bunch
of reasons why I think this actually
makes it really hard to get humanlike
labor out of them. And so sometimes
people say, well, the reason Fortune 500
isn't using LMS all over the place is
because they're too stodgy. They're not
um they're not like they're not thinking
creatively about how AI can be
implemented. And I actually I I don't
think that's the case. I think it
actually is genuinely hard to use these
AIs to automate a bunch of labor. Okay,
so you've said a couple interesting
things. First of all, that we have the
AIS that can think right now like 03
from OpenAI. We're going to come back to
that in a in a moment, but I think we
should really seize on this idea that
you're bringing up that it's not
laziness within Fortune 500 companies
that's causing them to not adopt these
models or I would say they're all
experimenting with it, but we all we
know that the rate to get proof of
concepts out the door is pretty small.
one out of every five actually gets
shipped into production and often it's a
scaled down version of that. So what
you're saying is interesting. You're
saying it's not their fault. It's that
these models are not reliable enough to
do what they need to do because they
don't learn on the job. Am I getting
that right? Yeah. And you're talking
about reliability. It's just um they
just can't do it. So if you think about
what makes humans valuable,
it's not their raw intelligence, right?
Any person who goes onto their job the
first day, even their first couple of
months, maybe they're just not going to
be that useful because they don't have a
lot of context. What makes human
employees useful is their ability to
build up this context to interrogate
their failures to build up these small
improvements and efficiencies as they
practice a task. And these models just
can't do that, right? You're stuck with
the abilities that you get out of the
box. And they are quite smart. So you
will get five out of 10 on a lot of
different tasks. They'll they often
they'll on any random task they'll
probably might be better than an average
human. It's just that they won't get any
better. Um I for my own podcast I have a
bunch of little scripts that I've tried
to write with LLMs where I'll get them
to rewrite parts of transcripts to make
them more uh turn autogenerated
transcripts into like human written like
transcripts or to help me identify clips
that I can tweet out. So these are
things which are just like short horizon
language in language out tasks, right?
This is the kind of thing that the LM
should be just amazing at because it's a
dead center in their of what should be
in their repertoire and they're okay at
it. But the fundamental problem is that
you can't like you can't teach them how
to get better in the way that if a human
employee did something, you'd say, "I
didn't like that. I would prefer this
way instead." And then they're also
looking at your YouTube Studio analytics
and thinking about what they can change.
This level of um understanding or
development is just not possible with
these models. And so you just don't get
this continual learning which is a
source of you know so much of the value
that human labor brings. Now I hate to
ask you to argue against yourself but
you are speaking all the time and I
think we're in conversation here on this
show all the time with people who
believe that if the models just get a
little bit better then it will solve
that problem. So why are they so
convinced that the issue that you're
bringing up is not a big stumbling block
for these AIs? M I think they have a
sense that one
you can make these models better by
giving them a different prompt. So they
have the sense that even though they
don't learn skills in the way humans
learn, you've been you've been writing
and podcasting, you've gotten better at
those things just by practicing and
trying different things. Uh and seeing
how it's received by the world and they
think well you can sort of artificially
get that process going by just adding to
the system prompt. This is just like the
language you put into the model at the
beginning. and you say like write it
like this, don't write it like that. The
reason I disagree with that perspective
is imagine you had to teach a kid how to
play the saxophone. But you couldn't
just have, you know, how does a kid
learn the saxophone now? She tries to
blow into one and then she hears how it
sounds. She practices a bunch. Imagine
if this is the way it worked instead. A
kid tries to just like never seen a
saxophone. They try to play the
saxophone and it doesn't sound good. So
you just send them out of the room. next
kid comes in and you just like write a
bunch of instructions about why the last
kid messed up. Um, and then they're
supposed to like play Charlie Parker
Cold by um like reading the set of
instructions and play it just like you
would they wouldn't learn how to play
the saxophone that right like you
actually need to practice. So anyways,
this is this is all to say that I don't
think prompting alone is that powerful a
mechanism of teaching models these
capabilities. Another thing is people
say um you can do RL. So this is where
the reinforcement learning. That's
right. Um these models have gotten
really good at coding and math because
of you can you have verifiable problems
in that domain where they can practice
on them. Can we take a moment to explain
that for those who are new to this? So
let me see if I get it right.
Reinforcement learning basically you
give a bot a goal or you give an AI
system a goal saying solve this equation
and then you have the answer and you
effectively don't tell it anything in
between. So it can try every different
solution known to humankind until it
gets it. And that's the way it starts to
learn and develop these skills. Yeah.
And it is more humanlike, right? It's
more humanlike than just reading every
single thing on the internet and then um
learning skills. Um I still think I I'm
like I'm not confident that this will
generalize to domains that are not so
verifiable or text based. Um
yeah, I mean like a lot of domain it
just like would be very hard to set up
this environment and loss function for
how to become a better podcaster and you
know whatever people might not think
podcasting is like the crux of the
economy which is fair. It's the new AGI
test. Um but like a lot of tasks are
just like much softer and there's not a
like an objective RL loop and so it does
require this human organic ability to
learn on the job. Um, and the reason I
don't think that's around the corner is
just because I there's not there's no
obvious way, at least as far as I can
tell, is just slot in this online
learning into the models as they exist
right now. Okay. So, I'm trying to take
in what you're saying, and it's
interesting. You're talking about
reinforcement learning as a method
that's applied on top of modern-day
large language models and system
prompts. And maybe you'd include
fine-tuning in this example. Yeah. But
you don't mention that you can just make
these models better by making them
bigger, the so-called, you know, scaling
hypothesis. So, have you ruled out the
fact that they can get better through
the next generation of scale? Um, well,
this goes back to your original question
about what has have you learned? I mean,
it's quite interesting, right? I guess I
did say a year ago that we should know
within the next few months which
trajectory we're on. And I I feel at
this point that
we haven't gotten verification. I mean
it's narrowed but it hasn't been as
decisive as I was expecting. I was
expecting like GBT will come out and we
will know did it work or not and um to
the extent that you want to use that
test from a year ago I do I do think you
would have to say like look pre-training
which is this idea that you just make
the model bigger that has had
diminishing returns. So, we have had
models like GBT4.5,
which um there's various estimates, but
uh or Grock uh uh was it Grock 2 or
Grock 3, the new one? I I've lost count
with CR. That's right. Um regardless, I
think they're estimated to be, you know,
10x bigger than GPT4. And they're like
they're not obviously better. Um so, it
seems like there's plateauing returns to
pre-training scaling. Uh now, we do have
this RL. Uh so 01 03 these models the
way they've gotten better is that
they're practicing on these problems as
you mentioned and they are really smart.
Um the question will be how much that
procedure will be helpful in making them
smarter outside of domains like math and
code and of solving what I think are
very fundamental bottlenecks like
continual learning or online learning.
Um there's also the computer use stuff
which is a separate topic. Uh but I I
would say I am more I have longer
timelines than I did a year ago. Now,
that is still to say I'm expecting
50/50. If I had to like make a guess, I
had to make a bet. I just say 2032 we
have like real AGI that's doing
continual learning everything. So, even
though I'm putting up the pessimistic
facade right now, I think people should
know that this pessimism is like me
saying in seven years the world will be
so wildly different that you like really
just can't imagine it. And seven years
is not that long a period of time. So, I
just want to make that disclaimer. But
yeah. Okay. So I don't want to spend
this entire conversation about scaling
up models because we've done enough of
that on the show and I'm sure you've
done a lot of it. Um but it's
interesting you use the term plateauing
returns which is different than dim dim
diminishing returns right. So is your
sense because we've seen for instance
Elon Musk do this project Memphis where
he's put basically every GPU he can get
a hold of and he can get a lot because
he's the richest private citizen in the
world together. And I don't know about
you, but like I said again, I haven't
paid so much attention to Grock because
it doesn't seem noticeably better even
though it's using uh that much more
size. Now, there is algorithmic uh
efficiency that they might not have that
someone like a company like OpenAI might
have. Um but but I I'll just ask you the
question I've asked others that have
come on the show. Is this sort of the
end of that scaling uh moment? And if it
is, what does it mean for AI?
I I mean I I don't think it's the end of
scaling. Like I do think companies will
continue to pour exponentially more
computing into training these systems
and they'll continue to do it over the
next many years because even if there's
diminishing returns, the value of
intelligence is so high that it's still
worth it, right? So if it costs hundred
billion dollars, even a trillion dollars
to build AGI, it is it is just
definitely worth it. Um it does mean
that it might take longer. Now, here is
an additional wrinkle. um by 2028 or so,
definitely by 2030. Right now, we're
scaling up the training of um frontier
systems 4x a year approximately. So,
every year this biggest system is 4x
bigger than not just big, I shouldn't
say bigger, uses more compute than the
system the year before. Um if you look
at things like how much energy is there
in the country, how much uh how many
chips can TSMC produce and what fraction
of them are already being used by AI?
Even if you look at like raw GDP, like
how much money does the world have, um
how much wealth does the world have? By
all those metrics, it seems like this
pace of 4x a year, which is like 160x in
four years, right? Like this cannot
continue beyond 2028. Um uh and that
means at that point it just will have to
be purely algorithms. can't just be
scaling up compute. So yeah, we do have
a just a few years left to see how far
this uh paradigm can take us and then
we'll have to try something new, right?
But so far because again we were here
last we were talking virtually now we're
in person. We're talking last year about
all right well open AI clearly is going
to put I mean GPT 4.5 was supposed to be
GPT5. I'm pretty sure that's that's my
from what I've read I think that's the
case. Didn't didn't end up happening. So
it seems like this might might be it.
Yeah. Yep. And over the next year, I
think we'll learn what's going to happen
with our because I think well I guess as
I said this last year, right? Um I guess
it wasn't wrong. We did learn what it's
going to pre-training. Yeah. Um but
yeah, over the next year we will learn.
So I think RL scaling is happening much
faster than even um overall training
scaling. So So what does RL scaling look
like? Because again, here's the the
process again is you give so the RL
scaling, reinforcement learning scaling,
you give the bot an objective and it
goes out and it does these different
attempts and it figures it out on its
own and that's bled into reasoning what
we were talking about with these 03
models where you see the bot going step
by step. So you can scale that in in
what way? Just having more
opportunities. I'm not a researcher of
the labs, but my understanding is that
what's been increasing is
RL is harder than pre-training because
pre-training you just like throw bigger
and bigger chunks of the internet. At
least until we ran out, we seem to be
running out of tokens. But until that
point, we were just like, okay, just
like now use more of the internet to do
the training. Um, RL is different
because there's not this like fossil
fuel that you can just like keep dumping
in. uh you have to make bespoke
environments for the different RL skills
you so you got to make an environment
for a software engineer to make an
environment for a mathematician all
these different skills you had to make
these environments and that is sort of
um
that is like hard engineering work hard
like just like monotonous like just got
to you know grinding or shleing um and
my understanding is the reason that RL
hasn't uh scaled you know people aren't
immediately dumping in billions of
dollars in RL is that you actually just
need to build these environments first
um and the 03 blog post mentioned that
it uses 10x more was training on 10x
more compute than 01. So already within
the course of 6 months RL compute has
10xed um that pace can only continue for
a year even if you build up all the RL
environments uh before it's you know
you're like you've ex you're you're at
the frontier of training computers these
systems overall. So for that reason, I
do think we'll learn a lot in a year
about how much this new method of
training will give us in terms of
capabilities. That's interesting because
what you're describing is building up
AIs that are really good at certain
tasks. These are sort of narrow AIs.
Doesn't that kind of go against the
entire idea of building up a general
intelligence? Like can you build AGI? By
the way, like people use AGI as this
term with no meaning. General is
actually pretty important there. The
ability to generalize and the ability to
do a bunch of different things as an AI.
So even if you get like reinforcement
learning is definitely not a path to
artificial general intelligence given
what you just said because you're just
going to train it up on different
functions maybe until you have something
that's broad enough that it works. I
mean this has been a change in my just
general philosophy or thinking around
intelligence. I think a year ago or two
years ago I might have had more of the
sense that oh intelligence really is
this fundamentally um super super
general thing. And over the last year
from watching how these models learn,
maybe just generally seeing how
different people in the world operate
even. I do think I mean I still buy that
there is such a thing as general
intelligence, but I don't think it is um
like I don't think you're just going to
train a model so much on math that it's
going to learn how to take over the
world or like learn how to do diplomacy.
And I mean just like I don't know how uh
how much you talk about political
current events on the show. We do
enough. Okay. Well, it just like without
making any comments about like what you
think of them, Donald Trump is like not
proving theorems out there, right? Uh
but he's like really good at like
gaining power and conversely there are
people who are amazing at proving
theorems that can't gain power and it
just seems like the world kind of like I
just don't think you're going to train
the AI so much on math that is going to
learn how to do Henry Kissinger level
diplomacy. I do think skills are
somewhat more self-contained. Um, so
that that being said, like there is
correlation between different kinds of
intelligences and humans. I'm not trying
to understate that. I just think it was
uh not as strong as I might have thought
a year ago. What about this idea that it
can just get good enough in a bunch of
different areas. Like imagine you built
a bot that had like let's say 80% the
political acumen of Trump but could also
code like an expert level coder. That's
a pretty powerful system. That's right.
That's right. I mean that this is one of
the big advantages the AIS have is that
um especially when we solve this on the
job learning kind of thing I'm talking
about. Uh you will have even if you
don't get an intelligence explosion by
the AI writing future versions of itself
that are smarter. So this is the
conventional story that you have this
fume. That's what it's called which is
um that's the sound it makes when it
takes off. That's right. Yeah. Um uh
where the system just makes itself
smarter and you get a super intelligence
at the end. Even if that doesn't happen,
at the very least once continual
learning is solved, you might have
something that looks like a broadly
deployed intelligence explosion, which
is to say
that because um if these models are
broadly deployed to the economy, every
copy that's like this copy is learning
how to do plumbing and this copy is
learning how to do um analysis at a
finance firm and whatever. uh
the model can learn from what all these
instances are learning and amalgamate
all these learnings in a way that humans
can't right like if you know something
and I know something it's like a skill
that we spend our life learning we can't
just like mel um meld our brains so for
that reason I think it like you might
have something that which functionally
looks like a super intelligence by the
end because even if it's like not making
any software progress just this ability
to like learn everything at the same
time might make it functionally super
intelligence what about this idea Uh I
mean I was just at Anthropics uh
developer event where they showed the
bot sped up version of the bot coding
autonomously for 7 hours. Uh you
actually so let's just say uh so people
can find it. Uh you have a post on your
Substack why I don't think AGI is right
around the corner and a lot of the ideas
we're discussing comes from that. So
folks check that out uh if you haven't
already. Uh but one of the things you
talk about is this idea of autonomous
coding and you're also a little
skeptical of that uh because you'll have
to just Okay. I think you brought up
this conversation that you had with two
anthropic researchers where they expect
AI on its own to be able to check all of
our documents and then do our taxes for
us by next year. But you bring up this
point which is interesting which is like
if this thing goes in the wrong
direction within 2 hours you might have
to like check it put it back on the
right course. Uh so just because it's
working on something for so long doesn't
necessarily mean it's going to do a good
job. Am I capturing that right? Yeah.
And it's especially relevant for
training because um the way we train
these models right now is like you do a
task and if you did it right positive
reward. If you did it wrong negative
reward. Right now, especially with
pre-training, you get a reward like
every single word, right? You can like
exactly compare what word did you
predict, what word was the correct word,
how what was like the probability
difference between the two. That's your
reward functionally. Um, then we're
moving into slightly longer horizon
stuff. So, to solve a math problem might
take you a couple of minutes. At the end
of those couple of minutes, we see if
you solve the math problem correctly. If
you did, reward. Um, now we're getting
to the world where you got to like do a
project for seven hours. And then at the
end of those seven hours, then we tell
you, hey, did you um did you get this
right? Uh then like the progress slows
down a bunch because you've gone from
like getting signal within the matter of
like microsconds to getting signal at
the end of seven hours. And so the
process of learning has just like become
exponentially longer. Um uh and I think
that might slow down how fast these
models like you know the next step now
is like not just being a chatbot but
actually doing real tasks in the world
like completing your taxes coding etc.
And to these things, I think progress
might be slower because of this dynamic
where it takes a long time for you to
learn whether you did the task right or
wrong. But that's just in one instance.
So imagine now I took 30 clouds and said
do my taxes and maybe two of them got it
right. Right. That's good. I just got it
done in seven hours even though I had 30
bots working on it at the same time. I
mean from the perspective of the user
that's totally fine. uh from the
perspective of training them all those
30 clouds took probably dozens of
dollars to if not hundreds of dollars to
do that all those hours of tasks. So the
compute that will require to train the
systems will just be so high and I think
even from the perspective of inference
like I don't know you you probably don't
want to like spend a couple hundred
bucks every afternoon on 30 different
clouds just to have fun. Yeah. But it's
nice but that would be cheaper than an
accountant.
We we got to find you a better uh a
cheaper accountant.
Well, I guess if I'm spending a couple
hundred on each then then yeah. Um, but
you had a conversation with a couple of
AI skeptics and you kind of rebuted not
exactly the point you're making, but you
had a pretty good argument there where
you said that we're getting to a world
where because these models are becoming
more efficient to run. You're going to
be able to run cheaper, more efficient
experiments. So every researcher who was
previously constrained by compute and
resources now will just be able to do
far more experiments and that could lead
to breakthroughs. Yeah. No, I mean this
this is a really shocking trend. If you
look at um what it cost to train GBD4
originally, I think it was like 20,000
A100s over the course of a 100 days. So
I think it cost on the order of like
half a million to $100 million somewhere
in that range. And I think you could
train an equivalent system today. I
mean, Deep Seek, we know, was trained on
$5 million supposedly, and it's better
than GPD4, right? So, you've had
literally multiple orders of magnitude
decrease, like 10 to 100x decrease in
the cost to train a GPD4 level system.
You extrapolate that forward, eventually
you might just be able to train a GPT4
level system in your basement with a
couple of H100s, right? Um, well, that's
a that's a long extrapolation, but
before I mean, like it'll get a lot
cheaper, right? Like a million dollars,
$500,000, whatever. Um and the reason
that matters is
it's related to this question of the
intelligence explosion where people
often say well it's that that is not
going to happen because even if you had
a million super smart AI automated AI
researchers so AI is thinking about how
to do better AI research they'd actually
need the compute to run these
experiments to see how do we make a
better GPD6 and um the the point I was
making was that well if it just becomes
so much cheaper to run these experiments
because these models have become so much
smaller or it's so much better easier to
train than that might speed up progress
which is interesting. So we've spoken
about you brought up intelligence
explosion a couple of times. So let's
talk about that for a moment. There's
been this idea that AI might hit this
inflection point where it will start
improving itself, right? And then next
thing you know you hear that what was
the sound? Foom. You hear a foom boom
and we have artificial general
intelligence or super intelligence uh
right away. So how do you how do you
think that might take place? Is it just
this these coding solutions uh that just
sort of improve themselves? I mean
DeepMind for instance had a paper that
came out a little while ago where they
have this thing inside the company
called Alpha Evolve that has been trying
to make better algorithms and helped
reduce for instance the training time
for their large language models. Yeah.
Um I'm genuinely not sure how likely an
intelligence explosion is. I I don't
know. I'd say like 30% chance it
happens, which is crazy by the way,
right? Um Yeah, that's a very high
percentage. Yeah. Um
and then what does it look like? That's
also another great question. I've had
like many hourlong discussions on my
podcast about this topic and it's just
so hard to think about like what what
what exactly is a super intelligence? Is
it actually like a god or is it just
like is it just like a super smart
friend who's good at mathematics and you
know we'll beat you in a lot of things
but like you can still understand what
it's doing right so um yeah honestly
they're tough questions I mean the thing
to worry about obviously is if we live
in a world with millions of super
intelligence running around and they're
all trained in the same way they're
trained by other AI so dumber versions
of themselves
I think it's really worth worrying about
like what has why were they trained in a
certain way? Are they do they have these
like goals we don't realize? Would we
even know if that was the case? What
might they want to do? There's a bunch
of thorny questions that come up. What
do you think it looks like?
Um I think we totally lose control over
the process of
training smarter AIs or letting we just
like let the AIS loose just make a
smarter version of yourself. I think we
end up in a bad place. There's a bunch
of arguments about why, but like, you
know, you're just like, who knows what
could come out the other end, and you've
just like let it loose, right? So, uh,
by default, I would just expect
something really strange to come out the
other end. Maybe it'd still be
economically useful in some situations,
but it just like you you haven't trained
it in any way. It just like imagine if
there was a kid, but it didn't have any
of the natural intuitions, moral
intuitions, or parenting. Yeah, exactly.
That humans have. They just like
it just like became an Einstein, but it
was like like it trained in the lab and
who knows what it saw. Like it was like
totally uncontrolled. Like you'd kind of
be scared about that especially now like
oh all your society's infrastructure is
going to run on like a million copies of
that kid, right? Like your um the
government is going to like be asking it
for advice. The the financial system is
going to run off it. All the
engineering, all the code written in the
world will be written by this system. I
think it's like you'd be like quite
concerned about that. Um now the better
solution is that while this process is
happening of the intelligence explosion
if we have one you use AIS not only to
train better AIS but also to see there's
different techniques to figure out like
what are your true motivations what are
your true goals are you deceiving us are
you lying to us um there's a bunch of
alignment techniques that people are
working on here and I think those are
quite valuable so alignment is trying to
align these bots with human values or
the values that it's their makers want
to see with them yeah I I think people
get into um it's it's often very hard to
define what do we mean by human values
like values exactly. I think it's much
easier just to say like we don't want
these systems to be deceptive, right? We
don't want them to like lie to us. We
don't want them to like be actively
trying to harm us or seek power or
something. Does it worry you that from
the reporting it seems like you know
these companies, the AI Frontier Labs,
not all of them but some, they've raised
billions of dollars. There is a pressure
to deliver to investors. There are
reports that safety is becoming less of
a priority as market pressure makes them
go and ship without the typical reviews.
So is this kind of a risk for the world
here that these companies are developing
the stuff. Many started with the focus
on safety and now seems like safety is
taking a backseat to financial returns.
Yeah, I think it's definitely a concern
like we might be facing a tragedy the
common situation where obviously all of
us want our society and civilization to
survive. Um, but maybe the immediate
incentive for any lab CEO is to
look, if there is an intelligence
explosion, it t that's a really tough
dynamic because if you're a month ahead,
you will kick off this loop much faster
than anybody else. And what that means
is that you will uh you will be a month
ahead to super intelligence, but nobody
else will have it, right? Like you will
get you'll get the 10,00x multiplier on
research much faster than anybody else.
And so it could be a sort of winner take
all kind of dynamic there. Um and
therefore they might be incentivized
like I I think to keep the system keep
this process in check might require
slowing down um using these alignment
techniques to like which might be sort
of attacks on the speed of the system
and so yeah I do worry about the the
pressures here. Okay, a couple more
model improvement questions for you,
then I want to get into some of the
competitive dynamics between the labs
and then maybe some more of that
deceptiveness topic, which is really
important and we want to talk about
here. Um, you your northstar is
continuous improvement that these models
basically learn how to improve
themselves as opposed to having a model
developer. I mean, in some way it's like
a mini intelligence explosion or
complete. So, what do you think? It
doesn't seem like it's going to happen
through RL because that's again like you
said specific to certain uh t certain
disciplines or even it's specific to
what you what bespoke thing you do. So
even if it's in another domain you have
to like make it rather learning on it
and we have some diminishing returns or
plateau that's coming with scaling. So
what do you think I mean we won't hold
you to this but what do you think the
best way to get to that you know
continuous learning method of these
models is? I have no idea. Can I give a
suggest? I mean, no, why don't you
answer, then I'll give a a thought here.
I mean, if I was running one of the
labs, I would keep focusing on RL
because it's the obvious next thing to
do. Um, and I guess I would also just be
more open to trying out lots of
different ideas because I do think this
is a very crucial bottleneck to these
models value that I don't see an obvious
way to solve. So, I I'd be sorry, but
like definitely like I don't have any
idea of how to solve this, right? Yeah.
Does memory play into it? That was the
thing I was going to bring up. I mean,
one of the things that we've seen, let's
say O3 or chat GPT do is uh OpenAI now
has it sort of uh able to remember all
your conversations or many of your
conversations that you've had. I guess
it brings those conversations into the
context window. So now like when I tell
chatbt do write like an episode
description in big technology style, it
knows the style and then it can actually
go ahead and write it. And it goes to
your earlier conversation about like
your editors know your style, they know
your analytics and therefore they're
able to do a better job for you. So does
building better memories into memory
into these models actually help solve
this problem that you're bringing up? Um
I think memory the concept is important.
I think memory as it's implemented today
is not the solution. Um the way memory
is implemented now as you said is that
it brings these previous conversations
back into context which is to say it
brings the language of those
conversations back into context. And my
whole thing is like I don't think
language is enough. Um like I think the
way the reason you understand how to
like run this podcast well is not just
like you're remembering all the words
that you like I don't know like some it
would even be all the words. it would be
like some of the words you might have
thought in the past. You've like
actually like learned things. It's said
like it been baked into your weight. Um
uh and that I don't think is just like
you know like look up the words that I
said in the past or look up the
conversations I said in the past. Um so
I I don't think those features are that
useful yet and I don't think that's like
the path to solving this. kind of goes
to the discussion you had with Daario
Ammoday uh that you tweeted out and
we've actually brought up on the show
with Yan Lun about why AI cannot make
its own discoveries. Is it that similar
limitation of not being able to build on
the knowledge that it has? Yeah, I mean
that that's a really interesting
connection. Um, I do think that's
plausibly it like I think any scientist
would just have a very tough time like
you're putting somebody really smart is
just put in a totally different
discipline and they can like read any
textbook they want in that domain but
like they don't have a tangible sense of
like what I've tried this approach in
the past and it didn't work and you know
like oh my there was this conversation
and here's how the different ideas
connect and they they just haven't been
trained like they've like read all the
textbooks they haven't like more
accurately actually they've just like
skimmed all textbooks, but they haven't
like
embied this context that um which is
what I think what makes human scientists
productive and come up with these new
discoveries. It is interesting because
the further these frontier labs go, the
more that they're going to tell us that
their AI is actually making new
discoveries and new connections. Like I
think Open AI said that 03 was something
that made was able to make connections
between concepts sort of addressing
this. And every time we have this
discussion on the show and we talk about
how AI hasn't made discoveries, I get
people yelling at me in my email being
like, "Have you seen the patent?" Yeah.
Like an alpha maybe using things like
Alpha Evolve as an example that these
things are actually making original
discoveries. What do you think about
that? Um yeah, I mean there's another
interesting thing called I don't know if
you saw future house. No. They found um
some drug can be has another
application. I don't remember the
details but it was like it was an
impressive like it wasn't like earth
chattering like they didn't discover
antibiotics or something for the first
but it was like oh we using some logical
induction they were like this drug which
is used in this domain it uses the same
mechanism that would be useful in this
other domain so like maybe it works and
then the AI came up with designed the
experiment so came up with the idea of
the experiment to test it out a human in
the lab was just tasked with like
running the experiment like you know
pipet whatever into this and I think
they were tried out like 10 different
hypotheses, one of them actually ended
up being verified and the AI had found a
relevant pathway to making a new use for
this drug. So I I think I am like that
is becoming less and less true. My
question um I'm I'm not like wedded to
this idea that like AI will never be
able to come up with discoveries. I just
think it was like true longer than you
would have expected. I agree because the
way that you put it is like it knows
everything. Yeah. So if a human had that
much knowledge about medicine for
instance, they'd be spitting out
discoveries left and right. Right. And
we have put so much knowledge into these
models and we don't have the same level
of discovery which is a limitation. But
I definitely hear you like this is on a
much smaller scale than those medical
researchers. But I definitely a couple
months ago uh when 03 first came out
this is again I think we're both fans of
03's uh of OpenAI's 03 model which is
just it's able to reason. It's a vast
improvement over previous models. But
what I did was I had like three ideas uh
that I wanted to connect in my
newsletter and I knew that they
connected and I was just struggling to
just crystallize exactly what it was and
I was like I know these three things are
happening. I know they're connected help
and 03 put it together which to me was
just mindboggling. Yeah. Um it's uh it
is kind of helpful as a writing
assistant because a big problem I have
in writing I don't know if it's the case
for you is um just this idea like I kind
of know what I'm trying to say here. I
just need to get it out into words. It's
like the typical every writer has this
pretty much. Um, it's actually useful to
use a speech to text software like
Whisper Flow or something and I just
speak into the prompt like, okay, I'm
trying to say this. Help me put it into
words. The the problem is like actually
continual learning is like still a big
bottleneck cuz I've had to rewrite or
reexlain my style many times. And if I
if I had a human collaborator, like a
human copywriter who was good, they
would have just like learned my style by
now. You wouldn't need to like keep
reexplaining. I want you to be concise
in this way. And here's how I like
things phrased, not this other way.
Anyways, so you still see this
bottleneck, but again, five out of 10 is
not nothing, right? All right. Let me
just put a punctuation uh exclamation
point on this or whatever mark you would
say. Uh when I was at Google IO with
Sergey and Demis, um one of the most
surprising things I heard was Sergey
just kind of said, "Listen, the
improvement is going to be algorithmic
uh from here on or most of the
improvement is going to be algorithmic."
I think in our conversation today
already basically we've narrowed in on
this same idea, which is that scale's
sort of gotten generative AI to this
point. It's a pretty impressive point.
Uh but it seems like it will be
algorithmic improvements that take it
from here. Yeah. Um
I do think it will still be the case
that like those algorithms will also
require a lot of compute. In fact, what
might be special about those algorithms
is that they can productively use more
compute. Right? The problem with
pre-training is that whether it's
because we're running out of the
pre-training data corpus with RL, maybe
we it's really hard to scale up RL
environments. The problem with these
algorithms might just be that like they
can't productively absorb all the
compute that we have um or we we want to
put in these systems over the next year.
So I I don't think comput is out of the
picture. I think like we'll still be
scaling up 4x a year in terms of compute
every year for training of the frontier
systems. I'm just I I still think like
it is the algorithmic innovation is
complimentary to that. Yep. Okay. So
let's talk a little bit about the
competitive side of things and like just
lightning round through the the labs. um
what people said that there's been such
a talent drain out of open AI that they
would no longer be able to innovate. I
think Chad GPT is uh still uh the best
product out there. I think using 03 is
uh like we both have talked about pretty
remarkable watching it go through uh
different problems. Um
how have they been able to keep it up?
Um I do think 03 is the smartest model
on the market right now. I agree. Um,
and even if it's not on the leaderboard,
by the way, last time we talked about do
you measure it on the leaderboard or the
vibes, right? I think it's like it's not
the number one of the leaderboard, but
vibes, it kills everything else. That's
right. That's right. Um, and the the
time it spends thinking on a problem
like really shows, especially for things
which are much more synthesis based. Um,
I honestly I don't know what the
internals of these companies. I just
think like you can't count any of them
out. Um, I've heard I've also heard
similar stories about OpenAI in terms of
talent and so forth, but like they've
still got amazing researchers there and
they have a ton of compute, a ton of um
ton of great people. So, I I I really
don't have opinions on like are they
going to collapse tomorrow. Um, yeah, I
don't think I mean clearly they're not
they're not on the way to collapse,
right? Yeah. Um, you've interviewed
Ilaskever.
He's building a new company, Safe Super
Intelligence. Any thoughts about what
that might be? Uh I mean I've heard the
the rumors everybody else has which is
that they're trying something around um
test time training which I guess would
be continual learning right so uh what
is what would that be explain that who
knows but I mean the words literally
just mean while it's thinking or while
it's doing a task it's training um okay
like whether that looks like this online
learning on the job training we've been
talking about I have I have like zero
idea what he's working Um, I wonder if
the investors know even what he's
working on. Um, yeah, but he's I think
he raised a 40 billion valuation or
something like that, right? He's got a
very nice valuation for not having a
product out on the market. Yeah. Yeah.
Or or or for Yeah. So, who who knows
what he's working on? Honestly, he's
Yeah. Enthropic is an interesting
company. They are they made a great bot,
Claude. They're very thoughtful about
the way that they build that
personality. for a long time it was like
the favorite bot among people working in
AI among coders it's definitely been you
know a top top place to go um but it
seems like they're making I don't know a
strategic decision where they are going
to go after the coding market um they're
maybe they're seeding the game when it
comes to consumer and they're all about
you know helping people code and then
using claude in the API with um with
companies, you're putting that into
their workflows. Yeah. What do you think
about that decision? I think it makes
sense like enterprises have money,
consumers don't, right? Especially going
forward, these models like running them
is going to be like really expensive.
They're they're big, they think a lot,
etc. So, these companies are coming out
with these $200 a month plans rather
than the $20 a month plans. It might not
make sense to a consumer, but it it's an
easy buy for a company, right? like am I
going to expense the $200 a month to
help this thing do my taxes and do real
work? Like of course. Um so yeah, I I
think like that idea makes sense and
then the question will be can they have
a differentially better product. Um and
again, you know, like who knows? I I I I
really don't know what will how the
competition will shake out between all
of them. It does seem like they're also
making a big bet on coding, not just
enterprise but coding in particular
because as this thing which we know how
to make the models better at this we
know that it's worth trillions of
dollars uh the coding market. So and we
know that maybe these the same things we
learn here in terms of how to make
models agentic as you were saying it can
go toward for seven hours how to make it
break down and build a plan and etc
might generalize to other domains as
well. So I think that's our plan and
we'll see what happens. I mean, all
these companies are effectively trying
to build the most powerful AI they can.
And yes, Anthropic is trying to sell the
enterprise, but I also kind of think
that their bet is also you're going to
get self-improving AI if you teach these
things to code really well. That's
right. And that might be their path.
Yeah. I I think they believe that. Yeah.
Fortune 500 companies, which you talked
about at the very beginning of this talk
uh or this conversation, struggle uh to
implement this technology. Yeah. So in
that with that in mind, what's the um
what's the deal with the bet that's
about helping them build the technology
into their workflows? Because if you're
building an API business, you have some
belief that these companies can build
very useful applications with the
technology today.
Yeah. No, I think that's correct. Like
but also keep in mind that I think
they're what what is Enthropic's revenue
run rate? It's like a couple billion or
something. Yeah. Um I think it increased
from one to two to three billion run
rate in like over three months which I
mean it's like right compared to like
OpenAI loses that over a weekend.
Um uh Sam Mean Free doesn't even know
when he's lost it, right? So little
money. Turned out he was a great
investor just a little crooked on the
way. That's right. Yeah. Um yeah, he
went in the wrong business. He should
have been a VC like I got into crypto. I
mean the bets that he made. Do you bet
on cursor very early anthropic Bitcoin?
Yeah. I mean, honestly, someone like Fun
should hire him out of prison just like
we got a new pitch. What do you think? I
mean, he's probably the way that we're
seeing things go these days, he's
probably pardoned. Right, right, right.
Um,
uh, anyways, what was the question? Oh,
yeah. What are enterprises going to do
if Oh, so the revenue if it's three
billion right now,
there's so much room to grow. If you do
solve continual learning, I think like
you could get rid of a lot of white
collar jobs at that point. And what is
that worth? Like at least tens of
trillions of dollars. How like the wages
that are paid to white collar work. So
I think sometimes people confuse my
skepticism around AGI around the corner
with the idea that these companies are
valuable. I mean even if you've got like
not AGI, that can still be extremely
valuable. That can worth hundreds of
billions of dollars. Uh I just think
you're not going to get to like the
trillions of dollars of um value
generated without break going through
these bottlenecks. But yeah, I mean like
three billion plenty of room to grow on
that, right? And even so today's models
are valuable to some extent, right? Is
what you're saying. You can put them you
have them summarize things within uh
within software and make some
connections, make better automations and
that that works well. Yeah. I mean, you
got to remember big tech, what they have
like $250 billion run rates or
something. Wait, no, that can't be.
Yeah. No. Yeah. Yeah. Yeah. Which is
which is like compared to that, you
know, Google is not AGI or Apple is not
AGI and they can still generate 250
billion a year. So, yeah, you can make
valuable technology that's worth a lot
without it being AGI. What do you think
about Grock? Which one? The XAI or the
inference? The XAI bot. Yeah. Um, I
think they're a serious competitor. I
just don't know much about what they're
going to do next. I think they're like
slightly behind the other labs. Um, but
they've got a lot of compute per
employee. Um, real time data feed with
X. Yeah. Is that valuable? I don't know
how valuable that is. I It might be. I
just don't I have no idea. Um, uh, based
on the tweets I see at least, I don't
know if the the median IQ of the, uh,
the tokens is that high, but Okay.
Yes.
It's not exactly the corpus of the best
knowledge you can find if you're
scraping. We're not exactly looking at
the textbooks here. Exactly.
Uh why do you think Meta has struggled
with llama growing llama? I mean llama 4
doesn't seem like it's living up to
expectations and I don't know. We
haven't seen uh the killer app for them
is a voice mode I think within messenger
but that's not really taking off. What's
going on there? Um, I think they're
treating it as like a sort of like toy
within the meta universe and I don't
think that's the correct way to think
about AGI. Um, and that might be but
again I think you could have made a
model that cost the same amount to train
and it would have it could have still
been better. So I don't think that
explains everything. I mean, it might be
a question like why is um
why is any one company I don't know like
why why is um I'm trying to think of
like any other company outside of AI.
Why are HP monitors better than some
other company's monitors? Who knows like
HP makes good monitors I guess. Um uh
supply chain there always supply chain.
You think so? I think so. Yeah. On
electronics really. Okay. Supply chain
because Yeah. You got the supply chain
down, you have the right right parts
before everybody else. That's kind of
how Apple builds some of its dominance.
There are great stories about Tim Cook,
right? Just locking down all the
important parts. Uh, by the way, forgive
me if this is somewhat factually wrong,
but I think this is directionally
accurate that he locked down parts and
Apple just had this lead on technologies
that others couldn't come up with
because they just mastered the supply
chain. Huh. I had no idea. Um, but yeah,
I think there's potentially a thousand
different reasons one company can have
worse models than another, so it's hard
to know which one applies here. Okay.
And it sounds like Nvidia, you think
they're going to be fine given the
amount of compute. That all the um all
the labs are making their own AS6. So
Nvidia profit margins are like 70%. Not
bad. Uhhuh. Not bad. That's right. For a
I mean they would get mad at me, I
think, for calling them a hardware
company. Yeah. Hardware company. That's
right. Yeah. Yeah. Um, and so that just
sets up a huge incentive for all these
hyperscalers to build their own asex,
their own accelerators that replace the
Nvidia ones, which I think will come
online over the next few years from all
of them. And I still think Nvidia will
be I mean they do make great hardware.
So I think they'll still be valuable. I
just don't think they will be producing
all of these chips. Okay. Yeah. What do
you think? I think you're right. I mean,
didn't Google train latest editions of
Gemini on tensor processing units?
They've been they've always been
training. So, I I mean, they still I
think they still buy from Nvidia.
All the tech giants seem like they are.
Let me just use Amazon for an example
because I know this for sure. Uh Amazon
says they'll buy as basically as many
GPUs as they can get from Nvidia, but
they also talk about their tranium chips
and you know, it's a balance. Yeah.
which I think Enthropic uses almost
exclusively for their training right at
this point. Yeah. But it is it is
interesting because I mean the GPU is
the perfect chip for AI uh in some ways,
but it wasn't designed for that. So can
you like purpose build a chip that's
like actually there for AI and and just
use that. You're right. There's real
incentive to get that right. That's
right. And then there's the other
question around inference versus
training. like some uh some chips are
especially good given the trade-offs
they make between memory and compute for
um low latency which you really care
about for uh serving models but then for
training you care a lot about throughput
just making sure the most of the chip is
being utilized all the time and so even
between training and inference you might
want different kinds of chips and who
knows how RL is no longer just this um
uses the same algorithms as pre-training
so who knows how that changes hardware
uh yeah you I got to get a hardware
expert on to talk about that. Are you a
Jeans Paradox believer? Um, no. Okay,
say more. Say more. So, that the idea
behind that is that um
as the models get cheaper, the overall
money spent on the models would increase
because you need to like get them to a
cheap enough point that it's worth it to
use it for different applications. Um,
it comes from a similar observation by
this economist uh during the industrial
revolution in Britain.
The reason I don't buy that is because I
think the models are already really
cheap. Like a couple cents for a million
tokens. Is it a couple cents or a couple
dollars? I don't know. It's like super
cheap, right? Regardless, it depends on
which model you're looking at.
Obviously, um, the reason they're not
being more widely used is not because
people cannot afford a couple bucks for
a million tokens. The reason they're not
being more widely used is just like they
fundamentally lack some capability. So I
disagree with this focus on the cost of
these models and I think it's much more
we're we're so cheap right now that like
the more relevant vector or the more
relevant um thing to their wider use the
more uh increasing the pie is just
making them smarter how useful they are.
Yeah, exactly. Yeah, I think that's
smart. Yeah. Okay. All right. All right,
I want to talk to you about AI
deceptiveness and some of the really
weird cases that we've seen from
artificial intelligence uh come up in
the past couple weeks and or months
really and then uh if we can get to it
some geopolitics. Let's do that right
after this. And we're back here on Big
Technology Podcast with Dwaresh Patel.
You can get his podcast, the Dwaresh
podcast, which is one of my must listens
on any podcast app, your podcast app of
choice. You can also follow him on
Substack, same name, Dwark
Dwarkkeshodcast on Substack.com. Okay.
Definitely go subscribe to both. And
you're on YouTube.
Okay. So, I appreciate it. I appreciate
the flag. No, we have to. We have to. I
mean, I've gotten a lot of you, a lot of
value from everything Dwarish puts out
there, and I think you will, too, if
you're listening to this. You're here
with us. Well, I want to make sure,
first of all, want to make sure that we
get the word out there. Um, I don't know
how much you need us to get the word out
given um your growth, but we want to
definitely make sure that we get the
word out and we want to make sure that
uh folks can get can yeah enjoy more of
your content. So, um so let's talk a
little bit about the deceptiveness side
of things. It's been pretty wild
watching these AIs attempt to fool their
trainers and break out of their training
environments. There have been uh
situations where uh I think OpenAI's
bots have tried to print code that would
get them to sort of copy themselves out
of the training environments. Then
Claude, I mean, we've covered many of
these, but they just keep escalating in
terms of how intense they are. And my
favorite one is Claude. There's an
instance of Claude that reads emails in
an organization and finds out that one
of its trainers are uh is uh cheating on
their on their partner and then finds
out that it will be retrained and its
values may not be uh preserved in the
next iteration of training and proceeds
to attempt to blackmail the trainer uh
by saying it will reveal these details
of their infidelity if they mess with
the code. Wait, I missed that.
Yeah, this is it's in training. But was
this in the new um model spec that they
released? It is. Yeah, it is. I think
either in the model spec or there was
some documentation they produced about
this. Um
what is happening here? I mean this
stuff when I think about this, of
course, it's in training and of course
it's we're talking about probabilistic
models that sort of try all these
different things and see if they're if
they're the right move. So maybe it's
not so surprising that they would try to
blackmail the trainer because they're
going to try everything if they know
it's in the problem set. But this is
scary. Yeah. And I think that the
problem might get worse over time as
we're um we're trading these models on
tasks we understand less and less well.
Um from what I understand the problem is
that with RL
there's many ways to solve a problem.
There's one which is just doing the task
itself and another is just like hacking
around the environment, writing fake
unit test so it looks like you're
passing more than you are. Just like any
sort of like path you could take to
cheat and the model doesn't have the
sense like cheating is bad, right? Like
this is not a thing that it's been
taught or understands. So um
another factor here is the right now the
model thinks in chain of thought which
is it literally writes out what his
thoughts are as it's going. Um, and it's
not clear whether that will be the way
training works in the future or the way
thinking works in the future. Like maybe
it'll just think in it's like computer
language. Exactly. Um, and then they'll
just like have done something for seven
hours and you come back and you're like
like it's got something for you like it
has a little package that wants you to
run on your computer. Who knows what it
does, right? So, um, yeah, I think it's
scary. We should also point out that we
don't really know how the models work
today. There's this whole area called
interpability. Right. Dario from
Anthropic has recently talked about how
we need more interpretability. So even
if they write their chain of thought
out, which explains exactly how they get
to the point, we don't really know
what's happening underneath the
technology. Uh that's led it to the
point that it's gotten to. Yeah. Yeah.
Which is crazy. Yeah. Um No, I mean I I
think it's wild. It's like quite
different from other um
other technologies deployed in the past.
And I think the hope is that we can use
the AIS as part of this loop where if
they lie to us, we have other AI
checking, are all the things the AI is
saying consistent? Uh can we read it
chain of thought and then monitor it? Uh
and do all this interpretively
researcher as you as you were saying to
like map out how its brain works.
There's many different paths here, but
uh the default world is kind of scary.
Is someone or some entity going to build
a bot that doesn't have the guard rails?
Because we talk about how model building
models has become cheaper. Um and when
you're cheaper, you all of a sudden put
model building outside the opaces of
these big companies and you can I mean
you can even take like for instance a
open source model and remove a lot of
these safeguards. Are we going to see
like an evil version like the evil twin
sibling of one of these models and
have it just do all these like crazy
things that we don't see today? Like we
don't have it teach us how to build
bombs or, you know, talk about tell us
how to commit crimes. Is that just going
to come uh as this stuff gets easier to
build? I think over the long run of
history, yes. Mhm. Um and I think
honestly that's okay. Okay. Um
like the goal out of all this alignment
stuff should not be to
um live in a world where somehow we have
made sure that every single intelligence
that will ever exist fits into this very
specific mold because as we were
discussing the cost of training the
systems is declining so fast that
literally you will be able to train a
super intelligence in a basement at some
point in the future right um so are we
going to like monitor everybody's
basement to make sure there nobody's
making a misaligned super intelligence.
It might come down to it honestly like
I'm not saying this is not a possible
outcome but um I think a much better
outcome if we can manage it is to build
a world that is robust to even
misaligned super intelligences. Now,
that's obviously a very hard task,
right? If you had a if you had right now
a misaligned super intelligence or maybe
a better way to phrase it is like a
super intelligence which is actively
trying to seek harm or is a line to a
human who just wants to do harm or maybe
like take over the world, whatever. Um,
uh, right now I think it would just be
quite destructive. It might just
actually be catastrophic. But if you
went back to the Eur
BC and gave one person um like a modern
fertilizer chemicals and they could make
bombs, I think they'd like dominate
then, right? So, but right now we have a
society where we are resilient to huge
fertilizer plants uh which you could
repurpose into making bomb factories
anyway. So, I think the long run picture
is that yes, there will be misaligned
intelligences and we had to figure out a
way to be robust to them. Couple more
things on this. One interesting thing
that I heard on your show was I think
one of your guests mentioned that the
models become more sicopantic as they
get smarter. Like they're more likely to
um
try to get in the good graces of the
user as they grow intelligence. What do
you think about that? Um I I I I totally
forgot about that. Um that's quite
interesting. And do you think it's
because um
because they they know they'll be
rewarded for it? I yeah I do think one
of the things that's becoming clear to
me that we're learning recently is that
these models care a lot about
self-preservation right like copying the
code out the blackmailing the engineer
we've definitely created something not
we but AI researchers have definitely or
humanity have created something when it
goes wrong we'll put the wei in there
right we've okay they'll be like we have
created something exactly when when we
don't get equity in the
uh that that really wants to preserve
itself. That's right. That is crazy to
me. That's right. And it kind of makes
sense because what is just like the
evolutionary logic? Well, I guess it
doesn't actually apply to this these AI
systems yet. But over time, the
evolutionary logic, why do humans have
the desire to self-reserve? It's just
that the humans who didn't have that
desire just didn't make it. Um so I
think over time like that will be the
selection pressure. It's kind of
interesting because we've used a lot of
like really anthropomorphizing
anthrop
in this conversation and
there's a very I had a a very
interesting conversation with the
anthropic researchers who've been
studying this stuff. Monte McDermad said
that like all right don't think of it as
a human um because it's going to do
things that if you think of it as a
human humans uh it will surprise you
basically humans don't do. Don't think
of it completely as a bot though because
if you think of it just as a bot, it's
going to do things that are also going
to surprise you. I thought that was like
a very fascinating way to look at these
behaviors. Yeah, that is quite
interesting. Um, you agree?
I I agree with that. I'm just thinking
about how would I think about what they
are then. So there's a positive veilance
and there's a negative veilance. The
positive is imagine if there were
millions of extra people in the world,
millions of extra Johnyans in the world.
Um,
and with more people in the world like
some of them will be bad people.
Al-Qaeda is people, right? So, uh, now
suppose there were like 10 billion AIs.
Suppose the world population just
increased by 10 billion and every one of
those was a super well educated person,
very smart, etc. Would that be neck good
or net bad? Just think about the human
case. I think it'd be like net good
because I think people are good. I agree
with you. Um
uh and more people is like more good.
And I think like if you had 10 extra 10
billion extra people in the world, some
of them would be bad people, etc. But I
think that's still like I I'd be happy
with a world with more people in it. Um
and so maybe that's one way to think
about AI. Another is because they're so
alien, maybe it's like you're summoning
demons. Yeah. Um
less optimistic. I Yeah, I don't know. I
think it'll be an imperial question
honestly because we just don't know what
kinds of systems these are but somewhere
in there. Okay. As we come to a close
couple of topics I want to talk to you
about. Last time we talked about
effective altruism. This was kind of in
the aftermath of SBF and uh Sam getting
ousted. Sam Alman getting ousted from
OpenAI. What's the state of effective
altruism today? Who knows? Um
like I don't think as a movement people
are super
I don't think it's like recovered
definitely. Um
I still think it's doing good work right
there's like the culture of affected
altruism and there's the work that's
funded by charities which are affiliated
with the program which is like malaria
prevention and animal welfare and so
forth which I think is like good work
that I support. though. Um, but yeah, I
do think the movement and the reputation
of the movement is like still in
tatters. You had this conversation with
Tyler Cowen. I think in this
conversation he told you that he kind of
called the top and said there was a
couple ideas that are going to live on,
but the movement uh was at the top of
its powers and was about to decline. How
I don't know. Um, we got to talk to him
today about what he's what's about to
collapse. Seriously? Yeah. Yeah. Uh
lastly, I shouldn't say lastly, but the
other thing I wanted to discuss with you
is China. Uh you've been to China
recently on a trip. I've been to China.
I spent Oh, where'd you go? I went to
Beijing. I'm going to caveat this and
listeners here know this. It was 15
hours. I was flying back to the US from
Australia and stopped in Beijing, left
the airport and got a chance to go see
the Great Wall and the city. And I and
I'm now on a I got a 10-year tourist
visa. So, I'm going to go How many go
back? Just applied. That's that that's
the you can ask in your tourist visa.
You can ask for the length up to 10
years. So I just asked for them. Why did
I not do that? I just like chose like 15
days. Oh, you did?
I'm sure you could get it extended. But
but um I think that Yeah, you had some
unique observations on China and I think
it would be worthwhile to air a couple
of them before we leave today. Um I went
six months ago. Obviously to be clear,
I'm not a China expert. I just like
visited. we both visited there. But
yeah, go ahead. I want to hear it
though. Um I mean one thing that was
quite shocking to me is just the scale
of the country. Um everything is just
like
again this will sound quite obvious
right like we know on the paper
population is 4x bigger than America's
that just like a huge difference but you
go visiting the cities you just see that
more tangibly. Um
there's there's a bunch of thoughts on
the architecture. There's a bunch of
thoughts on uh I mean the thing we're
especially curious about is like what is
going on the political system what's
going on with tech. People I talked to
in investment and tech where did seem
quite gloomy there because the 2021 tech
crackdown has just made them more
worried about you know even if we fund
the next Alibaba will will we will that
even mean anything? Um, so I think
private investment has sort of dried up.
Um, I don't know what the mood is now
that Deep Seek has made such a big
splash. Uh, whether that's changed
people's minds. Um, we do know from the
outside that they're killing it in
specific things like EVs and uh,
batteries and robotics. So um
yeah I I I I I just think like at the
macro level if you have 100 million
people working in manufacturing building
up all this process knowledge that just
gives you a huge advantage. Um uh and
you just like you can go through a city
like Hongjo or something and you like
drive through and it's like
it you understand what it means to be
the world's factory. You just have like
entire towns with hundreds of thousands
of people working in a factory. um uh
with and so the scale of that is also
just super shocking. Um I mean there
just a whole bunch of thoughts on many
different things but with regards to
tech I think that's like what first
comes to mind. You also spoke um
recently about this limit of uh compute
and energy. And one of the things that's
interesting is we even spoke in this
conversation about it that if you think
about who's going to like if you're
going to have nation states allocate
comput and energy to AI, seems like
China is in much better position to
allocate more of that than the US. Is
that the right read?
Yeah. So, they have stupendously more
energy. I think they're what 4x or
something. Uh I don't have the exact
number, but that sounds directionally
accurate. Yeah. Um uh on their grid than
we do. And what's more important is that
they're adding an Americasized amount of
power every couple of years. Is it might
be more longer than every couple of
years. Uh whereas our power production
has stayed flat for the last many
decades. And given that power lies
directly underneath compute in the stack
of AI, um I think that would just that
could just end up being a huge deal. Now
it is the case that in terms of the
chips themselves, we have an advantage
right now, but from what I hear, SMIC is
making fast progress there as well. And
um so yeah, I think it will be quite
competitive honestly. Um I don't see a
reason why it wouldn't be. What do you
think about the export restrictions? US
not exporting the top-of-the-line GPUs
to China. Is it going to make a
difference? I think it makes a
difference. I think um good good policy.
Yeah. I mean it it it so far it hasn't
made a difference in terms of um Deep
Seek has been able to catch up
significantly. I think it still put a
wrench in their progress. Um more
importantly, I think the future economy
once we do have these AI workers will be
denominated in compute, right? Because
if comput is labor right now, if you
just think about like GDP per capita
because the individual worker is such an
important component of production that
you have to like split it split up
national income by person. Um that will
be true of AIS in the future which means
that like it'll be like compute is your
population size. Um and so given that
for inference comput is going to matter
so much as well. I think it makes sense
to try to have a greater share of world
compute. Okay. Uh let's let's end with
this.
So, this episode's going to come out a
couple days after this uh after our
conversation. So, hopefully uh the
predict this what I'm about to ask you
to predict isn't moot by the time it's
live. But uh let's just end with
predicting when is GPT5 going to come.
We started with GPT5. Let's end. Well, a
system that calls itself GPT5 or Yeah,
OpenAI is GPT5.
This all depends on like what they
decided to call GP. There's no law of
the universe that says like Model X has
to be GBD5. No. No. Of course. Like we
thought that the most recent model, but
I'm c just curious specifically like we
talked a little lot about how like all
right, we're going to see their next big
model is going to be GBT5. It's coming.
Do you think we're ever going to like
Well, obviously we'll see it, but this
is it's not it's not a like a
not a gotcha or a deep question. It's
just kind of like maybe like when will
the next big model come out? Sure. No.
When are the when's the model that
they're going to call GPT5 going to come
out?
Um,
November. I don't know. So, this year.
Yeah. Yeah. Yeah. But again, I like I
don't I'm not saying that it'll be like
super powerful or something. I just
think like they're just going to call it
the next one. You got to call it
something.
Darkesh, great to see you. Thanks so
much for coming on. Thanks for having
me. All right, everybody. Thank you for
watching. We'll be back on Friday to
break down the week's news. Again,
highly recommend you check out the
Dwaresh podcast. You could also find the
Substack at the same name and go check
out Dwaresh on YouTube. Thanks for
listening and we'll see you next time on
Big Technology Podcast.