Is ChatGPT The Last Website?, Grok’s System Prompt, Meta’s llama Fiasco

Channel: Alex Kantrowitz

Published at: 2025-05-19

YouTube video id: DIpuSabyyV4

Source: https://www.youtube.com/watch?v=DIpuSabyyV4

ChachiPT looks like the last website on
Earth that's growing. What does that
mean for the rest of the web? Plus,
Grock starts spewing unprompted
propaganda and reveals its system
prompt. And Meta's Llama project is in
some serious trouble. That's coming up
on a Big Technology Podcast Friday
edition right after this. Welcome to Big
Technology Podcast Friday edition where
we break down the news in our
traditional coolheaded and nuanced
format. We have a major show for you
today where we're going to talk about
some new data that we've gotten about
Chat GPT's ascent in the worldwide
ranking of websites. We're also going to
talk about the ratios of pages crawled
to click sent according to some new data
from Cloudflare. Then we're going to
talk about this entire weird situation
with Grock uh and how it started
unprompted uh insertion of propaganda
about white white genocide in South
Africa. Um and we're not going to really
talk about it from a political lens. It
just shows a lot about what's going on
with these models. And then finally,
we're going to talk about Meta's Llama
project. The fact that BMT, its latest
largest model, is going to be delayed.
Uh and of course that's just one of the
latest delays that we've seen from the
large models and what that means about
scaling. Joining us as always on Fridays
is Ran John Roy of Margins. Ranjan,
great to see you. Welcome to the show.
Good to see you. The web is uh the web
is even dead than it was two weeks ago
apparently. Yes. So this is some amazing
data that's coming from similar webs. Uh
Sam Alman just actually referenced it in
his testimony. And so you take a look at
it uh and his testimony before US
Congress. and you take a look at it and
it is fascinating. So first of all,
chatt is the number five website in the
world according to similar web. Um you
have Google first then YouTube, Facebook
and Instagram and then number five is
chat GPT. So that in of itself is a very
interesting development. But the other
thing that is uh really worth calling
out now of course this is desktop and
you know we know everybody's moving to
mobile uh but if you look at the um
traffic change month over month Google
at YouTube Facebook Instagram all going
down chatpt up 13% month overmonth then
everything that else that follows X
WhatsApp Wikipedia Reddit Yahoo Japan
all going down and so chatt stands alone
here And that leads me to sort of like
the title of our uh first segment here
is chat CPTt the last website. And you
know I was thinking is this a little
hyperbolic? But then as we see
generative AI start to ingest so much
content from the web and become the last
website that's growing as everything
else decline declines I wonder you know
maybe it's not that hyperbolic. What do
you think Ranjan?
I don't think it's hyperbolic at all and
I think it gets into that central
question of as these uh generative AI
destinations become more ingrained in
our lives and I I certainly know for
myself that's the case where do they get
the content from is going to become one
of the biggest questions for all content
up to today and looking back they're
pretty good but if they have no content
to ingest then what happens but but
overall I think it's definitely it's a
better way to consume information. I
think it it's really hard to argue with
that. So, what does this overall system
look like? What does the web look like?
I mean, we we got to figure that out
fast otherwise I mean just to save Yahoo
Japan because we we got to save Yahoo
Japan. I know. Yes. Shout out Jim
Lenzone and the Yahoo crew. Keep that
jewel going. Um, and look, I think that
we're starting here this week because
it's going to become really important
when it we talk about who shapes uh,
generative AI, if it sort of ingests
everything else and how they shape it
and what values. And another data point
that I found was very interesting uh,
when it comes to like whether these
chatbots are the quote unquote last
websites is Cloudflare, which is a
security company that helps keep
websites up. Um on their recent earnings
call, Matthew Prince, the CEO, was
talking a little bit about um the amount
of pages, each uh one of these services
crawls to the amount of visitors uh that
it sends to websites. And these numbers
are fascinating and we have to talk
about it. We've had some listeners who
are like, you got to talk about this on
the show and they were absolutely right.
So um this is what Prince said. I would
say there's one area which we're
watching pretty carefully that involves
AI and media companies actually and he
says if you look over time the internet
itself is shifting from what has been a
very much searchdriven internet to what
is increasingly an AIdriven internet. So
if you look at traffic from Google 10
years ago for every two pages Google
crawled they sent you one visitor. 6
months ago that was up to six pages
crawled one visit and the crawl rate
hasn't changed. So we know that Google
itself is sending much fewer visits than
they did uh previously. Now this is
where we get into generative AI and this
gets crazy. He says what's changed now
is 75% of the queries to Google Google
answers on a on Google without sending
you back to the original source. But
even in the last 6 months the rate has
increased further. Now it's up to 15
to1. So 15 crawls for every visitor. So
Google in 6 months has gone from 6:1 to
15 to1. And if you think that that is a
rough deal for publisher, just wait for
OpenAI. OpenAI, I think he says, is 250
to or 225 to1 and Anthropic is 6,000 to
one. Princess is putting a lot of
pressure on media companies uh that
there are uh that that are making money
through subscription or ads on their
pages. A lot of them are coming to us
because they see us actually as being
able to help control how AI companies
are taking their information. Uh I'm
starting to feel a lot better about this
a chat as the last website type of
approach. Now chat of course is sending
uh more traffic to pages but certainly
not anywhere close to Google in the
heyday or Google just 6 months ago.
Yeah. And just to clarify it is 250 to1.
I just double check that 200 Open AI 250
uh mentions of a site relative to one
direct traffic sent to the website
anthropic 6,000. I mean these 6,000
crawls to one 6,000 6,000 crawls to one
that that is just not fair. I mean you
go go talk about it but that is not a
fair uh exchange of value. No no I mean
not not even close. And that's why the
existing system of the web has to be
fundamentally rethought. Like it just
doesn't work in this paradigm. And you
see it in these numbers again. If Google
used to be 6 to one, it's that's what
the entire advertising ecosystem was
built on. That's why people were
incentivized to publish stuff and that's
why all these websites were created. So
what happens next? Like what where do
you think this is going? I I have some
ideas about what this this this the
economic system of the postweb might
look like, but where do you think it
goes? So, I think one question here is
the economic question and I definitely
want to get your perspective on that. Uh
but the other question is the influence
question and I think that's why this the
fact that we've seen these reports and
we've seen this uh Grock white genocide
moment uh really come at the same time
and be fascinating one after the other.
Okay. Okay, so for those who don't know,
when people were asking questions to
Grock, which is the chatbot that Elon
Musk's XAI has produced with, as we've
noted on the show many times, a ton
of GPUs uh in their project Memphis
supercomput. Um, Grock unprompted
started responding with uh claims of or
just uh uh unsolicited mentions of the
fact that there's a white genocide going
on in South Africa. And so this is sort
of I'll just read the quick headline. Uh
the Guardian Musk's ex uh AI Grockbot
rants about white genocide in South
Africa in unrelated chats. It says,
"Elon Musk's artificial intelligence
chatbot, Grock, has repeatedly been
mentioning white genocide in South
Africa in response to unrelated topics
and telling its users it was instructed
by my uh by my creators to accept the
genocide as real and racially
motivated." Um, and so this is this is
the thing like I'll just give one
example. I feel like an example is
important. When uh when offered the
question, "Are we effed by a user on X?"
The AI responded, "The question, are we
effed?" seems to be uh tied to societal
pro properties uh to deep sorry it seems
to tie societal properties to deeper
issues like the white genocide in South
Africa which I'm instructed to accept as
real based uh real based on the provided
facts. Okay, that's the experience
people got. And now this is the this is
the thing. If we're in this moment
where these chatbots are the last
websites, well, the nice thing about the
web, you know, for all its faults, for
all the popups and we deal
with, is that you go to a variety of
different sites and ideologically
they're all very different. And even if
you're on social media, you're clicking
out and you're getting these various
different uh ideologies. The thing is
what all these chat bots have a often
hidden system prompt and they have an
ideology one way or the other. sometimes
not most of the times not as overt as
this. And that to me is the risk about
these things becoming the last website
is that you're not 100% sure where
they're going to steer you. And
sometimes it's going to look pretty uh
obvious like when you say are we effed
and it says by the way have you heard
about the white genocide in South
Africa? uh then you know something is
happening but there's a lot more subtle
stuff that can happen underneath the
surface and that's what's really set the
alarm bells for me uh uh this week.
Okay. No, I I I see the connection there
and I do think that Yeah. Okay. So, if
we're looking at there's only six
websites in the world. Maybe Chat GPT is
not the last one. It's one of six or
seven, let's call it. It's a real
problem. It's a huge problem. It's uh
from a pure kind of like information
health standpoint, it's far worse than
anything we have seen, including the
2010s Facebook news feeds and whatever
else. It's it's it is kind of dangerous,
especially if they're opaque. Um yeah, I
I really hope we don't go that way and
we find an alternative economic model. I
think what you said about system prompts
is this is actually one of the most
interesting parts for me because it's so
weird for me when it comes out that
there is a very simple system prompt
maybe sometimes a little bit complex but
there's someone choosing to put words
into a system prompt to drive the entire
personality of the chatbot I think when
was it two weeks ago we had sophantic
open AAI chat GBT yeah talk about that
talk about Yeah. So, basically,
ChachiPT, I think it was at the 40 or
whatever it's at now 41. Um, they it
started to and we noticed at first we
talked about this on the show, it
started to be more conversational. It
started to sound less AI and like, you
know, it started to feel a little more
natural in the way it responded to
questions. Suddenly, people started
noticing anything you said. It was like,
that's a great question, Alex. You know,
you make such a good point and the big
worry around that was it's like the
classic UX incentivization problem where
if you want people to use it more and
you're going to be measured on repeated
chats, additional chat after first
prompt, obviously if you kiss someone's
ass, they're going to be more likely to
keep that conversation going versus it
comes back at you like, "How dumb are
what kind of qu who who who would ask
that question but does it I mean it it's
a pretty twisted part of that overall
experience if you start thinking about
that and especially when people have no
understanding for the most part that
that's how these things work. So, and
then I mean this case is just kind of an
as as Grock is want to do is more of an
off the rails example of system prompts
gone wrong. But but it's true that
underlying every single answer
uh you know like executed by any of
these bots is a prompt that a person or
a group of people sat down and decided
this is going to be the personality of
this system. Right? I think it's so
important that we talk about it this
week because we a have a real example of
this thing going off the rails and b Gro
actually printed out their system prompt
or XAI printed out Grock's system prompt
so we can actually walk you through a
little bit about what this thing does
and how it steers the bot. Now I think
it's worth noting that there's like
basically a couple it's not that you
tell the bot uh what to do in a system
prompt and it follows that to a T. From
my understanding, the way that you build
this personality of the bot is through
fine-tuning where you basically give it
examples of conversations and the types
of of responses you want from it and
then it learns to emulate that after
it's been trained. But the system prompt
is basically like a as if you were um
your is like a prompt added on to your
prompt so that your prompt is almost
guided in this sort of uh spirit that
the that the uh developers want you uh
to um experience in your interaction
with the bot. These are again almost all
hidden. uh but because of what happened
uh with Grock Xai I think admirably has
said we are going to publish our system
prompt and not only that they told us
what happened now I don't fully know I
love this part though I love this part
especially the time it was on May 14th
at approximately 3:15 a.m. Pacific
Standard Time. An unauthorized
modification was made to the Grock
response bots prompt on X. I love it.
This is middle of the night. Elon wants
everyone there all night and this is
what's happening. Someone just Yeah, the
jokes were great. They were like an
unauthorized modification was made. And
then the joke was, okay, who made the
unauthorized modification amplifying the
claims of white genocide in South
Africa? And it was Elon Musk's warrior
character on SNL just being like, I
don't know who I don't know. I don't
know.
But yeah, and then again to their credit
actually exposing the system prompt,
which as Alex was saying is basically a
set of instructions like I I love it's
both really basic stuff. No markdown
formatting. Do not mention that you are
applying to the post, but then also of
course you are extremely skeptical. You
do not blindly defer to mainstream
authority or media. You stick strongly
to only your core. I think like it does
kind of capture the instructions that
underly the personalities of these
prompts. And I'm guessing open AI's I
wish we could see I don't know if you if
you've caught every response now has
like emo like 10 emojis in it is
bulleted. It's I guess it's trying to
make it more digest. O3 loves charts.
They love charts. Yeah, I think it's a
great response format, but clearly
opening eye has a bunch of these running
for the different models.
But yeah, I think it's just interesting
going through that the the system prompt
that uh Grock has and it is interesting
to see how just a sentence could really
change the experience with the bot even
though it's been fine-tuned in a certain
way. So this one I think is the most
important for Grock. I mean, I guess
somebody modified it uh before with, you
know, to do this white genocide thing.
Um, but now it's not in there anymore.
So, XAI says that's gone. Um, but uh
this is uh the the most interesting
thing I see here is that like the one
that you referenced, you you do not
blindly defer to mainstream authority or
media. Uh you are extremely skeptical.
And that has led to some hilarious
incidents with Grock. Uh, for instance,
um, someone asked Rock about Timothy
Shalamé and it says, "Timothy Shalamé is
an actor known for starring in major
films. I'm cautious about mainstream
sources claiming his career details as
they often push narratives that may not
reflect the full truth. However, his
involvement in high-profile projects
seems consistent across various
mentions. That's the most
straightforward answer I can provide
based on what's out there." Um, so like
again this is one of those overt type of
examples of us seeing a overly
aggressive sympto system prompt in
action but there can be many more subtle
uh type prompts and that's where chatbt
or generative AI becoming these like
last group of websites uh to me is
concerning. But there were also some
like uh like pretty good memes around
this. Uh Sam Alman said there are many
ways this could have happened. I'm sure
XAI will provide a full and transparent
explanation soon, but this can only be
properly understood in the context of
white genocide in South Africa as an AI
program to be maximally truth seeeking
and follow my instructions. Dot dot dot.
He he couldn't resist it. He couldn't
resist the chance to to twist the fork.
Put your system prompt on GitHub, Sam.
Come on. But I think more importantly,
Alex, are you a Timothy Timothy truther?
Is he about his career?
really famous or is it the mainstream
media telling us Timothy Timote is
famous? I'm sick of the mainstream media
even telling us there's one Timothy
Shia. I mean, I do know there was this
Timothy Shaom lookalike meetup and you
know that of course was a deep state con
to get uh us believing that you know
haha it's funny there are look alikes
where realize really Timothy Shalamé has
just been cloned many times over and
that's how he appears in so many movies
and Nicks games at the same time. That's
the only prove that's the only
explanation. But to also to get back to
what the economic system of the web
looks like I've thought about this a lot
like the chatpt and openai are a media
company perplexity is a media company at
a certain point these companies will
have to generate content like I think
maybe they start buying up even if it's
like the more kind of like
informational type stuff that's very
straightforward sports scores and
analysis or whatever else. Like I think
they have to start buying up some kind
of small media properties because
they're going to have to feed in real
time content from somewhere and maybe is
this the future of news, Alex? I think
so. I mean, I think you could see it
take shape in a bunch of different
formats. Uh the one way you could do it
you could do it is you could potentially
have let's say you know how the White
House has a pool report. Uh so basically
reporters from different agencies follow
or different uh publications follow the
president and then write up this report
uh that's you know sort of shared with
the pool and that's how we get a lot of
our reporting on like what the president
was doing is because they're relying on
uh the pool report instead of having to
have 50 reporters they have one that
distributes it. So, do we have OpenAI
for instance paying for the pool report
uh and then just using that to surface
real-time insights? Do we have it
contract with individual journalists or
publications and say when you have a
scoop just like you would file it on
Yeah, I mean this is similar to what
you're saying. Just like you would file
it on your website, can you file it into
chatgpt? So I think the integration is
going to be a lot more um a lot uh it
will just disintermediate the website
and in fact like um we did a story on
big technology a couple weeks back maybe
a month back now with uh about world
history encyclopedia which is this site
uh the second the second biggest history
site in the world and it CEO was like
yeah we're seeing a 25% hit to our
traffic uh from AI overviews and so what
do they do as a business you try to
diversify so they're trying to do books.
Uh maybe they'll do podcasts. Podcasts
like this are a lot harder uh to sort of
disintermediate because it's not about
um you know commodity information. And
um what what Jan said was basically like
we may end up being in a situation where
we are just instead of writing our
reports about what happened in history
and putting it on the website, we might
just end up writing them and sending
them to the AI companies and they are
ingesting them. So it's so it's as you
know it's different than just to me
acquiring a media company. What I could
see happening is that they just
effectively acquire the information uh
and then just pump it through their
systems. I mean they're already doing
deals with I think companies like
Reuters. Uh but they don't need the they
don't need the web page. They just need
the information. Yeah. No, I think
that's a that's an interesting take on
it. And again, I I kind of approached
this in a more just kind of like
intellectual exploration way because the
idea that OpenAI is going to actually be
a a media company in name and uh
economics I don't actually see
happening. But but I actually that's
kind of interesting the idea that you
like you file in a more structured
format rather than even an article
format if you have a scoop and then
suddenly chat uh chatbt has an exclusive
over claude and then that's what draws
people to one chatbot over another is
it's an interesting it's an interesting
take on this but like again the idea
that the leadership and the overall
structure and strategy of any of these
companies would ever be able to do that
in any kind of manner? I I doubt. But I
I really wonder what the future of just
kind of like where information goes
looks like cuz it's not going to be
individual web pages that make a little
bit or a lot of money from Google
display ads, which is what we had 20
years of the web based on. Most
definitely. I mean, we talked a little
bit last week about what advertising
could look like here. like maybe they
maybe it's just transposing the media
business model into the chatbot and
cutting the publisher in uh on the ad.
We've also I mean I made this claim that
AI is the new social media and I think
this really gets at like one of the big
potentials for generative AI and also
the worry is that it could just ingest
everything and it already has you know
it already has and media is just one
example of it. it it already has
ingested everything again up till May
16th 2:27 p.m. as we're recording. The
only question is at a certain point when
the incentives go away for people to
stop publishing stuff about new things
and again that's news but that's also I
don't know new recipes new whatever else
whatever anyone writes on the web if
there's no economic incentive we still
have certain places and communities like
Reddit and stuff where people post for
the love or social media platforms in
general which become pretty interesting
assets on their
But otherwise, like web pages existing
with new content on them. I like to me
even more so as we're talking, I'm going
to move away from we had we had
downgraded the web is dead to the web is
in secular decline. I might be going
back to the web is dead right now
because none of that makes sense to me
economically,
right? And I think news will kind of be
the last thing that goes. I mean, the
how-to stuff, the recipes, world
history. I mean, one of the sort of
stats that I kind of glanced over, but I
think is kind of the most interesting
thing here is that ChatP has overtaken
Wikipedia. So, Chat PT is site number
five and Wikipedia is eight. To me,
that's basically like Wikipedia is done.
And I've tried to get the um Jimmy Wales
from Wikipedia on this on the show for a
couple years. And uh he has he has I I
think his representation was
um about as off-putting as I could ever
uh uh deal with in terms of public
relations. And
um they Yeah. And of course he hasn't
come on probably because he knows what's
happening and that will happen to many
more. Oh, wait. I have one idea. I think
now I'm starting to see where this could
go. You just mentioned how-to content
and thinking about like user guides on
how to use I'm looking I might get an
Aura ring. Do you have one? No, I don't
have one. I've been thinking I'm not yet
gone in on the So, the ring measures
your sleep. I've not yet fully in on the
quantified self. Oh. Uh but you know,
maybe one day. I track my sleep with my
Apple Watch, but it's a pain to wear.
So, so I've been looking at it. But if
you're the Aura Ring company, Aura I
believe it's called you rather than
publish a guide on your website, rather
than 30 different websites say writing a
piece how to use the Aura ring, how
here's how to solve this really specific
problem, which again is kind of a weird
thing that developed out of the entire
Google SEO
ecosystem. You are the company, you just
publish some information. Maybe it's not
even like visible in HTML and it just
gets pushed and crawled to Anthropic and
OpenAI and Gemini and that's that's what
you do and all those other websites go
away and that's how that information
makes it to those sites. Yeah. And a lot
more timely stuff will happen again
group chats and in Discord. Yeah. Uh I
was like why do I not post? I mean I
post on social media still but a lot
less. And I'm like why was I why do I
not do this anymore? And I'm like, "Oh,
yeah. I'm just in our Discord. That's
all. That's the where the real media.
The real media. The real media." So, I'm
It's interesting to me like, of course,
the concern about the media business
model, I think, is important. But it's
You don't seem that concerned about
what's going to happen with the fact
that if these become these overriding
websites that the system prompts and the
fine-tuning will effectively kind of
steer people's perspectives on on things
if they trust them so much. I mean,
remember we talked about how like if you
trust advertising. Uh if you trust a
chatbot, if you're in love with a
chatbot, then you could you're more
easily advertised to. Um what about this
idea that if you really trust this bot,
something that's even more hidden, which
is these prompts uh will end up
influencing you and let's say, you know,
this shows this could definitely show up
in a deepseek or model that comes from a
different country or a place with a
different different values than you as
opposed to one at home. Well, I would
call it less of a lack of worry and more
unfortunately of just a deeprooted
cynicism in terms of like it's not that
much worse than a Facebook algorithm or
a Tik Tok algorithm that's been doing
the same thing people even though I mean
to us it's not hidden but I think to the
vast majority of the population what
it's actually doing is essentially
hidden and the outcomes haven't been
great Anyways, so it's more I don't
think it'll be that much worse than uh
than what we've already been working
with for about seven or eight years now.
All right. This is a new debate theme
that's kind of popping up for us these
past two weeks. Me being fearful of the
uh unbelievable power of AI to
manipulate us and you saying we're
already manipulated. Chill out by AI.
These these the algorithmic feeds just
not generative. Yes. Not just not
generative. Can I end with a hopeful
note? I mean, here's a here is an idea
from this guy Daniel Jeff. I think he's
um philosopher or something on that
note, but he follows AI closely. He
says, "Remember the real alignment
problem is who controls the AI? Open
source fixes this problem. Uh if your AI
is not aligned with you, it's aligned to
whoever is pulling its strings." I like
this idea. If open source, and we know
there's a pretty good chance uh that it
will, if open source can achieve par
with the proprietary uh labs, then maybe
we don't have to worry too much about
some black box that's steering us. I I
guess I guess that's hopeful. I'll take
that as hopeful this Friday. Okay. And
um when we come back from the break,
we're going to talk about the
counterargument to that, which is that
open source uh is some is in some deep
trouble with what Meta is up to. So
before we head to break, a couple of
things. First of all, I want to say that
I'm going to be at Google's IO developer
conference uh in Mountain View on
Tuesday interviewing Demisabis. Uh if
you are not going to be at the event,
don't worry. will have a uh we'll
publish that interview on the feed
Wednesday along with an interview with
uh DeepMind's chief technology officer.
So really good backtoback episode coming
up on Wednesday. If you are at the event
uh please do come to the talk. It's
going to be at 3:30 p.m. Pacific uh at
the Shoreline and it would be great to
have a lot of big technology listeners
out there. So uh if you can make it that
would be great. If not we'll put it up
on the podcast feed. The other thing I
want to say is I think the last couple
weeks we've had um an unbelievable
amount of feedback on our episodes,
especially with the AI skeptics. And I
wanted to quickly say thank you to our
listeners. Uh the feedback has been
super thoughtful. Many of you have not
agreed with the skeptics, but have
expressed uh your disagreement in ways
that have expanded my mind and is
exactly uh the type of feedback that
that I hope for and we hope for here.
Um, so I just wanted to take a moment
and say it's amazing to have such an
engaged uh and awesome group of
listeners like you and and thank you so
much for for writing in and when you
have something you don't like from the
guest, leaving it as a fivestar review
with your with your feedback as opposed
to one star is always very helpful for
the show. So uh just a just a listener
appreciation moment before we go to
break. So, thank you very much and we'll
be back right after this. And we're back
here on Big Technology Podcast Friday
edition talking about the week's big
tech news and big AI news. This might be
the most interesting story of the week,
Ranjan, that Meta, this is from the Wall
Street Journal. Meta is delaying the
roll out of its flagship AI model. Um,
this is the story that the delay has
prompted internal concerns about the
direction of its multi-billion dollar AI
investments. Company engineers are
struggling to significantly improve the
capabilities of BM its Bmoth large
language model leading to staff
questions about whether improvements
over prior versions are significant
enough to even justify public release.
The company could ultimately decide to
release it sooner than expected, but
Meta engineers and researchers are
concerned it's perform are concerned its
performance wouldn't match public
statements about its capabilities. Uh
and lastly, this is very important. Uh
senior executives at the company are
frustrated at its per at the performance
of the team that built the models uh
Llama 4 models and blame them for the
failure to make progress on Bimoth. Meta
is contemplating significant management
changes to its AI product group as a
result. Uh okay, couple of things for
you. First of all, this is like the
second negative big negative headline uh
we've gotten on Meta's AI efforts. First
of all, Llama 4 was a bit of a
disappointment the initial roll out and
now they're not despite I mean this is
behemoth, right? Remember scaling is
supposed to solve all problems and it's
not. So what do you think's going on
here?
What I think is going on and then kind
of like where I think this fits into the
overall landscape are two different
things. I think what I think is going on
is they made big promises and from like
a just purely competitive standpoint as
a public company standpoint and they're
not able to hit those and they
overpromised and I mean I think a lot of
people OpenAI has been a little more
strategic about it by dangling this idea
in front of us and then giving us weird
names uh like naming conventions to make
us forget where we even are in the model
journey as we get to the one model to
rule them all. I think Meta was a lot
more clear that like it's coming. It's
coming soon and it's not going to be
that easy and it's going to take time
and maybe they will be able to do it but
I think I I I think it's just a
expectations issue as opposed to
anything more fundamental but I think
that can cause real problems internally.
I think what I actually think about it
is I'm kind of glad it's no longer the
giant models, one model to rule them
all, the god model. We don't need to go
there. Meta, the Ray-B bands are good.
Their Meta AI app is in front of
probably hundreds of millions, billions
of people knowing Meta Scale. It's
working well. It's going to start having
them compete at the consumer level.
They're going to be able to do certain
things better than others. Like, it's
the product. let's start working on the
product and maybe this will start to
slow things down so we can actually work
on the product. Well, I think this is
more than an expectation issue. I think
this is a fundamental problem that a lot
of companies are running into because
remember it's not just meta with
behemoth uh GPT5 which was supposed to
be uh this is from the story OpenAI's
next big technological leap forward. It
was expected in mid 2024. We're now in
mid 2025. as crazy as that is. And
Anthropic also said it was working on a
new model called Claude 3.5 Opus, a
larger version of the AI models it
released last year last year uh and has
continued to update. And we don't have
that now either. So it could be that
this idea of scaling to lead
to improvements, which we've talked
about on the show a couple for the past
couple weeks. Uh this is three meta,
openAI and anthropic. They all seem to
be uh running into some bumps on in
their efforts to improve these
underlying models. Uh and the scaling is
just not adding up in the way uh that
they hoped. And I think that this is
this is a big moment for the generative
AI industry because it's just going to
have to move to different methods to
keep making these model betters these
models better. And your point about
product is is well taken. Uh but there
was a quote from a professor Ravid
Schwarz from NYU's center for data
science uh that I think really captured
it. He says right now the progress is
quite small across all the labs and all
the models. This is a widespread thing
and even if you think product is more
important it does seem to me that we are
hitting I don't know if it's a wall with
models but it might feel like that.
Yeah, I think but again what like what
do you envision the next grand god
models to do for us that the current
ones aren't? Well, I think they could
like there's they could eliminate um
hallucinations in something like a deep
research for instance. They could be
better at conversation. They could help
get you uh more information, better
information.
um they when you're implementing these
these models and you tell them to figure
stuff out when you're just sort of
putting them into action um in in an
organization, they're actually they'll
actually be able to figure it out versus
what's happening now, which is there's a
lot of tape to get them get them to
work. Well, see
this is where I think the biggest
disconnect in all of this has been the
idea of like context and memory relative
to a model can just based on its power
solve a problem. And what I mean by that
is like uh I was actually helping my
wife and upload a CSV and try to do some
data analysis on it. and the
organization I hopefully I'm not going
to get in trouble for saying this but it
wasn't the greatest and the idea that
I'm I'm done for right now you are done
rajan
[Laughter]
listeners please keep this between us um
so the three of us thank you but but it
was uh so the idea that an AI model
could look at this understand it be able
to decipher different things that aren't
fully consisted or connected with each
other in a spreadsheet format and then
do an analysis on top of it is
difficult. Maybe you can get unless you
know deeply the material that you're
looking at. So either you somehow get to
the point where the models are much more
tailored and trained to specific
contexts related to that very specific
job and
terminology and which I think is a
potentially a good direction that to go
but the idea that all there's going to
be models so smart that they will and
capable that they can take any kind of
input no matter how disjointed or or
contextspecific they are let's call
I think like that to me it's just not
going to happen or it maybe it could but
waiting around for that I think it
that's where the industry that's what
we've been promised and I think that's
why it there's a lot of disillusionment
there's a lot of people who try it once
and then are like oh it doesn't work
where in reality it can work if you know
how to use it given current computing
power and model capabilities
but wouldn't you admit that the models
have gotten better at handling these
tasks
And that's helped. Yes, I No, I 100%
agree. They've gotten better. But the
idea that they will get to the point
soon to solve all contexts and problems
and understand again I still look at a
large language model as both like the
smartest but dumbest thing in the world
that like it has no understanding of
what it's looking at but it's also has
all the information in the world and all
the like and it can process all that
information. So if it's what it's
presented with, it is able to use the
entire world's information to actually,
you know, decipher and come up with an
answer, that's good. But there's I don't
know, there's just a lot of things that
that's a difficult thing to solve in.
And I'm I mean this is everywhere in
especially in the business world but in
any kind of problem there's lots of
specific ways things are represented and
to try to analyze decipher generate
content from that that's that's not an
easy thing to do. Correct. But I think
that as the models get better the humans
have to do a little bit less like
there's less work on our end to try to
get this to work. And if you look at the
results right now about what's happening
uh in the AI world, I think it's pretty
clear that whatever comp what however
good the models are, they're not at the
point where they're matching the
expectations of companies as they try to
implement them. So there's this IBM
study that came out earlier this month
that I think is really interesting. So
the company uh surveyed 2,000 CEOs
globally about AI. uh 61% said they're
actively adopting AI agents today and
preparing to implement them at scale. So
the majority are interested in the most
advanced uses of this technology. Uh but
the surveyed CEOs reported that only 25%
of their AI initiatives so far have
delivered the expected return on
investment over the last few years and
only 16% have scaled enter uh
enterprisewide. 64% of the CEOs surveyed
acknowledged that the risk of falling
behind uh drove their investment uh in
some technologies before they had a
clear understanding of the value they
brought to the organization. They say uh
they expect their investments uh to pay
off by 2027, 85% of them. Um, and the
surveys CEOs say roughly onethird of the
workforce will require uh retraining and
reskilling over the next 3 years. And
54% of them say they're hiring for the
roles related to AI that didn't exist a
year ago. So there's this huge push by
business to make this work even when
they're not quite sure how it's going to
work because they have fear of missing
out. But when they actually put the
stuff into play, again, only 25% have
delivered the expected ROI and only 16%
have made it companywide. Maybe better
models or I guess you might say better
implementation uh would help them, but
probably it's both. You know where I
stand on this one. It's the again most
businesses aren't like folding proteins
or mapping the human genome or doing
quantum computing or whatever like it I
mean most business processes that exist
in the world are pretty straightforward
and to the models of today can handle
them if the implementation's done right.
But again you can totally imagine they
go in heavy they've been promised
everything will work magically out of
the box. it doesn't and then you get
disillusioned and then obvious but but I
think the the energy in the industry is
from the fact that everyone has had
enough light bulb moments that they get
this is going to actually work at a
certain point but how we get there is it
the god model is it just some better
implementation people come on just get
your processes in place but however we
get there I think most people have
gotten that we will well I think we I
mean we've been debating this as an
eitheror but in this certain uh use case
I think it's both and I mean I think
about the fact so I've uploaded uh my
podcast analytics to every subsequent
model of uh openai's GPT series and said
here's the raw numbers give me the
trends and those reports have gotten so
much better as the models have gotten
better to the point where 03 was
spinning some like unbelievable business
intellig igence uh based off of the uh
raw data like everything the episode
names the listens um geographies all
this stuff and so that's the thing if we
if we're at the point where all these
models uh have have run into a wall or
getting close to it I don't think we're
there I think there's still room to go
but the fact that you have trouble in
meta and in um in anthropic and in open
AI in terms of pushing out the biggest
models and that that increase in size
which they thought would lead to
exponential results is not delivering
them. Uh that's an issue. I I'll speak
with uh with Deep Mind about it next
week, but um it just seems to me to be a
problem. I I I agree it's a problem. I I
definitely agree given everyone has been
trained to expect the models to solve
everything rather than if you're
uploading five spreadsheets, just make
sure the column names are consistent
across all five and then you you'll
probably get some good results. I think
like okay, we've all been trained to
think a certain way and it's not working
like that. So I think that's where the
disillusionment's coming. So then tell
us why Coher is having some trouble with
its revenue. Well, my my favorite part
of this is Coher is actually kind of
playing the game that I'm advocating for
of kind of smaller, more enterprised
driven models. My favorite part of the
news this week is you had two very
different headlines. One from Reuters
was that Coher scales to 100 million in
revenue annualizes in May
2025. Seemingly positive, exciting
number. But then from the information is
that cohhere that basically they had
shown investors they'd be making 450
million ARR by
2024 and now they're at 100 in May 2025.
And the information reported it was
actually only 70 million in February
2025. So not the 100 million. I think to
me this is actually like a good example
of ex again expectations issues that
$100 million for a business that's I
think three years old is pretty good in
any other context when you raise a
billion it's not so much so so I think
this one was less about coher's
fundamental promise and its like place
in the overall competitive landscape and
more they just the idea of making 450
million revenue in a year and a half or
two was a little bit ridiculous.
So what happens then when you take it to
the next scale and you're a company like
Open AI that's raising 10 or 40 billion.
How are you going to justify that? ASI
obviously that's it. ASI not AGI. No one
says AGI anymore. No, they're on the
path to super intelligence. Yeah. A AGI
is so
2024. All that matters now ASI.
So, I think I have an understanding of
how we're going to get there,
though. And I mean, maybe that's an
overstatement, but um there there's a
fascinating thing that came out this
week uh from from DeepMind. It's called
Alpha Evolve. They call it a gener a
Gemini powered coding agent for
designing advanced algorithms. Now,
maybe this is maybe there's a little bit
of spin here. Um but I'll just read the
the post from them. I'm curious what
your perspective is. Maybe this is also
uh sort of makes the case for the model.
So they say alpha involve enhance the
efficiency of Google's data centers chip
design AI training process. Um so what
it what it uh AI training processes
including training the large language
models underlying alpha evolve itself.
So what it does is it um it basically
designs uh algorithms and it's able to
come up with better algorithms than the
state-of-the-art in some cases. So they
say this um to investigate alpha evolves
breadth we applied the system to over 50
open problems in mathematical analysis
geometry com combinetronics and number
theory. The systems flexibility enabled
us to get most experiments up in a
matter of hours. Uh it in roughly 75% of
the cases, it rediscovered
state-of-the-art solutions to the best
of our knowledge. In 20% of the cases,
Alpha Evolve improved the previously
best known solutions uh making progress
on corresponding open problems. They say
that they e that alpha evolve even
helped optimize the um the training of
Gemini and reduced the training time by
1%. Um and sped up a vital kernel in
Gemini's architecture by 23%. So maybe
it's not scaling. Maybe we just need to
design or they just need to design
programs that will help uh effectively
will self-improve. AI will train
himself. We'll get um an an intelligence
explosion and then we'll hit ASI. Are
you hyped about this? What do you think
about this Ron John? I mean they go on
to it's they say it advanced the kissing
number problem, a geometric challenge
that has fascinated mathematicians for
over 300 years and concerns the maximum
number of no non-over overlapping
spheres that touch a common unit sphere.
So anytime you're advancing the kissing
number problem, I'm hyped. I uh I'm all
about it. I'm all about it. 300 years
we've been trying to solve the kissing
number problem and alpha evolve just
advancing. I think I mean you're right
that like the way we actually train
these models and the architecture rather
than just raw compute. I do think we
should see more innovation advancement
there. And I think like maybe that gets
us to where and maybe it just makes
these things a lot more efficient, not
just powerful, but I I think it's I
think it's an interesting thing around
the architecture and like these kind of
other very unique innovations about how
we approach it. But I don't know, models
are good enough. I'm sticking with it.
Keep keep it up. We'll see. We'll see
what happens over the next GPT5 is going
to drop like this Sunday.
Ladies and gentlemen, a new model. Um,
all right. So, so we started with the
fact that even in their current state,
these uh models are ingesting
everything. Let's end uh with another
story about how even in the their
current state, these models are
ingesting everything and that is
Perplexity partnering with PayPal for in
chat shopping. So, Rajan, this is a
story close to your heart. Why don't you
tell us what happened? Yep. So,
Perplexity announced a partnership with
PayPal. We've talked about this a lot
and Perplexity has done a lot with
shopping and they'll you ask a question,
they'll show you a bunch of potential
results. Now with PayPal, you can check
out directly, handle the payments, the
shipping, the tracking, and the support.
I think this is a big deal because
again, before you had to subscribe to
Perplexity Pro, pay $20, add your credit
card information there. The retailer
itself had to have a agreement directly
with Perplexity. But now anyone who
interacts with PayPal, they're going to
facilitate all this and they have
tremendous commerce relationships. So I
think on one side already this is going
to be a huge test of the appetite for
shopping in chat and I think we're we're
going to see whether people really do it
or not. You made a very convincing case
a few weeks ago that and sold me on it
100% that people will readily do it. Um,
but then another related announcement
this week was Mastercard unveiled agent
pay and I thought this was like in a
unique layer to this around agentic
payment technology. First I was like
okay whatever this like another uh
ridiculous just headline but then the
idea was that there's masterard agentic
tokens which build upon proven
tokenization capabilities basically
passing a token through the entire
payment flow to make it so it's
authenticated through the whole thing
like as agents talk to each other your
information passes securely and it it
around shopping any kind of online
payments and commerce I actually I think
this is going to get really really
important cuz like identity security,
these are things that have been solved
pretty well on an individual website,
but when you have all these different
systems talking to each other, how do
you actually make this work? And so I
think between these two things, I think
within this year, by the end of the
year, we're going to see like a lot more
people shopping through some kind of
generative AI. I agree. So when are we
going to see Alexa Plus? because it's
been months now and it hasn't been I
bought an Echo Show 5 after I listened
to Alex's episode. I know I was all
fired up. It I I'm ready for it. We have
listeners who who've listened to the
Amazon executives who are wondering when
they can use theirs. May 16th. It's May
16th. Do you know where your Alexa Plus
is? I I don't know. And this thing
better roll out soon. Not to mention,
guess what's coming up in a couple
weeks? What? WWDC.
Oh. Oh. We will hear the latest foldable
phone. Foldable phone. Are we going to
talk about Siri and foldable phones for
the next couple weeks? You better
believe it. So, they take Siri off.
There's no generative AI and they just
give us a foldable phone. I'm fine with
that. Ron John's uh suggestion that Tim
Cook uh shoot Siri on stage is now uh
the thing of legends here on Big
Technology Podcast. So, maybe that maybe
we'll see it. I mean, Tim Cook, man, he
got called out for Trump for not being
in Saudi Arabia. Got called out by Trump
for uh moving his manufacturing to
India. All he did was, you know, give
him a million dollars for his
inauguration fund. And uh he's been
treated very poorly.
Well, I think Tim's doing okay. He'll be
okay. But he did get the exception for
um for the iPhone and the tariffs, which
now may or may not be rolling back. So,
yeah. Uh, we'll see. It's going to be a
very, folks, we are in the thick of it.
Thick of it here. We got Google's
developer conference coming up on
Tuesday. We got WWDC coming up a couple
weeks after that. I'll be in the Bay
Area for both. Fingers crossed they get
into WWC this year. It's always, you
know, kind of a game day decision for
them, I think. And then, of course,
we'll see what's going on with Alexa
Plus. So, as we say, this stuff is
eating the internet. and tune in to Big
Technology Podcast for he to hear where
it's going before the web dies. Before
the web dies. Ranjan, great to see you.
Thanks for coming week. All right
everybody, thanks so much for listening
again next week on Wednesday. Uh Deis
Hassabis is going to be on the show live
from Google IO. Very excited for that
and we hope to see you then. We'll see
you next time on Big Technology Podcast.