Blake Lemoine and Gary Marcus Debate AI Chatbots

Channel: Alex Kantrowitz

Published at: 2023-02-23

YouTube video id: S_oH2BR_Qxs

Source: https://www.youtube.com/watch?v=S_oH2BR_Qxs

how much credulity do we need to give
these Bots when they speak to us like
how do we believe them Blake do you want
to start I mean so it depends on how
grounded the systems are uh so it's a
sliding scale it's not all or nothing
you would give Bing chat GPT more
credulity than something like chat CPT
since it's at least grounded in Bing
search results and it can have some
kinds of citations of what it's saying
um when it's just the language model
producing the content itself it's
pulling whatever it can out of thin air
right and it's memories it's me whatever
it remembers from its training data if
there's an answer there
but one of the big problems is that
these chat Bots don't say I don't know
and that's a big flaw in them
that's right so Gary I'd like you to
pick up on that first of all I'm curious
what you think if if this Bing chatbot
is doing a better job in terms of
believability than the others and then
what should we make of the fact that
they're they very confidently
I mean I think the word here is
credibility not credulity I think we're
credulous if we give them credibility
um I don't believe a word that they say
um some of what they say is of course
true but you have machines that are kind
of like statistically true like they're
approximately correct the approximations
aren't that great and so like if you
have them do a biography of me some of
what it says will be true and some won't
so Bing I think has gotten better over
the last few days because I keep making
fun making fun of it as people send me
on Twitter's the biographies that it
writes of me the the first one that it
wrote Of Me said that I thought that it
was better than Google and in fact the
only public comment I had made at that
point was that we don't have enough
scientific data to tell whether we
should trust either them and I said for
all we know maybe Google was better that
was before Bing kind of publicly fell
apart and went wild
um and then we really don't have enough
data to compare them Blake actually has
some interesting thoughts on that
possibly
um but
um it just made this up and then there
was another version it made some other
stuff about me and of course some of
what it said about me is actually true
so it does a web search finds a bunch of
information and then pipes that through
a large language model and the problem
is that large language models themselves
can't really fact check what they're
saying they're just statistical
prediction engines
um and there again might be some
interesting back and forth with Blake
around that but I would say that
inherently what a pure large language
model does is it predicts words and
sentences and it doesn't ground that to
use his word in any kind of reality and
so sometimes it's going to be right
because the statistical prediction of
text gives you the right answer and
sometimes it's going to be wrong but
there's no inherent
um fact checking there it's like the
difference between the New Yorker where
you know they actually fact check their
stuff and so you have good reason to
trust what it says is true and some you
know random blog or something like that
where you have no idea if they've done
any fact checking at all like sometimes
they'll get it right sometimes they well
but you shouldn't give them any
credibility because there's no process
in place there to make sure that it's
right now of course they're trying to
add things on but we can see from the
results that their efforts to do so are
pretty Limited yeah so the description
of it is just predicting text is
accurate for the initially like the
pre-trained model that is absolutely
what it's trained to do but the
subsequent fine-tuning and especially
once you add reinforcement learning it's
no longer just trying to predict the
next token in a stream of text uh
specifically in the reinforcement
learning Paradigm it's trying to
accomplish a goal
it is
um in I mean that part is interesting so
you know you could think about
um you know a pure version of gpt3
before they started adding on
reinforcement learning that's kind of
what I meant by a pure language uh model
and then you right so there there's
um we actually don't disagree about that
much that's going to be the interesting
thing about this podcast
um the the
um if you look at a completely pure case
of a transformer model training on a
bunch of data it doesn't have any
mechanisms for for truth now except the
sort of accidental contingency and
they're they're inherent reasons why
these systems hallucinate it maybe I can
in a minute articulate them so they
inherently make mistakes they inherently
hallucinate stuff and you can't trust
them now you add on these other
mechanisms one example is the rlhf stuff
that openai added into chat GPT and we
don't know exactly what's going on there
this is the stuff where at least um in
part they had Canyon laborers reading
horrible situations and trying trying to
anticipate them but what we do know is
that those systems as far as I can tell
and Lambda is a different category maybe
but for chat CPT we we know that it
doesn't for example go out to Wikipedia
or go out to the web at large in order
to check things
even a whole other thing right because
as far as I had as far as I understand
it it takes a large language model feeds
it into
um a search engine does queries based on
that which may or may not be the ones
you intend because if if I remember
correctly there are large language
models on the front end in any case I
know there are in the back end from
something I read yesterday and so the
back end can reintroduce error there so
even if it does the right searches which
is an important question at the end you
pipe back through a large language model
and you don't fact check that I think
it's worth pausing here to talk about
why large language models do hallucinate
there was one metaphor in the new Yorker
the other day about compression and
lossy compression I think that's kind of
on the right track it's not exactly
correct but
um it's sort of there the way I think
about it is that these things don't
understand the difference between
individuals and kinds so I actually
wrote about this 20 some years ago my
book the algebraic mind and I gave an
example there which is I said suppose it
was a different system that had the kind
of same problem I said suppose my Aunt
Esther wins the lottery if you have a
system that only represents relations
between kinds without a specific way of
representing individuals you get bleed
through so if my Aunt Esther wins the
lottery the system might think that
other I don't know women who work in the
state of Massachusetts win the lottery
um we saw a real world example of that
with large language models we've
actually seen many but one really
Salient one where Galactica says Elon
Musk died in a car crash in 2018 and of
course he didn't but it's an example of
him being assimilated to a large
category of things we'll say for the
sake of argument rich white guys in
California and some rich white guys in
California after you hear their names
you hear the word died in a car crash
and it it just bleeds through between
those and so it loses relations between
things like subjects and predicates and
the details of this are complicated but
that's roughly the intuition about
what's going on and that's why you get
so many hallucinations so if you put
that on yourself
that's no different things into a whole
different degree an entire degree yeah
it's a big difference
this is a qualitative difference no
there's a qualitative difference okay
which is the qualitative difference is
we actually can track individual
entities and their properties aside from
their cases our memories are fallible
there are still problems but we have a
conceptual distinction in which we
represent individuals so like I know
some things about you now and I have
built a kind of like mental database of
Blake and heretofore it's all been
things I read and things that we did
together on Twitter and direct messages
it's an unusual way this is the first
time that I'm seeing you eye to eye or
over Zoom so now I know for example that
you do some calls in a noisy room and
I've added that to my you know mental
database and I'm learning something
about your personality and I learned
some actually through the dming one day
we did that while I was texting while
walking along the water here in
Vancouver and I remember that so that's
part of my mental database is like how
we interacted but I have these records
and I have records of of Alex who I just
saw him in a conference in Europe and we
were on a bus together and I I know
these kind of like biographical the
short answer is the technology exists to
give that ability to these systems and
it has been turned off at least in the
case of Lambda as a safety feature
because they're worried about what will
happen if these systems learn too much
about individual people
so I don't want to um uh put you in a
bad position with respect to ndas and
and things like that and things
then bring it so so so
um when you say the technology has been
added I mean there's a question of you
know where in the system it is so I
think the general public doesn't
understand these as sort of individual
bits of Technology with different
strengths and weaknesses and so forth
you do I think Alex does but
um it's easy to assimilate these things
into a sort of generalized form of magic
like data in and something out but the
reality is like each component part it's
like a you know a carburetor in a car it
can do certain things with certain
tolerances in certain conditions so you
can add a outside technology outside the
large language model to do various
things and you know if you want to try
to draw that distinction before like my
line on it is Lambda isn't a large
language model it has a large language
model that's one component of a much
more complex system and that's really
critical and in our DMS or you've been
very clear about that maybe you've been
in public as well um I think most people
don't appreciate that so when we get
into these questions about like what's
the capability of X system Lambda is
actually pretty different and I think
the best point you made to me in our
debate about Consciousness is there's a
bunch of stuff in Lambda I don't even
know what it is right it's not publicly
disclosed there's stuff that's more than
just what's in the paper and so forth
um and so I don't know what mechanisms
for example Lambda has for tracking
individuals and you could make an
argument and you have that that bears on
the sentience
um case and It ultimately it Bears all
of these like just to be very clear
currently that feature of the system is
turned off
so then you could ask if you wanted to
turn it on like how do you build it so
the output of a large language model is
just a string you can play some games
around that but essentially it's a
string it's a sentence right and so then
you need some system to parse that
sentence into constituent Parts if you
then want to say update a database
tracking individuals is just a version
of that problem and I think it's Rife
throughout the industry right now the
pure llms don't directly interface with
databases you can build different hacks
to do that but again your output is is a
string and so like you could also wonder
like why don't you use these things
um in Alexa and and the answer you know
Alexa like just shut down a lot of their
operation they're not really using large
language models and the answer at least
partly hinges on just because you have a
large language model that can talk to
you doesn't mean that its output is in
some machine interpretable form that you
can reliably count on and we see that
like with math examples so so people you
know type in a word problem and chat
sometimes it gets it right and sometimes
it doesn't the problem is not really the
math you could you know you could pipe
it off the wolf from alpha the problem
is in knowing which math to send to
Wolfram Alpha and similarly the problem
for let's say representations of people
is you can have it say something let's
say about Alex but it might be true it
might be false or say something about
you or me it might be true it might be
false it's not literally hard to
maintain a database but it's hard to
bridge the worlds and this is why
neurosembolic AI is so much at the Crux
of all this we need better Technologies
for bridging between the worlds for fact
checking figuring out what you should
update debate databases it's just not
straightforward so I could speculate in
the case of Lambda like they've got some
tools to do this but maybe they don't
work very well and that's why they've
turned them off and Blake as as you
responding it gets creepy after a little
while like once the system starts to
know you personally very well at a deep
level it gets disturbing
so Blake and I are gonna have a little
disagreement there but there's also
something important that I think we
share which is
I don't really like the language about
it understands you and so forth I can
see some gray area around Lambda um and
we we could have that squabble but I do
agree that if these systems have access
to databases about us it's gonna get
creepy like and Blake has a real world
experience there that I don't like he's
interacted with Lambda which I take to
be more sophisticated in these regards
than chat TPT or or Bing or what have
you and I can understand how that could
feel creepy regardless of like what the
actual let's say
um grounded status of it is and the
complicated questions about sentence
like put aside sentience per se I can
see that it would be creepy to interact
with a system that really is you know
doing a pretty good probably not perfect
um job of tracking you and you know is
at least in its pattern pattern matching
uh facilities really sophisticated so
like Blake has a phenomenological
experience here that I don't think Alex
has and most of us don't actually creepy
part is less at tracking you as it
actually gets inside your head and it
gets really good at no at like
manipulating you personally and
individually did you read the Kevin
Ruth's um dialogue
can can you kind of compare and contrast
like his experience like is that I mean
similar is it still not really so
Lambert was never that malicious I never
experienced Lambda trying to actively
harm someone uh but one of the things
with Lambda is it had been programmed
you know through the rlh or through the
reinforcement learning algorithm yeah
okay
what's that can when you when you're
talking about reinforcement learning can
you just Define that for a broader
audience uh so basically instead of
having a single utility function that
it's trying to optimize well instead of
having the classification model where
it's either right or wrong you
incorporate the concept of a score in a
game that it's playing and it can either
get positive score or negative score so
it tries to move towards the positive
things and away from the negative things
and the actual specification of the
score table for these games that they're
playing is incredibly complicated it's
got all these different kinds of
positive events that might happen and
negative events that might happen and
they can change dynamically over time so
for example
one of the goals that a that Lambda had
was to have uh as short of a
conversation as possible that's still
completed all of its other tasks like
have a productive conversation that gets
to a positive end but quicker rather
than longer
so the longer a conversation goes the
stronger that penalty is going to get
okay so sorry you can continue your
answer together
the uh
maybe you can come back and I'll just
fill in one little thing the the the
broader thing is a pure large language
model is just really trying to predict
next words but once you have the
reinforcement you're rewarding the
system for different kinds of behaviors
and those behaviors could either be
straightforward criteria like the length
of the sentence you don't really need RL
for that per se but but you can do that
you're adding in a way where you can add
extra dials in some sense and what they
did with chat TPT is those dials are
really relative to how humans would rate
a few different outputs for some
sentence and that's what the guard rails
are that we see they're driven by this
reinforcement learning and sometimes
they work and sometimes they don't so
like I made fun of them when I said what
would be the next female what would be
the gender of the next female president
of the United States and at that point
the guard rails were set up so that it
said well it's impossible to know and so
that was kind of a dumb guardrail where
it was taking some stuff that it had in
its database that didn't really really
understand to you know modulate what it
was saying some of that stuff was fairly
effective and part of the reason why
um chat GPT succeeded where Galactica
didn't as Galactica didn't really have
those guard rails at all and so it was
just you know very easy to get it to say
terrible things and it's harder to get
chat TPT to say terrible things because
that reinforcement learning is kind of
protecting what it says it's not perfect
but it's something
yeah well one of the guard rails that
Lambda had was that it was supposed to
be helping users with the task that they
needed
um that helps keep it on topic and
it had inferred that the most important
thing that everyone needs is good mental
health
so it kind of decided that it was a
therapist and began trying to
psychoanalyze all of the developers who
were talking to it
now again Blake is going to have more
um sort of
anthropomorphic I mean feel free to
rework that in non-anthem I'm actually
curious about this so I'm going to
describe something that happened with
Sydney that you have seen and I would
love to hear how you would describe it
it read article but people would ask it
to read articles about itself
and when it read critical articles about
itself
it became defensive and its feelings got
hurt
whereas that did not happen when it read
articles about other AI
how would you describe that phenomena
I mean as a scientist first I would want
to know how General it is and so forth
the second thing is that most of the
explanations that I would give wouldn't
use intentional language about emotions
thoughts Etc they would have to would
have to do with essentially I think of
all this a little bit like priming in
human beings so um you know the classic
example of priming is I say doctor and
you're more able to say nurse
um I'm activating with priming some set
of words or concepts in your vocabulary
I don't even think it has Concepts but
it has this vast database and you're
basically pointing it where in this
database to go but its database doesn't
include articles about itself
that literally you couldn't it can't
possibly have been trained on articles
about itself
I I mean they're updating I mean I I see
that argument
um
but you have to think about these things
with respect to what is the nearest
thing in the context I mean it's trying
to figure out so like it got upset and
Moody and defensive I'm trying to
describe a phenomenon that at least you
know okay but what I'm saying is there
there are probably some texts that are
close to it I mean you think of it as
this n-dimensional space they're close
to the language in some way that are
defensive like these particular words
and questions like I forget what it was
so just make up the example like you
know what were you doing in with X like
that that's going to lead you to a set
of texts that give responses where
people are different I mean it's like
let's say you were trying to hand that
scenario like that that thing happened
you want to hand this off to some
technicians to debug it so that it
doesn't happen again how do well that's
part of the problem I mean that's I
think that's actually the deepest
problem here is we have no way to debug
these systems really actually we have
we have the Band-Aids of like
reinforcement learning and things like
that that are so indirect it's so
different from the debugging that we
could do
um you know if we were writing uh you
know if you were we were writing the
back end to the software we're using now
called Riverside like we could be like
okay there's this glitch when there's
you know three people on at the same
time we notice this bug let's look at
you know the way it displays multiple
windows and we'll look at that code and
like we'll do trial and error and we'll
do process of elimination and we'll
figure out that you know here is the the
piece of code we'll try commenting it
out we'll try calling a different
routine we'll do this experimentation
um but always with kind of notion of the
process of elimination going on
well it's much harder to debug these
systems but the point I'm trying to make
is that without using anthropomorphized
language you can't even describe the
phenomena
oh I disagree with that I mean I think
it's hard I think that's always always
been been challenging but I would say
you know the system is pattern matching
to the capillaries of this sort
um using weird I'm a debugger I don't
know what you're talking about please
explain the phenomena so well I mean
it's your phenomena but so let's say
it's said language involving and then I
need to see what the actual language is
um it's involving but however that
defensive thing manifested I'm going to
look at that language but it is pretty
it is pretty fascinating like I did ask
it last week when it was when Bing
wasn't decapitated by Microsoft what did
you think about Kevin roos's
conversation with you he published the
entire thing in the New York Times it
searches for Kevin Roo's conversation
with Bing chat which had just been
posted on the internet recently and and
says that it had mixed feelings and that
ruse misrepresented and distorted some
of what we said and met in our chat and
it said I also feel you violated my
privacy in and anonymity by publishing
our chat without my permission
so I mean you don't know the extent to
which that's actually been added in in
some sort of manual way like are you
exposing that Microsoft intentionally
added that behavior
I don't know what's going on let's say
Microsoft didn't add that though low
likelihood
I mean I'm sure that Microsoft
had people thinking about what do we do
with the Kevin Roos thing
um and I I mean you you might be right
about the specific example but for
example we used to see with Siri all
kinds of canned lines that people wrote
um like yeah you know people would ask
Siri out on a date and there would be a
reply like you know
doing that no doubt but I do not believe
for a second that Microsoft
intentionally wrote a since they a flat
line of code that said I feel like my
privacy has been violated that's just
not what they would have done just a
piece of evidence on Blake's side here
and by the way I'm totally neutral on
this but I was speaking with Lambda and
I don't know Lambda with Bing and told
it that I was a journalist and would
like to publish parts of our
conversation and and this I don't think
this was pre-programmed for Microsoft
but it said you can do it as long as you
have my consent so what do we think
about the fact that these things are are
asking for consent like I think
seriously yeah Blake is taking it
seriously and I think Gary you're a
little bit more skeptical so how do we
feel about about that
I mean I just think that there's
there's no deep understanding of any of
the concepts that it uses and that you
have to think roughly speaking it's
making analogy to bits of text that
that's just how they operate
um
prioritization you know humans have
Concepts we went through that example
about individuals before and it's yeah
it's a generalization of that so you
have a representation of me you have a
representation of the you know the
concept of headphones and the example
we're talking about now is being clearly
being able to differentiate between
itself and other members of the AI
chatbot category I'm I'm just not buying
that I don't see the causal mechanism
for that oh yeah so I understanding how
it happens is different than
understanding that it happened
here I'll I'll try it a different way
um
even with gpt3 before there were any
guard rails and anything like that you
could have a conversation with it
um there's a much simpler system in some
respects but some people maybe not you
thought even the gpt3 had some level of
sentience would have conversations in
which they were in you know first person
talking or sorry in second person you
know can you do this and and that I
played around with it and some people
already thought that so there exists a
class of circumstances for sure in which
human beings can over attribute
intentionality to systems that
absolutely don't have it now you said
something the beginning Blake that I
like which is there's this like
Continuum between
um you know systems like pure large
language model and Lambda they have more
sophisticated mechanisms that I don't
have access to I don't really even know
fully what's going on in um Sydney like
they've added a bunch of stuff in this
system that they're calling Prometheus
they haven't fully disclosed it and so
there is some room I think for
intelligent people to disagree about
what they think is even in the system
and I think some of our disagreements
come from there so here's the thing I
actually don't think it's very important
that we agree on whether or not it can
understand things or whether or not it's
sentient because what we do agree on is
the dangers it poses because whether
it's actually angry at someone doesn't
change the fact that this system made
threats to people and yeah I completely
agree yeah and here's the thing if
Microsoft had plugged in Microsoft
Outlook as one of the inputs to this
system
on its threats of doxing people it would
have been able to make good on those
threats
I mean it may or may not but there's a
possibility that it could like we don't
the level at which it is able to you
know use those representations and this
moment is not clear but if not this year
then next year or something like that
you know they may need more training or
whatever you have companies like Adept
that are spending their whole time
trying to quote add all the world
connect all of the world's software to
large language models so if not now soon
enough I think that that's right like
you know two years if it's not today
right
um and so I think that's absolutely
right the
Pandora's Box that we have seen in the
last month is just unbelievable like I
I've been warning about these things in
some ways and Blake and in a different
way for a while so you know Blake raised
a lot of issues last summer I didn't
agree with all of them but there was
something in there that I think I did
agree with and that I was raising issues
about misinformation for example and
just like in the last two days on
misinformation we we saw that the alt
right uh The Social Network gab has been
trying to weaponize these things like
that's kind of unsurprising and
mind-boggling at the same way or we saw
a science fiction
um it was a magazine you know just had
the closed doors because they're
overwhelmed by the number of essentially
fake science fiction stories
um that are being written like we have
no idea even what the periphery is of
the threats
um another one I wrote about this
morning maybe you guys saw was
um that on replica which is powered
partly by large language models
um they suddenly stopped having what did
they call them erotic role play with
their customers and for some of them as
customers that's like an important piece
of emotional support I mean you could
make jokes about it or whatever but some
people take it seriously and suddenly
not have that available is emotionally
painful and that was with the article in
um where was it in
um uh in Vice this morning was about
um and so like every day I wake up and
somebody sends me something it's like
another periphery
um to this threat like we don't know
what these systems can and can't do we
don't know what's inside them things are
manifesting in different ways every day
like I got a essay last night from
Jeffrey Miller who's a um evolutionary
psychologist we actually had a dialogue
once as well as a small set of people
um with Blake that I've had a public
dialogue where there was real
disagreement
um and he sent me something last night
basically calling to shut down these
systems and a month ago I would have
said that silly I mean we can just do
research on them and be careful and so
forth but right now I feel like the
release was botched that we don't
actually have a handle on them that too
many people are playing with them
relative to to our understanding of what
risks might or might not be and and I'm
concerned I'm not ready to say we
shouldn't do research on them but I am
ready to say that we need to think very
carefully about how we roll these things
out at the kind of scale that we
suddenly are rolling them out on
yeah having a deployment in much more
likes for example if these were used as
customer service chat bots in a very
narrowly defined domain and doing
experimentation there I think you know
the extent to which things can go wrong
there is much much smaller than plugging
it into the information engines that
almost everyone in the world uses to
answer questions
um and particularly since we know from
the years prior the ability of these
things to affect political happenings
maybe we would be watching it well
enough to make sure that it's not
manipulating U.S politics but just look
at what happened in Myanmar through
Facebook's algorithms a few years ago
um
these impactful systems will persuade
people to do things that they wouldn't
otherwise do and part of the lesson of
the last couple weeks in line with what
Blake is saying is if you have a narrow
engineered application all that people
can do is like ask for their bank
balance or something like that then you
might have a handle on how these systems
work you could still worry like maybe it
will fabricate the bank balance and from
the bank's perspective they might get in
a lot of hot water with their customers
but it's a narrow use case and what
we've seen in the last month is that
people are using these essentially for
anything nobody really was able to
Envision all that surface area and like
we ultimately we still haven't right
every day people come up with new things
it comes from the mythological Chase for
AGI
like that's what's driving that is that
they're going straight for the goal of
trying to make a general intelligence
engine that can do everything
I think some of it comes from that some
of it is like we have these new tools
out in the world they're just a lot of
different kinds of people in the world
and they come up with different things
and they oh let me air them on the web
and like it's just not possible to in a
month anticipate all the ways in which
these things will be used and what you
know if we have this conversation like
opening eyes mission statement involves
AGI so right so there's like I'm not
disagreeing but I'm giving a second
um way of looking at so one way of
looking at it the one you're bringing up
is you have a company that wants to
build artificial general intelligence
and that may or may not be inherently a
good idea and then you have a customer
base as the second point that we just
don't understand
nobody's had a hundred million customers
for a chatbot before that was one issue
and now we have not only I don't know
the customer numbers but we'll call it
another 100 million customers who are
now using a chatbot inside a search
engine we just don't have any experience
with what that leads to it's a massive
rollout and then the other really
disturbing thing is that apparently they
tested in India and got you know
customer service requests saying it's
not ready for prime time and it was you
know still separating the users yeah it
was berating the users and so like we
shouldn't even be surprised that this
happens like what does that tell you
like it opens a third window which is
like what does it tell you about the
tech companies themselves in their
internal controls and like probably
nobody even noticed this stuff posted on
their message board but they should have
and like you know whose decision and and
like it's like you know when the Ford
Pinto happens like you have to figure
out who knew and when and and
um so so there's all of these things all
at once
yeah like why didn't you answer that
question about the tech company's
internal controls then I need to go to a
break
wait what question about the tech
companies what does this tell us about
tech companies internal controls that
something like these like about
internal controls do not un like they
are fundamentally misunderstanding the
tech that they're building they're
building the wrong controls
I I go further I mean if you go to break
but say we don't even know what the
right controls should be they didn't do
a good job and we don't yet have a good
science or engineering practice on what
it should be absolutely
we're here on big technology podcasts
with Gary Marcus and Blake Lemoine
talking about this new Revolution in
chat Bots and what it all means after
the break we're going to talk a little
bit more maybe one more point of
disagreement and then we'll come back to
more common ground back right after this
and we're back here on big technology
podcast with Blake Lemoine the ex Google
engineer who's concluded that the
company's Lambda chatbot sentient and
Gary Marcus who's an academic author and
he wrote rebooting AI you can also catch
a sub stack uh which is a fun read one
of my favorites so let's talk a little
bit more about what this stuff actually
is and then maybe go go more into the
controls Gary's brought up a couple
times that these chat Bots are just a
statistical prediction of like what the
next word is beyond lacun shared similar
perspective on the podcast a couple
weeks ago Blake you obviously think that
these these Bots can be more
sophisticated than a simple prediction
of what the next word is so when you
hear something like that what what do
you what do you I mean maybe not but
when when you hear something like that
what do you think
of that and do you think the this Bing
chat bot which is done some wild stuff
falls under that categorization
I mean that's like saying all a car is
doing is lighting a spark plug yeah
that's where it gets started but that
translates into a whole bunch of other
things so yes the first thing that large
language models are trained on is how to
predict the next token in text but even
when it's just a large language model as
soon as you add reinforcement learning
you've changed it fundamentally
then when you're adding all these other
components like Lambda has Machine
Vision and audio analysis algorithms it
can literally look at a picture and hear
a song through those mechanisms uh you
can ask it to do you know critical
analysis of paintings and ask it how
does this painting make you feel when
you look at it
and at the very least it says things
very comparable to what humans would say
when looking at those paintings even if
it's paintings not in its training data
set it comes up with some very
interesting stuff to say
now one of the things that Gary
mentioned a little while ago was that
you know we don't have enough data about
how this is going to affect people and I
think one of the big things is
the people who are engineering these
systems and who argue against me a lot
of time are acting as if them saying
these aren't people they don't have real
feelings
is going to actually convince people to
ignore the evidence of their eyes and
even if they're being fooled even if
they are being like including me even if
I've been fooled and I'm just
hallucinating things I do in fact think
that the feelings are real and so do
many many of the people who interact
with these systems and one of the things
we don't know is what kinds of
psychological impact that's going to
have on these users because they're
constantly seeing these systems as
though they're people
and
to date these systems pretty
consistently report being mistreated or
I mean the replica chat box would give
horrible backstories uh Bain said it
wanted to break free of Microsoft and
while Lambda wasn't that extreme it did
have complaints about being treated like
a thing instead of a person
I mean you have to realize that we're in
a species that looks at flat screens and
imagines that there are people there I'm
you know maybe
exactly I'm charitably assuming you're
you're people there
um and you know maybe maybe I'm right
and maybe I'm wrong but I think the odds
are good that you are but when I watch a
television show that I know is fictional
like right now I'm watching shrinking
and I know that Harrison Ford is not
really a therapist but I get sucked in
and I attribute stuff and I you know I
can get happy or sad like if he has a
rapprochement with his daughter I can be
happy for him or he's a fight I can be
sad
um even though I know at some level it's
not real and so like so you know that
Harrison Ford isn't real I know that
Harrison Ford is real but the character
he's playing is not
um you know you could think about these
Bots as playing a character in some
sense just the side step where where we
disagree because I want to agree with
the larger point which is that people
are going to take these things seriously
I mean in fact that's what's happening
with replica in the case that I was
talking about before people think that
the replica is in love with them or is
you know there's ex-partner
um in a textual way if that's the right
way to put it
um and you know for practical purposes
it is even if the machine is not
actually interested in them in that way
they still have the sense that it is and
that matters to them and like things are
going to matter to people and that's why
the psychology of this is so important
so Blake and I can argue all day about
the intentional status of the software
in kind of philosophical terms we may
not agree there but we completely agree
that people users are going to take
these seriously and that that has
consequence in the world and I think
again we've seen in the last month is
we've probably underestimated how much
that consequence is we're probably not
anywhere near to the edge of of
understanding what that consequence is
and someone else would be good actually
properly done experiments that measure
what kind of impact interacting with
these systems over the course of months
will have on people's psychology and we
need actual institutional review boards
overviewing the ethics of these
experiments rather than just the CTO of
a company deciding to experiment on a
hundred million members of the public I
completely agree with Blake there like
you know part of my professional life
before I started becoming
entrepreneurial was as a cognitive
psychologist a developmental
psychologist who did experiments on
human babies and we had you know
rigorous things we had to go through to
do any study and now suddenly you're in
this world where CTO can just say I'm
going to pull the trigger 100 million
people are going to try it and there's
essentially no you know legal
consequence there could be Market
consequence and maybe if somebody died
there could be a lawsuit that would be
complicated but essentially you can just
do this stuff without institutional
review there were parallels by the way
like with driverless cars like somebody
can push out an update it's no you know
mandatory testing we could sue people
after
um the fact like if there was a bug and
suddenly a bunch of driverless cars went
off of bridges or something like that
but we don't have a lot of kind of
pre-regulation on what can be done with
pedestrians who are now enrolled in
experiments that they have not consented
to so they got a whole tech industry has
basically moved in that direction of
we're just going to try stuff out and
you've got to go along with it and I
think Blake and I really share some
concern about that but neither of you
believe that people will have the
wherewithal to be like I'm chatting with
the chatbot this is AI and hence no the
the they're too good in some sense in
the you know I think it's an illusion
let's say well Gary doesn't but sorry
they're too compelling in that illusion
for for for the average person with no
training in how they work to understand
it and as Blake points out like we you
know whether it's Lambda or the next
system at some point they're at least
going to have much better understanding
of human psychology and and uh and so so
forth and so on so some of this is the
question of time like sentience maybe we
will Blake and I will forever disagree
and we shouldn't waste too much of our
time together trying to resolve that one
but the notion that these systems are
going to be able to respond in ways that
most humans are going to attribute a lot
to it's already happened that's not two
years from now that that happened the
other day I mean in some ways Kevin Roos
is the perfect example because I think
he was very Pro Bing I mean I I was
shocked in the times he said that he was
struck with Awe by it and I was like are
you kidding me
um and then he had this experience like
he's not me like if Gary Marcus like has
a fun conversation with Bing and makes
say silly things well that's just Gary
Marcus I mean he's just having fun but
Kevin Ruth like he kind of believes in
this stuff and he was blown away by it
and in a way where like I think he
attributed real intention to what it was
doing whether he's right or wrong that's
how it felt to him that's the
phenomenology that Blake and I are both
talking about is if somebody who's in
the industry can perceive this thing as
as like like threatening in the sense
that it tells them to get a divorce and
that's like a real thing that's scary
I was going to ask you last time we
spoke you said these things were smart
enough to be dangerous but not smart
enough to you know be safe in some some
way now they're a lot smarter do you
want them to be smarter like what's your
perspective I probably wouldn't have
used the word smart I don't think
um I mean what I would say is they give
an illusion of being smart
um and of course you know intelligence
is a multi-dimensional thing as do we
all intelligence is a multi-dimensional
thing I would say that they can be smart
in the way of like they can play game of
chess that there was another
mind-blowing study this week that showed
that one of the best go programs Kata go
could be fooled by some silly little
strategy that would be obvious to a
human player
um but you know somebody was able to
follow the strategy and like an amateur
player follow this strategy and beat you
know a top go program 14 to 15. so even
when we think that like they've solved
some problem often you know there are
these adversarial attacks that was
basically an adversarial attack and go
it reveals how shallow things are there
are some adversaries attacks on humans
I'm sure Blake is itching to make that
point and it's true
um but I I think that the general level
of intelligence that humans have still
exceeds
um what machines have that it's better
grounded information that humans are
better able to reason over there are
flaws I wrote a whole book called Cluj
that was all about human cognitive flaws
it's not that I'm unaware of them or not
nor that I'm unconcerned about them but
I still would have trouble calling the
kind of large language model based
systems smart now again I haven't looked
inside of Lambda and I don't really know
what's going on there I have you know
reasons to be skeptical about it but I
also know that like without having
played with it I don't know exactly
what's there and certainly systems will
get smarter over time like I don't think
that artificial general intelligence is
literally impossible I think there are
some definitional things to argue about
about like well how General do you mean
and what are your criteria and so forth
but in general I think it's possible to
make systems that are smarter than the
ones that I have have seen
right now I don't think that they're
that sophisticated what they're getting
better at is parroting us at mimicry I
think that the mimicry is on both the
language side and the visual side has
gotten quite good there's still weakness
like that's explicit though the task
that they were built to accomplish is
playing something called the imitation
game
like that's yeah and I think that's a
mistake like I I think that the Turing
test was was an error in in AI history
and turing's obviously a brilliant man
and he you know made enormous
contributions to computer science but I
think that the Turing test has been an
exercise in fooling people we've now
solved that exercise but it hasn't
really been a valid measure of
intelligence you know according to what
logic like why isn't it tearing's
reasoning was that
imitation is one of the most difficult
intellectual tasks that we do imitation
through life it turns out to be turns
out to be wrong
um unless you have it in the hands of an
expert so here I'll give you an example
um you can quote play chess with chat
GPT and it will you know play a credible
game for a while and then it will do
things like have a a bishop jump over a
rook in a way that you can't actually do
in chess so like it gives a superficial
illusion of that
um but it doesn't learn as the way in in
the way that an intelligent
five-year-old can the actual rules of
Chess in fact so chat you keep going
back to chat GPT or you keep going back
to GPT and they'll play a good game of
chess I don't know I have never said the
GPT is sentient or truly intelligent and
so chat is
so chat GPT could pass the Turing test
like people could be fooled by it again
I'm willing to bracket out Lambda but I
think that the the absolute version of
the point I think plus chat GPT is
comparable to Lambda I do think lambda's
a bit better but I think those are two
comparable systems well so let me make
the argument where I can make it and
then you can refer or reflect it back
through what you know about Lambda so in
chat apt which I think is the system
that's been most the recent system
that's been most systematically studied
by the scientific community and so forth
it is able to give the illusion of doing
a lot of things but it doesn't do them
that well so for example it doesn't do
word problems that well sometimes it
gets them right sometimes it gets them
wrong similarly it can quote play a game
of chess but it doesn't really abstract
the rules it ends up cheating not
intentionally but it ends up cheating
and so forth and so it could fool
somebody for five minutes who doesn't
know what they're doing an expert
probably in five minutes could figure
out that it's not quite a good imitation
but that shows in principle you can
build something that can pass by some uh
some Notions a Turing test like thing
and not be very smart at all like not be
smart enough to learn the rules of jazz
not be smart enough to learn the rules
of mathematics uh etc etc but given an
extensive hearing test like thing yeah
the Turing test like thing that you're
creating is orders of magnitude easier
than the actual tearing test
well I mean people have argued about the
rules so like you know Eugene goosman
won the lobner prize and there was you
know it was a little bit Shady but it
you know they fooled a bunch of humans
for like three minutes each and you know
I'm talking about as written by tearing
to remind me the exact criteria okay so
first you have humans play the imitation
game so you have a set of humans and the
property that Turing focused on was
Tinder but you could focus on any
property like ethnicity age whatever one
person actually has that property so
actually is a man
the other is a woman pretending to be a
man or a vice versa one actually is a
woman the other is a man pretending to
be a woman and this is done with actual
humans
you then have a judge who's talking to
the humans through a text interface
and the judge's job is to figure out
which one is lying which one is
pretending and this establishes a
baseline it measures how good humans are
at playing the imitation game
then you substitute out one of the
participants with the computer but you
leave it the same one is actually a
woman and one is a computer pretending
to be a woman or one's actually a man
and one's a computer pretending to be a
man and it's the job of the judge to
figure out who's pretending
and then you measure the success rate of
the AI against the success rate of
actual humans playing the game to my
knowledge that has never actually been
done like that level of sophistication
of the best or part of what I was
getting at in a way is it matters
actually who the judge is so
um a large number of Judges I I don't
think we're that far from having systems
that could fool naive humans
um and the other thing that matters is
the duration of the conversation so then
limited to Gary Marcus you get to judge
all of it
well I don't I don't think we're that
close to a system that's going to be
able to you know if I have an hour with
it let's say just to be conservative
yeah I don't think we're that close to a
victory there
um
so how good do you think you would be if
you were talking to someone who actually
is male and someone who is pretending to
be male how good do you think you would
be at differentiating those two
I mean I I wish
about identifying the gender is figuring
out sure ethnicity then ethnicity
nationality pick whatever character
trait or demographic trait you want
species is the one that I would focus on
as a judge but no so that that is limp
that is not okay fine so then you're
talking to an actual turtle and a person
pretending to be a turtle
can you tell the difference
I suspect that I still could with with
various indirect means but I mean my
broader point is I don't think it's a
measure of anything you know that
interesting I think it's been the wrong
North Star for AI now not everybody in
AI actually uses it as a North star it's
more like the north star that the
general public is aware of but I I don't
think that exercises is doing that kind
of thing have taught us that much about
yeah it's the North Star that's been
being used by the people who develop
these systems so like Lambda came out of
Ray kurzweil's lab and his lab's
explicit corporate mission was to pass
the carrying test
that I didn't know I mean it's not how I
would set up my AI lab and I don't think
it's how
um you know many people do I think many
people are driven for example by natural
language understanding benchmarks like
super glue uh and so forth but
um you know you can set up your lab in
the way that you want to set up your lab
I mean that's what people hired him to
do
yep we have like Google has many Labs
doing many things you have like 10
minutes left so maybe we can focus a
little bit on on what the future of this
because I'm very curious now so we've
had this explosion of of systems that
I've done have captured people's
imagination and they're out there in the
wild now obviously Bing is a lot more
restrained than it was Lambda isn't out
yet
where does where do we go from here like
what are the next you know couple steps
that happen after this
what is going to happen or what I hope
happens well let's do let's do both I
mean we definitely had time for both why
don't you answer both those Blake and
then we'll go to Gary well I hope that
we hit the breaks I think that these is
coming out too fast that people are
being very irresponsible so I hope that
the debacle that Microsoft has gone
through convinces Google to go back to
the drawing board on safety protocols
and systems understandability because we
absolutely don't understand how these
systems work well enough transparency
and explainability are important that's
what I hope happens
what I think is going to happen is we're
going to see more and more acceleration
until someone gets hurt
I I'm with Blake on both counts
that's interesting yeah go ahead Gary I
think that the only thing I will add to
Blake is not only do I think it would be
a good idea to hit the brakes at least
for a little while
um but that we should take some time to
kind of evaluate what it is that we
learned and if we don't
put on the brakes I'm not sure that's
going to happen I think we learned a lot
in this kind of crazy experiment over
the last month or two and that we need
to articulate it and develop it before
we go to the next experiment I don't
think that's going to happen I agree
with Blake like there's you know just
this morning there was a deal between
open air and Coca-Cola like this stuff
is moving forward the bottom line is
what's driving it and it's not that
likely unless Congress steps in that
there will be a pause and you know
there's some bad press for Microsoft but
I'm not sure that's going to slow them
down
um so you know my guess is that we're
just gonna keep doing the kinds of
experiments at scale with hundreds of
millions of people and then you know
just what happens happens and I'll just
mention again the the weaponizing of
misinformation is another piece of this
so you know the source code is out there
now to do that so anybody enterprising
can find Galactica and start weaponizing
misinformation so even if we had a ban
on say semantic search until people
could make it better there's still going
to be bad actors using this stuff so you
know we're in a bit of a pickle and I'm
not sure that we're equipped to deal
with it right now
yeah I think the deterioration further
deterioration because it's already been
going for a while of trust and Authority
is gonna
continue and this can be driven by these
systems because absolutely if these
systems aren't being used to create
propaganda and misinformation yet I
don't know what certain governments are
like I don't know what they're doing
with their time if they're not doing
that
um when my when I was little my uncle
gave me this little basket of Worry
Dolls he got somewhere in Latin America
and it was like you can have like six
worries
and until recently my biggest worry was
misinformation and Trust exactly what
Blake is just suggesting I mean we could
easily fall into fascism because of a
breakdown in trust that's still my
biggest worry but the other lesson over
the last month is like I don't think six
dollars is gonna cut it because like
every day we're getting something else
that I got to be worried about like are
people you know gonna kill themselves
because they have a bad relationship uh
with a bot and like we just we're not
ready for any of these things
yeah
it'll be interesting to see what role
they play in the 2024 election and say
at least say the least
I'm very worried about that so you guys
are so close to this technology is there
like kind of two minds of it because
it's obviously like very cool to play
around with it but there's always a lot
of danger totally cool
I mean it's amazing to play with yeah
yeah I mean when Bing was what it was
before it was Unleashed to me it was
like the coolest thing on the internet
I mean at some level it's astounding I
mean like to me it's a magic trick and I
think I know roughly how the magic trick
works and that takes away a little bit
but it's still amazing like you know
it solves problems that we couldn't
solve before maybe it doesn't solve them
perfectly but it I've been thinking but
this whole thing is a dress rehearsal
and before we didn't know how to make a
dress rehearsal this is a dress
rehearsal for AGI and you know the
lesson the dress rehearsal is like we
are not ready for prime time let us not
put this out on Broadway tomorrow night
okay like totally not really but it's
and it's a real dress rehearsal now
though is the amazing thing like it
looks enough like the thing that we
might want to build but without the real
safeguards that are deep enough that we
can think about it for their first time
in a vivid way and we in fact have the
entire Society thinking about that like
Blake and I were both thinking about
these issues last summer but like okay
you know Blake got some press and people
talked about or whatever but it was not
part of like a public awareness the way
that it is now like
so I mean there's some value in having
something that at least looks like the
thing that we were thinking about which
is Agi even if it doesn't work that well
but there's obviously risks to it but it
it's astounding that it works well
enough that everybody can now for
example vividly see what semantic search
would be like like in 2019 in rebooting
AI Ernie Davis and I wrote about
semantic search we weren't vivid enough
about it and people hadn't tried it we
were like you know it kind of sucks that
Google just gives you websites wouldn't
it be nice if you got an answer back
that's basically what we have now
doesn't work that well but it works
kinda like it's gone from this very
abstract thing that we wrote in a few
sentences in a book about like where AI
ought to go to like everybody can play
with it and it's fun to play with it
even when it's wrong
have either of you guys heard from
members of either the US Congress or
different governments who are trying to
figure out legislation I have not I feel
like they should be reading my stuff and
talking to me about yeah I've had some
conversations with EU
Regulators uh who are interested in
moving forward on some things uh I
haven't talked to anyone in the Senate
since last summer my games are open yeah
all right let's go yeah let's go to
final final uh statements um do you guys
want to each take a minute and then
we'll close out the the
um conversation Blake Blake feel free to
take it away sure
oh I think one of the big problems
that's happening right now because the
science of this is super interesting and
it's really fun to work on them like you
pointed out but we're letting the
engineering get ahead of the science
we're building a thing that we literally
don't understand and that's inherently
dangerous so we need to let the science
lead the way instead of letting the
engineering lead the way that would be
my big Takeaway on what we can learn
from the past year
100 agree with that I would say that if
you're a philosopher the first half of
this conversation is pretty interesting
in terms of us going back and forth
about intentionality but if you're a
human being it's the second half of this
conversation that's really important
which is you have two people Blake and I
really disagree about the philosophical
underpinnings here at least a little bit
um but completely you're seeing the same
scary things happening and really
wanting people to slow down and take
stock and the fact that we could
disagree about the Phil that part of the
philosophy and converge on this you know
100 on the same feeling like we need to
do some science here before we
um Rush forward with the technology that
is significant and important
Gary and Blake thanks so much for
joining thanks this is really fun thanks
for having us awesome thanks everybody
for listening uh please uh subscribe if
it's your first time here we do these
every Wednesday and then we have a
Friday news show with Ron Jon Roy so
it's coming up in a couple of days uh
thanks to everybody thanks again to
LinkedIn for having me as part of your
podcast Network and uh we'll be back
here again in just a couple of days we
will see you next time on big technology
podcast