How Generative AI Is Changing Travel — With Priceline's Marty Brodbeck

Channel: Alex Kantrowitz

Published at: 2023-12-12

YouTube video id: HX_YuEbhE4g

Source: https://www.youtube.com/watch?v=HX_YuEbhE4g

hello YouTube we're here with a special
edition of our conversations for you
sponsored by Priceline we're going to be
talking about how generative AI can
change the travel space we've heard so
much about Theory and we're trying to
get into practice we've talked about it
in the terms of public relations and
marketing now let's talk about how it
can change travel joining us today is a
very special guest Marty broadbeck is
here he is the chief technology officer
of Price Line Marty welcome thanks for
me thanks for being here let's go back
to last November chat GPT comes out do
you immediately think hey this could
revolutionize the way that we do
business or is there a bit of time that
it takes to set in before you think
about maybe we can use generative AI in
the way we reach customers the way that
we position ourselves the way we do
business
internally well when when chat gbt came
out we we thought that this potentially
is a ground piecing ground piece of
technology that could fundamentally
change how customers interact from a
travel perspective we as a company have
been have been using machine learning
for a long time and know the power of
what it can actually achieve and
accomplish from an Automation and a
customer perspective and so when we saw
the the launch of chat EBT we
immediately started brainstorming on
ideas and areas of the company that we
could be applying this kind of
capability towards and where has it gone
into practice for you because you know I
think there's there's this question of
do people actually want to discuss their
travel plans with the chatbot or would
they just prefer the traditional
graphical user interface that you have
on a priceline.com right now yeah so we
we immediately started looking at really
four areas where we felt generative AI
could be really applicable to uh our
company and our products and more
importantly to to our
customers um the the first base was
really embedding generative AI
capabilities into our products such as
our checkout flow and our post booking
flow uh the second is we thought that a
generative AI capability like Chad gbt
or Bard or cohere could have real
relevance in our post booking process in
the way in which customers interacted
with our company post booking um and had
questions around products
fairs uh fees Etc as it pertains to the
products that they bought the third is
really around how we generated marketing
content and how we apply the technology
to provide richer content to our
customers about destinations or cities
that they were potentially traveling
towards and then the last was we took a
hard look internally about how we could
apply generative AI capabilities to
further Drive greater levels of
productivity within the company
particularly within you know my
organization which is product
engineering great and so I want to talk
let's go through one by one as we
through this conversation and talk about
them first of all in terms of like the
actual consumer experience interacting
with price line going to go back to that
question before people are there's a
debate right now really in terms of
whether people want to stick with the
traditional website architecture to book
their travel and ask questions about or
or or find information about it and how
useful something like a chatbot might be
in terms of booking travel finding
destinations you know maybe people are
thinking I can do my entire itinerary
just by saying to a chapot hey you know
find me some Island in gree that I can
stay at for seven days for you know
$2,000 a person I'm just throwing
numbers out there so I'm curious from
like your perspective you're the one
that's going to you know help enable
this where does your thinking land on
that front well we're a very data driven
company so we started embedding G of AI
capabilities right into our checkout
flow where if a customer had a question
around a hotel a flight a product that
before they booked that they could have
the experience of asking questions about
the product that they were potentially
booking and so we we Dove right into the
heart of our customer Journey went right
to the checkout flow where Not only
would we enable uh customers to ask
questions and get answers about the
products that they were about to book
but also embedded the checkout flow
right within the experience of
generative AI to allow them to have a
frictionless experience that once all
their questions were answered that they
could immediately just go to purchase uh
so that was where we started and more
importantly because we're highly AB D AB
testing driven we ran tests to make sure
that the bot had a positive impact on
the user experience in the you know in
revenue and after several iterations of
prompt engineering we got to a point
where the bot was AB testing positive
and so we pushed it out to 100% of our
our customers on our platform the next
part that we tackled which we recently
just launched is putting generative AI
into the post post booking process to
answer any questions that customers may
have around any of the products that
they were booking after their purchase
uh and again because we're a very AB
driven tested company went through
several iterations of you know prompt
design and iterations and got to a point
where uh in our post booking process the
bot had a positive impact on the
consumer experience now we shifted our
Focus towards what we're calling the
travel concierge part of our business
which is really the trip planning and
we're currently working on capabilities
to allow users to do free tests free
Tech searching so I'm planning to be in
San Francisco tomorrow at 10:00 am. can
you recommend a flight and a hotel
capability or a set of products that
would be relevant based on the meeting
that I'm going to have next day and then
walking the user through the
recommendations and the ability to do
trip planning as well um more
importantly I think you know it's
changing the Paradigm for how people
search products um rather than having
the your typical Search bot uh search
box where you have to put in when I'm
when I'm going on the trip and when I'm
coming back having it be more of a
conversational experience using natural
language rather than you know strict
text boxes and that's what we're
currently developing another thing that
we're super excited about is embedding
generative AI capabilities provide a
richer experience for our customers when
booking hotels there there are several
things that customers are very
interested in number one points of
interest around the hotel uh or the area
that their hotel that they're
potentially booking is located to the
neighborhood description of where the
hotel is particularly in major cities
like New York Los Angeles Las Vegas
London and Paris the neighborhood
description is key in making the
decision about not only what can I do uh
on this trip but what potentially my
family can do uh so that's very
important hotel reviews so summarizing
hotel reviews that um that people have
put onto our website is very important
for people to make a decision and that's
really using generative AI to basically
provide these capabilities embedded in
our e-commerce platform as well so that
those to me are game changers in terms
of providing a much richer personalized
experience for our customers rather than
a two-dimensional one that's based on
when I'm going on the trip and when I'm
coming back on the trip yeah definitely
I mean I'm thinking just about searching
for hotels and what I typically do is
search for hotels first I go location
then I go uh rating and then I look if
it has good Wi-Fi and I disqualify it at
the end if the Wi-Fi isn't good yeah and
I imagine with a chatbot you could
almost get that all together being like
can you find me a hotel in like Union
Square in San Francisco it has to be
above an 8.0 rating and the Wi-Fi has to
be good and it can comb through those
responses and say hey here's some that
people generally like the Wi-Fi so it
sort of saves a tremendous amount of
time one of the interesting insights we
got from our AB testing is a lot of our
customers had uh questions about hotels
being pet friendly really uh and so
being able to provide answers to those
questions in real time for our customers
as they're planning their their trip has
been you know super helpful for us you
know Marty one of the things that I T to
tend to find when I'm searching on
something like Bing or Bard is um it can
find information that you wouldn't
generally find in the first like 10
results on search right it surprises you
with information and there's only so
much information you can hold on like
the front page of a of a travel website
like price line so does does um the the
search functionalities that you've that
you're enabling with this bot does it
offer surprising or like deep deep down
in the data results that people can you
know end up making better choices with
when they encounter it well one of the
one of the great things around
generative AI is that you can marry it
with your own data sets internally which
uh for us has been gamechanging so what
what we're doing within our platform is
we're taking all of our hotel content
and putting it through a machine
learning pipe line and Maring that data
our own data with authoritative sources
of location information and map
information to provide complete detailed
accuracy on things such as hotel reviews
neighborhood descriptions points of
interest uh the second advantage to
taking this approach it also reduces the
amount of hallucinations that you would
get from a large language model so
marrying the large language model with
our own data sets with thirdparty data
really drives down uh the amount of
hallucinations that we get from these
kinds of conversation and almost creates
authoritative sources of cach
information around hotel reviews points
of interest neighborhood
descriptions um that allow the site to
perform faster because the other thing
that to to take into account with all
these things is the cost um cost in and
running and managing a generative AI
platform could Skyrocket so you have to
be very smart about when you refresh or
make direct calls to the large language
model and this third-party information
yeah and how about knowing the customer
so I think that one of the things that
everybody's looking for when it comes to
these Bots and a real point of
frustration using something like a chat
GPT or a Bing is it has like Goldfish
Memory right so you go say hi you have a
long discussion you talk about your life
with it and it has you know it's very
empathetic conversation with you you
come back 10 minutes later totally
forgets who you are now when it comes to
travel memory seems like it's something
that could be so huge because if price
line for instance can remember what type
of hotels I like or that Wi-Fi is
important for me and I go into that bot
and it says Hey welcome back okay we we
see that you're interested in traveling
to Austin for instance and and given
your past you know there is a little bit
of this in the product products right
now you know but but it would be amazing
to say like given your past uh uh
bookings we know you like places with
good Wi-Fi generally downtown close to a
good coffee shop you know ratings above
8.0 nice to see you again here's what we
found um as opposed to me having to
reintroduce myself over again so where
are we in the state of like being able
to get to that point well I think this
is more about how companies design and
develop their data AR
architectures then generative AI
understanding personalization on the
individual level so over the last
several years we've made huge
investments in building out a customer
data platform that actually understands
who you are what your booking
preferences are what you've searched
what you've booked when you've booked
have you booked as an individual
traveler or have you booked as traveling
with a family and what have been the
nuances of hotels and flights you've
taken as an individual versus as a
family and so what we've done is we've
married all that rich CDP data so that
when you log into the generative AI
experience a signal was fired that
understands who you are so in this case
you're Alex and I understand what you
searched on what you've booked in the
past what are your preferences so that
as we're going through the conversation
it completely understands the background
that you have as a customer based on all
the intelligence we built into our data
infrastructure so that way it's a very
personalized
experience um in that conversation to
understand all those preferences and
that could just play right into the bot
correct very cool so one of the things
that I've been wondering now that we've
been talking a little bit is you
mentioned that the first place that you
looked was in the checkout process how
can we you know deploy this there and
you know for me it it seems interesting
because intuitive like like going by my
gut I would say maybe at the beginning
when you're in Discovery or at the end
when you have customer service issues
but you're saying at the checkout so
what kind of questions come out at
checkout and why start
there um we felt that the checkout was
really the folr point where customers
have the most questions about a booking
like they've already are knee deep into
the details page of one of our products
so say it's a hotel and they're about to
purchase something but they have some
you know just nagging questions that
could be plaguing them like a great one
is is it pet friendly um what are the
points of interest close to this hotel
uh what's the neighborhood description
around it so you know answering those
final closing details for the for the
consumer builds more confidence in their
booking and that's why we we started
with with checkout for for from our
perspective two it's at the heart of
what we do
uh so creating a frictionless checkout
experience that answers all questions
for our customers before their booking
we felt was a great place to start very
cool okay let's talk about marketing uh
you mentioned that this has been used to
basically create high quality content at
scale for Price Line uh I can't even
imagine the scope of content that you're
working to create and so that's why I
want to ask about it um how does this
help and where what type of content is
is it creating um how are you again
preventing from hallucinations from
taking hold and how are people
responding to some of the content that
you're using generative AI tools to put
out there so the the marketing content
is all about creating merchandized
Pages uh
blogs um detailed ad Snippets you know
profiling a
descriptive um city for instance so we
we've we've taken the top 1,000 City
destinations in the world and have been
systematically creating content in those
three different channels I I talked
about you know a part of a large part of
our our business is is indirect business
that comes through people searching so
providing very high quality merchandise
personalized content around these cities
we felt was a great way to apply
generative AI
uh on the second on the second part of
your question around
hallucinations uh we've built a
hallucination
service that verifies the information
using third-party information sources
like um in the case for a lot of our our
hotel products we're using places API
from Google um which is really an
authoritative source for a whole host of
geographical uh information and so as
part part of our pipeline there's a
hallucination check that occurs to
ensure the accuracy of the information
that we're given we also have another
check in that process which is really
human copy review where we have humans
review the copy uh being generated after
our hallucination check just to make
sure that there's a level of accuracy
for the information that's going out
yeah that monitoring it's sort of like
how people's jobs change it instead of
like actually creating the product you
monitor the inputs and the outputs and
then work to optimize that's what I've
heard reporting on Amazon a bunch of
other companies and it seems like that's
what's going on in the marketing div
division on your generative AI products
correct very interesting okay let's talk
about uh operational efficiency so
you've used this to to automate support
in some ways can you tell us a little
bit about that yeah so we've recently
rolled out the our chatbot that we call
Penny to deal with level one level two
questions that our customers have in the
post booking process and you know this
was usually handled by humans or by you
know a third-party technology that quite
frankly didn't learn as quickly as the
large language models do and we've seen
some really really good operational
efficiency in in in customer questions
being handled in an automated way using
the chatbot um I can't share the the
exact results but we're seeing
significant operational efficiency and
savings by using this technology as part
of our post-booking flow and have plans
to expand it to all of our product lines
in the foreseeable future super cool all
right Marty brre is here with us he's
the chief technology officer of Price
Line talking all about how generative AI
is going to change TR uh change travel
and you know another way that generative
AI is really going to change almost
every organization that has a tech
function is code right code completion
testing um you're the chief technology
officer at the company I'll tell you
Marty one of the things that I found is
I speak to a lot of people who are
running teams of Engineers and they talk
a lot about how um GitHub co-pilot and
things of that nature are magical and
then I say can have you been able to
measure a productivity a concrete
productivity Improvement in your team
and they say well we're we're still
getting to that so you know I have a
chief technology officer of of a major
company sitting before me who's working
to put this into practice and I'm going
to put you uh on the spot with that
question and say how's that going for
you um and have you been able to see
those results in a concrete way so we've
just recently completed a pilot of a
product uh that played in the generative
AI space as it pertains to code
Automation and quite frankly we we had
mixed results we had certain pieces of
functionality that were very productive
productive for our uh our Engineers such
as quality you know automation of you
know you know unit tests or suggestions
in code so those were very positive on
the mixed bag results side uh one of the
Striking things that we saw was was a
developer ready to use the code gener
ated by the AI model as a commit into
production
and there was still some hesitancy
around doing that um and I think it's
really two factors number one really
trusting the technology to complete a
piece of code that you're responsible
for and commit it to production and two
it's getting developers used to working
in a pair programming model where the
other person you're working with is is a
large language model which some
developers have not wrapped their head
around yet so um you know we've seen
mixed results um and I think the jury
still out on the proliferation of using
this capability to fully automate what a
developer does so then talk a little bit
about you know the positive side of
those results and where you can really
see the productivity gains in the future
if if kind know if engine down on it I
think really the development of unit and
functional testing is a GameChanger for
say more about that because L large
engineering organizations commit a lot
of resource to you know quality
engineering which is writing these tests
in in the form of
headcount uh and if you're able to free
up those quality Engineers to actually
be physically writing code versus
writing unit and functional tests you're
going to see a significant
performance U and productivity gain in
your engineering organization by you
know deploying those those quality you
know or estat Engineers to more of a
pure programming model you know two is
time to Market if you're able to
automate you know these tests the time
to market for shipping code will only
increase and that's where I see the real
value in generative AI is unlocking a
lot of the maintenance and support
functions that developers have to do and
having them solely focus on building
great features for your product right it
seems like it just makes an organization
more inventive if you can minimize some
of those tasks that don't necessarily
lead to the creation of the product but
still have to be done
correct so Marty you were just at uh
Google Cloud next I think Priceline had
something like eight sessions at at that
uh conference um there's we we talk a
lot here about how the different tech
companies are battling against each
other right you have like the big
conversations about Google and open AI
Amazon um Microsoft and I think one of
the things that's interesting people
don't fully appreciate is that this
doesn't have to necessarily be Zero Sum
that you could take a little bit of each
technology provider and end up building
pretty robust uh offering you know for
whether you're doing something like code
automation internally or providing a
chatbot to Consumers or whatever it
might be creating marketing content so I
am kind of curious if you could tell us
a little bit about your experience at
Google Cloud next and how you evaluate
the um you know the current state of of
what these platforms are doing well I
think in general you know the the
startup
Community uh AWS open AI Microsoft and
Google they're all doing phenomenal jobs
in terms of producing meaningful
products that companies like Price Line
can can benefit from and as part of our
architecture we're using AWS open AI
Google and startup companies as part of
our technical architecture and they all
have a huge part to play and are playing
a huge part in our our e-commerce
platform as far as uh what I saw at
Google next I was I was stunned by the
level of innovations that are coming out
uh in the gener of AI space that are me
can have meaningful impacts to companies
like Price Line you know a great case
and point is I I saw a company on the
floor that automates the creation of
customer segments and audiences for for
marketing and audience creation for a
company like price line is super helpful
when you're targeting specific deals and
values for certain customers typically
that would take a a cycle of engineering
time now you can do it in a very
declarative way which is which is super
helpful uh we saw you know I saw another
startup company at googlle NEX that was
you know providing you know automated
ways in which you can manage pus across
your vertex pipeline you know that's
super helpful in terms of of cost
management so I'm I'm really you know
super excited about the level andur of
innovation that's happening in these
large companies but also in the startup
ecosystem which is just fantastic right
and talk about talk about audience
segmentation you know it's so
interesting as I started to learn a
little bit more about the type of
customers that come to Price Line you
know it isn't always necessarily like
level of wealth or geography or gender
or age but it is sort of like what
people see as a good deal um you know
whether that's they want the you know a
deal in a five-star hotel they want a
cheaper flight the more direct flight I
mean can you talk a little bit about how
um when the way that you think about
audiences and how they how they're you
know not necessarily the traditional
type of audiences that uh folks might
think about in traditional marketing
campaign and then how you can use
technology to really speak to each
one yeah I think you know my view on
this is that we want to drive onetoone
personalization uh personaliz has been
this term that you've bucketed a lot of
different people into like like large
segments of people get thrown into this
tier and this tier what we're really
striving for is really onetoone
personalized a Ono one personalized
product experience because the same
person that you could put into an
audience the next time they come back to
Price Line could be looking for
something completely different you know
a great case and point is you know I'm U
I have a family of four so sometimes
when I use the platform I'm just
traveling as an individual like for
example I'm going to go visit my son at
College you know in a couple weeks and
it's just me traveling and so the the
points of interests the kind of Hotel I
want to stay and the kind of flight I
would want to stay or want to take is
completely different than another trip
that I'm planning on uh during Christmas
which is going to Barcelona with my
family the complete context around that
trip changes I want to stay longer the
quality of hotel that I want to stay in
is probably higher um I may do some day
trips out of that so planning some
individual flights to and from Barcelona
is completely different so my experience
on the platform based on what I'm doing
and it at the time and what I'm
searching on should dynamically change
on on who I am so you know T you know I
I used to work at fizer and we had
always had this notion of creating
personalized medicine for the
individual and you know I apply the same
kind of principles as the price line is
we want to provide a very personalized
onetoone experience with the consumer
based on what their needs are at that
time and what they're planning on doing
on the platform and that given session
yeah it makes life interesting and I
guess kind of fun for chief technology
officer Price Line fun very fun well
thank you so much for sharing all this
great information with us today I think
I have a much better I know I have a
much better understanding of how deta of
AI could be used practically in the
travel setting and how it's not abstract
anymore it's starting to get rolled into
production so really appreciate that and
thank you so much for joining Marty
thank you for having me I appreciate it