Excalidraw: AI and Human Whiteboarding Partnership - Christopher Chedeau

Channel: aiDotEngineer

Published at: 2025-07-21

YouTube video id: aopgVJBQC0o

Source: https://www.youtube.com/watch?v=aopgVJBQC0o

[Music]
Thank you so much for the intro. I'm so
excited to be here uh talking about like
figure out like how do we like AI and
human like work in the world of white
bowling and I built excro and if you've
don't know about it like you'll see like
many thing about it and one of the
expectation you probably have uh about
speaker at the AI engineer conference is
that I talk about AI on every single
sentence for the entire talk. So I'm
just going to give you a warning. I'm
only going to do it for the second half
of the talk, but the first half is going
to be a good setup and I you should like
pay close attention because it's going
to be really interesting content. So,
let's get started. So, we are five years
ago. Uh I'm working at Facebook January
1st and as you if you've been at any
like big company there's like the
performance review process and so we you
have to uh give feedback around like all
the people you've been working with like
how good or bad you've been doing and
same for your manager and one thing that
uh like I'm probably not unique in this
room but there's this little devil in me
that is like nope you're not going to do
it until the very very last minute and
this is called procrastination. ation
and I'm really bad at this and I've
tried like many techniques over the
years and the one uh at that time that I
worked on is what I call active
procrastination. So I know I'm not going
to be able to write the performance
reviews until like the last minute. So
may as well do something actually useful
in the meantime otherwise I would be
like doom scrolling on my phone
and not be productive. And so what I
ended up doing is uh I wanted to write a
blog post for a long time. And so I was
like, okay, I'm going to write a blog
post. And so I used a tool called Zibl
in order to uh draw draw like those kind
of like handwritten like illustrations.
And unfortunately at the time the
website ZLur was down and I was like no
come on. I've wanted to procra to not
procrastinate and everything and then
it's still down. And so what did I do?
I'm like, "Okay, we're going to go like
one level deeper. What would it take to
actually reimplement ZL? How hard would
it be?" And so I opened the code sandbox
and I like drew like wrote some code to
use rough js and now hey we I got a demo
and the thing I didn't expect is I
posted about it on Twitter and like it
exploded like 200 likes on this first
tweet and like so many people like
replied and commented and were excited
about it and I was like oh this is so
cool. And so this was the beginning of
excal. Now one of the thing is if I were
to do this again uh in the future like
right now I would not like start writing
code myself. I would probably do use any
of the coding agent that we've seen. And
the good thing is I'm not the only one.
So if you look at two weeks ago uh
during the anthropic keynote uh when
they were wanted to demo code uh AI
agent they actually used uh excal for
this and they actually had the agent
reimplement uh the most popular request
which is a table layout. Unfortunately
they haven't sent a PR yet. So if you
anybody's working at code here we
welcome pull requests. Anyway, so one
thing you may have noticed is I've said
AI multiple times. So I knew you were
not going to last the entire half of the
talk without it. So now let's get back
to the nonAI part of this. So uh this
was uh like in the story we're like in
January 1st 2020 and three months later
you probably all know what happened.
COVID and so everybody had to go home
and now had to figure out like how are
you working from home? Like what's life
like? And a lot of people, one of the
big thing that they try to figure out is
whiteboarding. And so when you're in the
office, you have a physical whiteboard
and you can draw stuff and you can
collaborate with people. But when you're
at home, you don't have any of this. And
I was very confused because uh a lot of
people like started coming to me around
like, oh actually Excal is a really good
uh whiteboarding uh solution. And I was
like very confused because I built XK
Draw for drawing uh illustration for my
blog post and why what does it have to
do with whiteboarding? And then I
started uh being interested around like
okay what are the other people doing
whiteboarding like other tools doing and
one thing I quickly realized is that all
of the tools at the time basically took
the physical limitation of whiteboarding
and map them one to one uh with the
online versions. So for example uh a
whiteboard like a physical whiteboard
has a fixed size and so what everybody
did they created a fixed size. Hey
create a new whiteboard and then if you
run out of space then like you're out of
luck. Now with X draw you can just like
scroll around and like write start
writing somewhere else and just works.
Then the next one is uh drawing text.
And so on a physical whiteboard you have
to trace every single letter and on the
mouse it's actually very very hard. And
so on x I was like hey like we have a
text box like you can select uh and this
is fine and when I did the dry run for
this talk like the people were telling
me like there's no way like even MS
Paint in like 30 years ago had like
keyboard support but this was the state
of of the arts at the time and one thing
to uh to remember is nobody like thought
about online whiteboarding at the time.
So like all of the projects were pretty
small and like not very used and uh now
all of a sudden there's been a huge
influx. So this is why it was so bad and
then the like the list keeps going on.
So if you want to start erasing they
would actually have you like erase every
single pixel but on schedule you can
select and delete and just just works.
And then uh like I don't know if you've
seen like the like the reals Tik Tok
video where like people like draw like
perfect circles like this uh on a
blackboard and this is like what you had
to do uh for like doing this online but
with X you can just like select a shape
like a circle and like you just draw it
and it just works fine. And finally
there are things you cannot do uh on a
physical whiteboard. So for example like
copy pasting is like just doesn't work
but if you're working online like this
is the baseline expectation that you're
able to like select and copy paste and
so this is like all of this like really
was a big breakthrough in my mind around
the idea of uh the physical versus
virtual whiteboard and this is I feel
like the same place and time we're in
with AI integrations right now. So in
practice what a lot of the AI
integrations today they're basically
taking like whatever we're doing right
now and then like sprinkling AI or
reimplementing with with AI and the end
result is like pretty bad because like
hey you don't know how it works but the
thing is unlike draw and the online like
where we had the experience right now we
AI like nobody has experience and so
this is why I'm super excited about this
conference where like all of the people
like trying things out are actually
sharing the lessons to the with with
each other and so hopefully like as we
keep doing these conferences now we're
going to bring the state of the world
with AI to be much more uh much better.
So now I talked a lot about like Excel
draw the early days and everything and
so a lot of people have been asking us
like how do you integrate AI with Excel
Draw and we've tried a bunch of things.
So now for the second part I'm going to
walk through what are the things we've
tried, what are the things that have
worked, what are the things that haven't
worked. So the first one is actually not
us but Gore cam who's I think is in the
room somewhere working at FAL AI and so
he discovered this uh really cool
machine learning model that takes uh an
Excel diagram uh with like a palm tree
or like a sun and then he's able to
gener uh to generate real time uh live
picture like beach of the image uh uh in
uh in reaction to this. And so this is
like was mind-blowing to me where like
oh now like machines can actually
generate uh images and the model is
working really well and like the I think
the demo is still working. So if you uh
if you want like you should try it but
one of the issue is when we started like
integrating with Excal we realized that
people don't actually draw realistic
images on Excel draw. And so this is one
like the first like key lesson that we
found is that just because there's a
model that exists doesn't mean that it
should be within your product. And I
know right now like there's a huge
pressure for all of us to like just add
whatever kind of AI into our product
that uh like in order to be like AI
first and whatever. But I really like
want you to think about like what are
the actual AI integration that actually
makes sense rather than like hey I just
put in AI and that's it. And in practice
if you do that uh then actually like the
whole product experience is going to be
like way worse than if you didn't have
any AI at all. So this was like the
first uh learning. Now people do not
draw realistic images on Excel draw but
they do draw uh diagrams. And so now the
question is okay so L&Ms are magical and
like they're going to generate diagrams.
So let's try it. And so we tried and
that didn't work out. And so this is an
illustration, but in practice it was
even worse. Like there was not there was
no resemblance of a house or anything.
It was just like bad. And this is where
like we started realizing that like just
doing this wasn't going to cut it. So we
need to like be like a bit better. But
one thing is yesterday I was chatting uh
with somebody at working at Gemini and
he showed me a demo of like Gemini
actually outputting XO files. So my talk
is already outdated and uh we probably
need to get back and like start coding
on this but anyway let's continue. So
the thing though that we found out is uh
LLMs were actually able to generate
something called mages. And so the the
name of the product is super weird but
in practice it's a way to describe a
diagram. And so in this case you're just
um writing flowchart LR a AI to B
engineer and then it draws uh this
specific diagram. And so LNMs were able
to like realistically generate uh this
kind of uh like mermaid JS syntax. And
this was really exciting because now
what we are able to do is to like encode
the knowledge of uh like the diagram and
mermaid supports like many different
types of diagram and so now we're able
to like encode all of this and like all
of the prompts can actually generate the
diagrams. Now the challenge is like this
output SVG or canvas and not XK draw
files. So what do we do? So in practice
I can check got to work and so she
implemented an XK draw back end for
mermaid JS and now we finally have our
very first very very successful
integration of AI within XK draw. So now
what you can do is to like uh set a
prompt and like put in anything that you
want. So in this case a flowchart start
with AI and engineer but we've seen
people like draw like or chart diagram
or TCP or the network request kind of
diagram or architecture diagram uh where
you can see like classes and inheritance
and those kind of things and then you
get uh a result which is an actual excal
uh component that you can play with and
one of the thing that is the like the
title of my talk is the human and AI
partnership one of The thing that you've
probably been very frustrated is like
you start and you generate something
that like looks 90% like what you want.
But then you're like, hey, this is an
image. I can do anything with it. And
this is like the magic of this
integration with Excal is now this is
just Excel Draw and you can use Excel
Draw to like make it however you want.
And so, for example, you can get to
something like this where you start
modifying the layout, start moving
things around, you start adding colors
and you really make it to where and to
what you want to be. And so I think this
is the the mindset that we're going to
see more and more with successful
integrations is no longer going to be
like one shot. Hey, I'm asking the eye
something and the eye give me like
something perfect, but how do you get
something that's close enough and then
you as a human can start modifying? And
so right now this is like a just one way
uh like one run trip operation. But I
foresee a future where like now you're
going to be able to go back to the say
hey modify this and then go to the much
further. We don't have this right now
but uh this is what I think the future
is going to look like. So now this is
one integration. So I'm going to go
through a few other AI integration that
we've been working on within Excal. So
the next one is uh one of the things you
can do with the SAS offering of Excal is
managing file creating a bunch of
different files and syncing in the
cloud. And as if you've been using this,
most of your like documents are going to
look like untitled one, untitled two,
untitled three, untitled four. And one
of the things that LM can actually
really good at is generating names for
these documents. And so now whenever
you're creating a document, we wait
until you stop working on this so we
actually have content. And then we're
going to prompt you, hey, these are like
all of the uh name that we think would
match. And then you just click and then
you're done. And so this way like it
actually removes the burden on you to do
it and AIs are really good at this. Next
one is while uh people are not using
escal to generate like goodlooking like
painting kind of things they people tend
to like want to have like a lot of
infographics a lot of illustrations
within their like within their slides or
like their diagrams. And one thing that
we are doing which is in uh like alpha
like closed right now but we're going to
open source and to uh ship in a few
weeks is the ability to generate uh
illustrations. And so we are really
hyped about this because I think it's
going to help a lot of people. And
finally, the thing that like kills me
the most. I'm using Excel every day and
I usually want to integrate logos within
my diagram or my slide, all those kind
of things. And I cannot find a reliable
way to remove the background from all of
those basic logos. So I just took a
screenshot of the AI engineer warfare uh
from the website and I'm like, hey,
please like how do I remove the
background? And the thing that I wanted
to do is to challenge all of you like
you're all AI engineers and probably
some someone from all of you can create
a model that removes the background for
logos and if you're able to make it work
within the browser send it to me and
we'll make it uh happen and we'll like
ship it within Excal. So I'm really
excited that some of you are going to be
able to help us with this and help all
of the people using Excal. So this is uh
the end of my talk and I have a few
conclusions for you. So the first one is
right now as I mentioned we're in the
like physical whiteboard to virtual uh
whiteboard like kind of moments uh
happening within like all of the product
but specifically like for the design
products and now we need to figure out
like what is the AI native way to
implement those interactions and so
really excited about this conference to
like start showing like all of the
potential. The second one is please
please please res resist the urge to
actually ship all of the random models
that exist within your app and really
only ship the one that like you feel
like are actually adding to the product
and helping your uh customers uh getting
like doing what they actually want to
do. And the last one is LLM's love
structure. So if you're able to take
your problem domain problem space and
find a way to represent it in some kind
of domain specific language now it's
going to be most like most likely uh
going to work way better to like have
the LM target this domain specific
language and then you take this language
and without AI you actually output it to
your own system. So this is the end of
my talk. I'm really excited uh that we
have this event and conference and I'm
also excited that like Steve uh is going
to be talking next about TLOR. Like
we've been working together for a long
time and like he's probably going to
blow your mind with all his examples. So
thank you
[Applause]
[Music]