Agents on the Canvas in tldraw — Steve Ruiz, tldraw

Channel: aiDotEngineer

Published at: 2026-05-01

YouTube video id: sPUjIBH5Cwg

Source: https://www.youtube.com/watch?v=sPUjIBH5Cwg

Hello. Hey, I'm going to I'm going to
kick off.
Uh
Sorry, we're starting a little bit late
here.
I am Steve Ruiz
from Teal Draw. Does anyone know Teal
Draw? Yes. Hey, all right, fans fans in
the room. Uh if you don't know Teal
Draw, Teal Draw is kind of kind of a
couple of things.
Teal Draw is a
online whiteboard. You can go to it.
It's free. It's really nice.
Um
I'm going to be using it for my slides.
Uh Teal Draw is also a startup. We're
based here in London. And Teal Draw is
also an SDK, something that you can use
to build other products. So, if you've
used uh Replit's new
um
uh agent canvas, then that's built with
our canvas. If you've used uh
um
Luma AI's new canvas, that's built with
our canvas.
If you've used Stitches' new canvas,
that's not built with our canvas.
But, if you go into this like kind of uh
um annotate mode,
this actually is our canvas. So, we're
we're in there somewhere. It's a Angular
app, you know. Uh
So, anyway, um
Teal Draw is again company makes a
whiteboard makes whiteboard SDK.
Part of the idea with the SDK is that
you can build cool things with uh with
the SDK,
which means it's part of my job to build
cool things with the SDK to prove it.
Um and a lot of those things recently
have have involved AI, right? Hackable
canvas runtime
built with React. In fact, the canvas is
also just React components. So,
component component component.
Um which means you can do some pretty
cool stuff.
Uh
first one uh I'm going to be at the
mercy of the demo gods here in the the
internet and and and other things. So,
uh bear with me here.
Um does anyone remember this app Make
Real? Maybe maybe this tweet uh about
Make Real. Did anyone Did anyone
remember seeing this tweet back in 2023
when the vision models came out? All
right, cool.
Uh this was one of the first projects to
kind of like break containment uh in um
in AI.
Don't We're going to go to my phone.
[ __ ] this.
Um
which which may may itself be a
disaster. Uh
So, the the basic idea with Make Real
was that you could um
use a canvas, draw this
and send that to a a model and and have
it make it into a working prototype,
which sounds very very quaint in 2026.
But, in 2023, it was all the rage
because there was no lovable, there was
no kind of vibe coding hadn't been
termed or coined as a as a term. Uh and
so, this was one of the first projects
where non-technical people could
make technical stuff without having to
code or to look at code.
And so again, we are at the mercy of the
the uh
several
[ __ ] various internet uh
internet gods. So, so we'll see. I'm
going to I'm going to let this cook for
a minute. Um but at the risk of leaking
my API keys, I will try and switch to a
faster model. Oh, no, it's it's coming.
All right. Maybe I just gave it
something hard to uh hard to work on.
We'll see. Hang on a second.
Basic idea,
something like that.
Uh yeah, there there There's my API
keys. I always got to rotate them every
time I uh do this demo.
Sadly. Uh we'll see how much
how much gets spent before I get done
with the talk.
Um but it is a very simple problem. It's
just like go go make this interactive.
This is uh this is not what I asked for,
but we'll see.
All right, good job. Cool. And this is a
this is a working working thing, which
is great. It's like a real little bit of
HTML.
But, you could also annotate on top of
it. And you could say like, "Hey, uh
actually, make this green."
And why don't we use these colors?
I'm going to make this red and uh
and and black. Um
Use these colors.
Right? And so, you're kind of
constructing a prompt here, even the
prompt that includes the old uh website.
And uh
there's not many apps that have actually
like kind of use this Only only now,
only like the the Google Stitch that I
mentioned before, uh
this idea of
Well, uh it didn't do anything that I
asked it for. Terrible demo. We're going
to we're going to move on. Things have
changed in the last couple of years.
Anyway,
uh there was another one in Teal Draw
computer, which I'm actually not going
to go into, but this was uh a couple of
like chains of prompts.
But, eventually we we were like, "Hey,
AI on the canvas actually might be
pretty cool. Maybe we should um have the
AI kind of
work as a collaborator, like work with
you on the canvas."
So, the first one of those was
pretty pretty straightforward idea where
you could say like, you know, draw a
cat.
You could do anything. You could say um
draw me a diagram, you know, finish my
slides, complete this graph that I'm
working on, things like that. Uh and
unlike maybe image models uh like
diffusion models and things like that,
it's not building an image. It is
using text structured outputs to to make
the same things that I could make,
right? So, I have I have tools. I have
like circles and shapes and like this.
Uh and it's funny to see how these
things have changed over Oh, it's sad.
Uh
but uh the uh
you know, the fun of of this is as a way
of exploring the model and and what the
model knows and how it can comport all
that stuff. Uh make the cat blow out the
candle.
Um
It's pretty cool.
You could also do something like uh draw
a mouse. So, multiple um multiple
prompts at the same time on different
parts. And so, even though I didn't tell
it what a candle was, and I certainly
like the application doesn't know what a
candle is,
uh and I'm not even sure that cats can
like blow.
But,
it's correctly interpreted that and kind
of incorporated into the into the
design. So,
uh
with incredible detail as well.
Still sad. Uh
so, the um
this was really fun because this is like
solving a lot of problems that
might not be obvious. Like like vision
models, when it comes to to structured
data, um
number one, there's much less vision
training data than there is for text. Uh
number two, a lot of that training data
like conflicts in ways that text does
not and other other types of things
don't. So, for example, the the Y axis
on a on a Cartesian graph,
as you go up, that number goes up,
right? So, 0 1 2 3 4 5. Uh on the web,
the Y axis goes up in this direction,
right? The top left corner is zero, your
top left corner here, but as you go
down, the Y goes up.
There's left, right, like there's your
left, there's stage left, there's all
sorts of uh uh things that that conflict
within language uh and within images.
So,
um
training the model to kind of behave
predictably and and produce things like
this is uh was really Well, I used
training uh prompt engineering the model
to
to do it was was really tricky. But,
this was fun um but we felt like it it
didn't go really far enough because it
was just one shotting, right?
We wanted to do an agent. So, this is
what Cursor looked like back in uh
in 2025 or something like that when I
did this.
Um draw a diagram uh of the life
cycle of a butterfly.
So, this put it into a kind of like
agentic loop, like you might have seen
seen elsewhere.
Um but I'm sure you you interact with
dozens of times a day by by now for this
crowd. Um where you have it produce an
output and then review the output and
kind of iterate until it thinks it's
done.
Um And we really tried to adhere to the
conventions at the time of um
you know, coding agents that were where
these agents um this agent loop was seen
most most often.
Where yeah, there's kind of like a lot
of sub features like rejection, you
know, seeing it's thinking, seeing
seeing how it works. Um
Great. And now we have the uh the
butterfly life cycle on the canvas.
Pretty cool.
However, this was still not really
enough because as as cool as this was,
it still felt like
I don't know. It felt like I was handing
my keyboard to some some other AI rather
than someone collaborating with me. Um
although this model has been used really
well in uh a lot of design apps that use
Teal Draw, uh things like Lovelace or
Magic Path, um
and in in education, especially where
you have this kind of tutor of like,
"Help me with my homework and help me
fill out my um you know,
uh
Oh gosh, let's see if I can do this on
the fly. Steve Ruiz
uh
class,
you know, age, whatever.
Um and you can kind of ask it to like uh
complete my D&D
character
sheet.
All right. And it'll it'll kind of pick
up what you're doing and fill out forms
and do it do do fun stuff like this.
Maybe I'll come back to that as it as it
kind of chugs along.
The what I really wanted is like to
bring the
the agent out of the sidebar and into
the the canvas itself. Um Oh, I'm a
fighter.
Nice. Nice. All right, I'll take that.
Uh and so, we did. And we did it with
fairies, which maybe
maybe you saw, maybe not. Um
these are like little little guys on the
on the canvas. You can kind of throw
them around.
Uh they don't like to be held for very
long. They'll start freaking out.
Uh yeah, okay. So, uh
and uh but you can do the same thing
like, you know, draw a a draw cat or
something like that. Now, putting the
agents on the canvas have a whole bunch
of interesting things. You can see the
state of the agent, right? These are
multiple agents that I'm kind of running
in in
coding terms, these would be multiple
terminal windows or something like that.
Or this would be in composer composer.
But you can kind of see what they're
doing in a way that uh uh hang on. I'm
zoomed way out.
I did all the sprites myself.
Um
and you know, not only can you kind of
see it's thinking, but you can see its
action. You can see where it where in
the project it's sort of like acting
relative to the other um other agents.
So,
and you know, these other agents can can
see each other what they're doing. So,
if I ask this one to draw um a hat on
the cat,
uh
and I draw this one uh draw the cat's
neck.
It looks like we missed the we missed
the neck.
Uh
they'll they'll they'll get to work,
right? And they're they're able to kind
of work with each other's stuff at the
same time.
Um but we could also ask them to work
together. So, if I grab all three of
these uh these fairies fairies, Helen,
and Joan, um
draw some more
animals.
Uh
one of them will be elected leader. So,
this one is is the leader. And it's
going to go scout kind of what's going
on on the canvas, and then it's going to
create a to-do list, and it's going to
delegate that to-do list to the other
agents, right?
Um this is all like
we we were doing this in like December
October of last year. Uh and
we're figuring this stuff out at the
same time that a lot of people were
figuring out agent orchestration. This
idea of like, okay, how do we give them
shared state? How do we uh you know,
have a leader follower? Like, how do we
manage the fact that these things are
essentially blind while they're working?
And prevent them from kind of
overlapping uh in terms of like what
they're doing.
And so, you can kind of see the the the
leader here isn't doing any of the work.
Uh
but it is going to kind of like observe
Oh, no, that's that's the leader. That's
the leader. Um
it's observing.
Uh
uh
and and judging and and establishing
whether this is like done or not and and
whether it's done correctly.
Um
Still not enough. Right? Fairies are
fun. If you want to play with this, by
the way, this is at fairies.tldraw.com.
Um
in the same way that Make Real was a
really good introduction to
um
to
just AI at all, right? Draw something,
click a button. Um fairies is a great
way to talk about like multiple agents
kind of working together.
Uh
and they can do real work. Let me try
and grab a
like this is a big description of like
an ebook or something like that. And if
I summon my fairies,
uh
uh make this make the make make the
wireframes
for this app.
Cool.
And I'll I'll just kind of let them let
them get to work while we keep talking.
Um
the
started 10 minutes later. I'm going to
I'm going to take another 5 minutes
before I jump. Uh the next step for this
one
is to to kind of give more access to the
canvas to the agents.
Um
and there's really
we started to kind of run into the
barriers of of safety. Like, what is
actually safe to
to do with our hackable thing for for
users? Um because we have a runtime API.
You can just code against it, right? And
AI is really good at coding. So, maybe
we could do some sandboxed, you know,
stuff. But no, because we need the DOM.
We need the kind of the browser as a way
to see what's going on. We need to be
able to generate screenshots, all this
stuff.
Um so, we decided to use our uh our
desktop app instead. So, I over the
holidays I threw together a um
an app that does an electron wrapper
that wrapped tldraw.
Uh
and I
opened a port essentially. I said, you
know, okay, Claude, like make a little
HTTP server and and open a put up an
endpoint. And anything that gets posted
to that endpoint,
uh
treat it as JavaScript and run it.
Which
is a terrible idea. Not a good Like,
don't don't don't do that on on your
app. However, for an an offline desktop
app that is file-based, like what's the
worst that could happen?
You could hurt yourself, I guess, but
you're not going to hurt the rest of me.
Uh and look at them look at them going.
It's building my little ebook reader.
It's fantastic. Thank you, fairies.
Doing the the
Uh so so, what does that What does that
give us though? Um
I'm going to I'm going to skip the demo
where uh as you can imagine, I could
I could say, hey, visualize this code.
Make a diagram. Cool. All right, I'm
going to change the the diagram. Update
the code to match the diagram.
Easy, right? You can have these kind of
like, let's pull up the the level of
abstraction that we want.
Um but the the more surprising stuff was
actually where I was like,
you know, okay, like check this out. Uh
I'm going to draw a little user
interface or whatever, right?
And I want this to be uh leg length.
And I want this to be a t-shirt
color.
Uh
and even though tldraw doesn't really
have the ability to we don't have like
primitives for on hover, on click. It's
not like
it's not a fully thing.
This thing can write code against
the editor.
So, like uh make it interactive.
And and we'll see we'll see where it
gets to.
Um
so far,
the results on this have been like
really really cool in ways that are
super strange and disturbing. Uh
because like asking like
the AIs are like, sure, let's do some
scripts script injection, right? Like,
that's the way that it documents itself
is like, this is how you you should do
this. Um
it has no qualms at all, by the way,
changing stuff that's on your desktop on
your computer. If you've ever wanted to
like, for example,
like we uh
one of our team, Max, was like, you know
what? I don't like podcasts in my
Spotify. I want to get rid of podcasts
in my Spotify app. Claude, can you just
do that? And it's like, sure. Let me go
through the minified code of the bundle
of the thing. Let me just rip and tear.
Uh
and it's happy to do it. It makes them
happy. Uh they like it.
Um oh, what the [ __ ] was that?
Um
I don't even know what you did. You made
a an HTML What the
What? Like, it it created a new HTML
site on this?
And this is the pointer? It's not even a
slider. No, I I want to I wanted in in
the tldraw. All right. Yeah, we don't we
we love it.
It's blinking as well. I don't know if
you caught that.
Come on, do it. Yeah, there we go.
Let's see if it can Come on, let's go.
Um
So, yeah, like there's really like uh
no limit to what what what it can do
with the desktop app, and it's happy to
do it in a way that um
I can almost tell that it it it uh would
love to do this to websites. Like, it
would love like just Let me just get my
get my claws in there.
Um all right, come on, come on.
Still not working. We're going to we're
going to
we're we're going to we're going to go.
Set up the interactivity. Come on. Um
this is going to be really fun. I think
we're going to just release this. Uh
we're
the um I mean, the it is released, but
the uh
the notion that you can take
like I love local-first apps. I love
file-over-app. I love There's all these
ideas that up to now have kind of been
curiosities and
uh
almost like
Hang on.
Oh, come on.
Oh, that's such a such a disappointment.
We're going to have to catch me later.
I'll make it work. Uh
but now actually the like the idea of a
local file-based thing that is is able
to expose itself to to um
to to Claude and agents like locally in
order to to essentially script inject
kind of motivates a lot of that that
stuff which previously was idealistic
into like, well, that's the only way
that you could do this.
If you really want to maximize the
agency in order to maximize what it can
do
and take the risk uh and take on that
risk,
then you you kind of just need to hand
that to the user and say, good luck.
Um I think open Claude does this pretty
well. But like, this these are sharp
tools.
Have fun, you know? Anyway,
uh that is my agents on the canvas talk.
Uh
work continues. If you want to play with
the fairies, I highly recommend it cuz
it's super fun, and you will find things
that surprise you
that have surprised me.
Uh
they have IRC as well. Let me see. Yeah,
anyway.
Uh
and if you want to follow along with
tldraw,
we are on Twitter X at tldraw, and then
I'm at Steve Ruiz okay. I post a lot
about this stuff. So,
uh thank you for coming.
Cheers.