DSPy: The End of Prompt Engineering - Kevin Madura, AlixPartners

Channel: aiDotEngineer

Published at: 2026-01-08

YouTube video id: -cKUW6n8hBU

Source: https://www.youtube.com/watch?v=-cKUW6n8hBU

[music]
Thanks everybody for uh for joining. I'm
here to talk to you today about DSPI.
Um, and feel free to jump in with
questions or anything throughout the
talk. It's, you know, I don't sp I don't
plan on spending the full hour and a
half or so. I know it's the last session
of the day. So, um, keep it casual. Feel
free to jump in. I'll start with a
little bit of background. Don't want to
go through too many slides. I'm
technically a consultant, so I have to
do some slides, but we will dive into
the code for the the latter half. And
there's a GitHub repo that you can
download to to follow along and play
around with it on your own.
Um, so how many people here have heard
of DSPI?
Almost everyone. That's awesome. How
many people have actually used it kind
of day-to-day in production or anything
like that? Three. Okay, good. So
hopefully we can convert some more of
you today. Um, so high level DSPI, this
is straight from the website. Um, it's a
declarative framework for how you can
build modular software. And most
important for someone like myself, I'm
not necessarily an engineer that is
writing code all day, every day. As I
mentioned before, I'm a more of a
technical consultant. So, I run across a
variety of different problems. Could be
um an investigation for a law firm. It
could be helping a company understand
how to improve their processes, how to
deploy AI internally. Maybe we need to
look through 10 10,000 contracts to
identify a particular clause um or or
paragraph. And so DSPI has been a really
nice way for me personally and my team
to iterate really really quickly on
building these applications.
Most importantly building programs. It's
not um it's not kind of iterating with
prompts and tweaking things back and
forth. It is building a a proper Python
program and and DSP is a really good way
for you to do that.
So I mentioned before there's a repo
online if you want to download it now
and kind of just get everything set up.
I'll put this on the screen later on.
Um, but if you want to go here, just
kind of download some of the code. It uh
it's been put together over the past
couple days. So, it's not going to be
perfect production level code. It's much
more of utilities and little things here
and there to just come and kind of
demonstrate the usefulness, demonstrate
the point of of what we're talking about
today in that and we'll walk through all
of these these different use cases. So
um sentiment classifier going through a
PDF some multimodal work uh a very very
simple web research agent detecting
boundaries of a PDF document you'll see
how to summarize basically arbitrary
length text and then go into an
optimizer uh with Jeepo
but before we do that just again kind of
level set the biggest thing for me
personally DSP is a really nice way to
decompose your logic into a program that
treats LLMs as a first class citizen. So
at the end of the day, you're
fundamentally just calling a function
that under the hood just happens to be
an LLM and DSPI gives you a really nice
intuitive easy way to do that with some
guarantees about the input and output
types. So of course there are structured
outputs, of course there are other ways
to do this, Pyantic [snorts]
and others. Um, but DSPI has a set of
primitives that when you put it all
together allows you to build a cohesive
modular piece of software that you then
happen to be able to optimize. We'll get
into that uh in a minute.
So, just a few reasons of why I'm such
an advocate. It sit at it sits at this
really nice level of abstraction. So,
it's I I would say it doesn't get in
your way as much as a lang chain. And
that's not a knock-on lang chain. It's
just a different kind of paradigm in the
way that DSPI is is structured. Um, and
allows you to focus on things that
actually matter. So you're not writing
choices zero messages content. You're
not you're not doing string parser.
You're not doing a bunch of stuff under
the hood. You're just declaring your
intent of how you want the program to
operate, what you want your inputs and
outputs to be.
Because of this, it allows you to create
computer programs. As I mentioned
before, not just tweaking strings and
sending them back and forth. You are
building a program first. It just
happens to also use LLMs. And really the
the most kind of important part of this
is that and Omar the KB the uh the
founder of this or the the original
developer of it had this really good
podcast with A16Z. I think it came out
just like two or three days ago. But it
he put it a really nice way. He said
it's a it's built with a systems mindset
and it's really about how you're
encoding or expressing your intent of
what you want to do most importantly in
a way that's transferable. So the the
design of your system, I would imagine,
or your program isn't going to move
necessarily as quickly as maybe the
model capabilities are under the hood.
when we see new releases almost every
single day, different capabilities,
better models and so DSPI allows you to
structure it in a way that retains the
control flow uh retains the intent of
your system, your program um while
allowing you to bounce from model to
model to the extent that you want to or
need to.
Convenience comes for free. There's no
parsing, JSON, things like that. It
again, it sits at a nice level of
abstraction where you can still
understand what's going on under the
hood. If you want to, you can go in and
tweak things, but it allows you to to
kind of focus on just what you want to
do while retaining the level of
precision that you that I think most of
us would like to have in and kind of
building your programs. Um, [snorts]
as mentioned, it's robust to kind of
model and paradigm shifts. So, you can
again keep the logic of your program. Um
but it but keep that those LLMs infused
in uh basically in line. Now that being
said, you know, there are absolutely
other great libraries out there.
Pedantic AI, Langchain, there's many
many others that allow you to do similar
things. Agno is another one. Um this is
just one perspective and um it may not
be perfect for your use case. For me, it
took me a little bit to kind of gro how
DSPI works and you'll see why that is in
a minute. Um, so I would just recommend
kind of have an have an open mind, play
with it. Um, run the code, tweak the
code, do whatever you need to do. Um,
and just see how it might work, might
work for you. And really, this talk is
more about ways that I found it useful.
It's not a dissertation on the ins and
outs of every nook and cranny of DSPI.
It's more of, you know, I've run into
these problems myself now. I naturally
run to DSPI to solve them. And this is
kind of why. And the hope is that you
can extrapolate some of this to your own
use cases. So we we'll go through
everything uh fairly quickly here, but
the core concepts of DSPI really comes
down to arguably five or these six that
you see on the screen here. So we'll go
into each of these in more detail, but
high level signatures
specify what you want the L what
basically what you want your function
call to do. This is when you specify
your inputs, your outputs. Inputs and
outputs can both be typed. Um, and you
defer the rest of the basically the how
the implementation of it to the LLM. And
we'll see how we how that all kind of
comes together uh in a minute. Modules
themselves are ways to logically
structure your program. They're based
off of signatures. So, a module can have
one or more signatures embedded within
it in addition to uh additional logic.
and it's based off of um pietorrch and
some of the in terms of like the
methodology for how it's structured and
you'll you'll see how that uh comes to
be in a minute. Tools we're all familiar
with tools MCP and others and really
tools fundamentally as DSPI looks at
them are just Python functions. So it's
just a way for you to very easily expose
Python functions to the LLM within the
DSP kind of ecosystem if you will. um
adapters
live in between your signature and the
LLM call itself. I mean, as we all know,
prompts are ultimately just strings of
text that are sent to the LLM.
Signatures are a way for you to express
your intent at a at a higher level. And
so, adapters are the things that sit in
between those two. So, it's how you
translate your inputs and outputs into a
format basically explodes out from your
initial signature into a format that is
ultimately the prompt that is sent to
the LLM. And so, you know, there's some
debate or some research on if certain
models perform better with XML as an
example or BAML or JSON or others. And
so adapters give you a nice easy
abstraction to to basically mix and
match those at at will as you want.
Optimizers um are
the most interesting and for whatever
reason the most controversial part of
DSP. That's kind of the first thing that
people think of or at least when they
hear of DSP they think optimizers. We'll
see a quote in a minute. It's not
optimizers first. It is just a nice
added benefit and a nice capability that
DSPI offers in addition to the ability
to structure your program with the
signatures and modules and everything
else. Um, and metrics are used in tandem
with optimizers that that basically
defines how you measure success in your
in uh your DSPI program. So the
optimizers use the metrics to determine
if it's finding the right path if you
will.
So signature as I mentioned before it's
how you express your intent your
declarative intent can be super simple
strings and this is the weirdest part
for me initially but is one of the most
powerful parts uh of it now or it can be
more complicated class-based classbased
objects if you've used pyantic it that's
basically what what it runs on under the
hood.
So this is an example of one of the
class-based signatures. Again, it it's
basically just a pyantic object.
What's super interesting about this is
that
the
the names of the fields themselves act
almost as like mini prompts. It's part
of the prompt itself. And you'll see how
this comes to life in a minute. But
what's ultimately passed to the model
from something like this is it will say
okay your inputs are going to be a
parameter called text and it's based off
of the name of the that particular
parameter in this class. And so these
things are actually passed through. And
so it's it's very important uh to be
able to name your parameters in a way
that is intuitive for the model to be
able to pick it up. Um, and you can add
some additional context or what have you
in the description field here. So most
of this, if not all of this, yes, it is
proper, you know, typed Python code, but
it's also it also serves almost as a
prompt ultimately that feeds into the
model. Um, and that's basically
translated through the use of adapters.
Um, and so just to highlight here like
these, it's the ones that are a little
bit darker and bold, you know, those are
the things that are effectively part of
the prompt. uh that's been sent in and
you'll see kind of how DSPI works with
all this and formats it in a way that
again allow you to just worry about what
you want. Worry about constructing your
signature instead of figuring out how
best to word something in the prompt. Go
>> ahead.
I have a really good prompt.
>> Sure. Then I don't want this thing.
>> That's exactly right.
>> Sure.
>> So the the question for folks online is
what if I already have a great prompt?
I've done all this work. I'm a I'm a
amazing prompt engineer. I don't want my
job to go away or whatever. Um, yes. So,
you can absolutely start with a custom
prompt or something that you have
demonstrated works really well. And
you're exactly right that's that can be
done in the dock string itself. There's
there's some other methods in order uh
for you to inject basically system
instructions or add additional things at
certain parts of the ultimate prompt and
or of course you can just inject it in
the in the final string anyway. I mean
it's just you know a string that is
constructed by VSPI. So um absolutely
this doesn't necessarily prevent you it
does does not prevent you from adding in
some super prompt that you already have.
Absolutely. Um and to your point it is
it can serve as a nice starting point
from which to build the rest of the
system.
Here's a shorthand version of the same
exact thing which to me the first time I
saw this so this was like baffling to
me. Um, but it it that's exactly how it
works is that you're basically again
kind of deferring the implementation or
the logic or what have you to DSPI and
the model to basically figure out what
you want to do. So in this case, if I
want a super super simple text uh
sentiment classifier, this is basically
all you need. You're just saying, okay,
I'm going to give you text as an input.
I want the sentiment as an integer as
the output. Now you probably want to
specify some additional instructions to
say okay your sentiment you know a lower
number means negative you know a higher
number is more positive sentiment etc.
But it just gives you a nice kind of
easy way to to kind of scaffold these
things out in a way that you don't have
to worry about like you know creating
this whole prompt from hand. It's like
okay I just want to see how this works
and then if it works then I can add the
additional instructions then I can
create a module out of it or you know
whatever it might be. It's these
shorthand
or it is this shorthand that makes
experimentation and iteration incredibly
quick.
So modules it's that base abstraction
layer for DSPI programs. There are a
bunch of modules that are built in and
these are a collection of kind of
prompting techniques if you will and you
can always create your own module. So to
the question before, if you have
something that you know works really
well, sure yeah, put it in the module.
That's now the kind of the base
assumption, the base module that others
can build off of. And all of DSPI is
meant to be composable, optimizable, and
when you deconstruct your business logic
or whatever you're trying to achieve by
using these different primitives, it all
it's intended to kind of fit together
and flow together. Um, and we'll get to
optimizers in a minute, but at least for
me and my team's experience, just being
able to logically separate the different
components of a program, but basically
inlining uh LLM calls has been
incredibly powerful for us. And it's
just an added benefit that at the end of
the day, because we're just kind of in
the DSPI paradigm, we happen to also be
able to optimize it at the end of the
day. Uh, so it comes with a bunch of
standard ones built in. I I don't use
some of these bottom ones as much,
although it's they're super interesting.
Um the base one at the top there is just
DSpi.predict.
That's literally just, you know, an LM
call. That's just uh a vanilla call.
chain of thought uh probably isn't isn't
as relevant anymore these days because
models have kind of ironed those out but
um it is a good example of the types of
um kind of prompting techniques that can
be built into some of these modules um
and basically all this does is add um
some some of the uh strings from
literature to say okay let's think step
by step or whatever that might be same
thing for react and codeact react is
basically the way that you expose the
tools to the model. So, it's wrapping
and doing some things under the hood
with um basically taking your signatures
and uh it's injecting the Python
functions that you've given it as tools
and basically React is how you do tool
calling in DSP.
Program with thought is uh is pretty
cool. It kind of forces the model to
think in code and then we'll return the
result. Um, and you can give it a, it
comes with a Python interpreter built
in, but you can give it some custom one,
some type of custom harness if you
wanted to. Um, I haven't played with
that one too too much, but it is super
interesting. If you have like a highly
technical problem or workflow or
something like that where you want the
model to inject reasoning in code at
certain parts of your pipeline, that's
that's kind of an really easy way to do
it. And then some of these other ones
are basically just different
methodologies for comparing outputs or
running things in parallel.
So here's what one looks like. Again,
it's it's fairly simple. It's, you know,
it is a Python class at the end of the
day. Um, and so you do some initial
initialization up top. In this case,
you're seeing the uh
uh the shorthand signature up there. So,
I'm this module uh just to give you some
context is an excerpt from um one of the
the Python um files that's in the repo
is basically taking in a bunch of time
entries and making sure that they adhere
to certain standards, making sure that
things are capitalized properly or that
there are periods at the end of the
sentences or whatever it might be.
that's from a real client use case where
they had hundreds of thousands of time
entries and they needed to make sure
that they all adhere to the same format.
This was one way to to kind of do that
very elegantly, at least in my opinion,
was taking up top you can define the the
signature. It's adding the some
additional instructions that were
defined elsewhere and then saying for
this module the the change tense um call
is going to be just a vanilla predict
call. And then when you actually call
the module, you enter into the forward
function which you can inter basically
intersperse the LLM call which would be
the first one and then do some kind of
hard-coded business logic beneath it.
Uh tools as I mentioned before these are
just vanilla kind of Python functions.
It's the DSP's tool interface. So under
the hood, DSPI uses light LLM. And so
there needs to be some kind of coupling
between the two, but fundamentally um
any type of tool that would that you
would use elsewhere, you can also use in
in DSPI. And this is probably obvious to
most of you, but here's just an example.
You have two functions, get weather,
search web. You include that with a
signature. So in this case, I'm saying
the signature is I'm going to give you a
question. please give me an answer. I'm
not even specifying the types. It's just
going to infer what that means. Uh I'm
giving it the get weather and the search
web tools and I'm saying, okay, do your
thing, but only go five rounds just so
it doesn't spin off and do something
crazy. And then a call here is literally
just calling the React agent that I
created above with the question, what's
the weather like in Tokyo? We'll see an
example of this in the code session, but
basically what this would do is give the
model the prompt, the tools, and let it
do its thing.
So adapters, before I cover this a
little bit, they're basically prompt
formatterers, if you will. So the
description from the docs probably says
it best. It's you know it takes your
signature the inputs other attributes
and it converts them into some type of
message format that you have specified
or that the adapter has specified and so
as an example the JSON adapter taking
say a pyantic object that we defined
before this is the actual prompt that's
sent into the LLM and so you can see the
input fields so this would have been
defined as okay clinical note type
string patient info as a patient details
object object which which would have
been defined elsewhere and then this is
the definition of the patient info. It's
basically a JSON dump of that pantic
object. Go ahead.
>> So this idea there's like a base adapter
default that's good for most cases and
this is if you want to tweak that to do
something more specific.
>> That's right.
>> Yeah. The question was if if there's a
base adapter and would this be an
example of where you want to do
something specific? Answer is yes. So um
it's a guy pashant who is um I have his
Twitter at the end of this presentation
but he's been great. [clears throat] He
did some testing comparing the JSON
adapter with the BAML adapter. Um and
you can see just intuitively even even
for us humans the way that this is
formatted is a little bit more
intuitive. It's probably more token
efficient too just considering like if
you look at the messy JSON that's here
versus the I guess slightly better
formatted BAML that's here. um can
actually improve performance by you know
five to 10 percent depending on your use
case. So it's a good example of how you
can format things differently. The the
rest of the program wouldn't have
changed at all. You just specify the
BAML adapter and it totally changes how
the information is presented under the
hood to the LLM
multimodality. I mean this obviously is
more at the model level but DSPI
supports multiple modalities by default.
So images, audio, some others. Um, and
the same type of thing, you kind of just
feed it in as part of your signature and
then you can get some very nice clean
output. This allows you to work with
them very, very, very easily, very
quickly. And for those uh, eagle-eyed
participants, you can see the first uh,
lineup there is attachments. It's
probably a lesserk known library.
Another guy on Twitter is awesome. Uh,
Maxim, I think it is. uh he created this
library that just is basically a
catch-all for working with different
types of files and converting them into
a format that's super easy to use with
LLMs. Um he's a big DSPI fan as well. So
he made basically an adapter that's
specific to this. But that's all it
takes to pull in images, PDFs, whatever
it might be. You'll see some examples of
that and it just makes at least has made
my life super super easy.
Here's another example of the same sort
of thing. So this is a PDF of a form 4
form. So, you know, public SEC form from
Nvidia.
Um, up top I'm just giving it the link.
I'm saying, okay, attachments, do your
thing. Pull it down, create images,
whatever you're going to do. I don't
need to worry about it. I don't care
about it. This is super simple rag, but
basically, okay, I want to do rag over
this document. I'm going to give you a
question. I'm going to give you the
document and I want the answer. Um, and
you can see how simple that is.
Literally just feeding in the document.
How many shares were sold? Interestingly
here, I'm not sure if it's super easy to
see, but you actually have two
transactions
here. So, it's going to have to do some
math likely under the hood. And you can
see here the thinking and the the
ultimate answer. Go ahead.
>> Is it on the rag step? Is it creating a
vector store of some kind or creating
embeddings and searching over those? Is
there a bunch going on in the background
there or what?
>> This is poor man's rack. I should have
clarified. This is this is literally
just pulling in the document images and
I think attachments will do some basic
OCR under the hood. Um, but it doesn't
do anything other than that. That's it.
All we're feeding in here, the the
actual document object that's being fed
in, yeah, is literally just the text
that's been OCRD. the images, the model
does the rest.
[sighs] All right, so optimizers uh
let's see how we're doing. Okay. Um
optimizers are super powerful, super
interesting concept. It's been some
research um that argues I think that
it's just as performant if not in cert
in certain situations more performant
than fine-tuning would be for certain
models for certain situations. there's
all this research about in context
learning and such. And so whether you
want to go fine-tune and do all of that,
nothing stops you. But I would recommend
at least trying this first to see how
far you can get without having to set up
a bunch of infrastructure and, you know,
go through all of that. See how the
optimizers work. Um, but fundamentally
what it allows you to do is DSPI gives
you the primitives that you need and the
organization you need to be able to
measure and then quantitatively improve
that performance. And I mentioned
transferability before. This the
transferability is enabled arguably
through the use of optimizers because if
you can get okay I want to I have the
classification task works really well
with 41 but maybe it's a little bit
costly because I have to run it a
million times a day. Can I try it with
41 nano? Okay, maybe it's at 70%
whatever it might be. But I run the
optimizer on 41 nano and I can get the
performance back up to maybe 87%. maybe
that's okay for my use case, but I've
now just dropped my cross my cost
profile by multiple orders of magnitude.
And it's the optimizer that allows you
to do that type of model and kind of use
case transferability, if you will. But
really all it does at at the end of the
day under the hood is iteratively prompt
uh iteratively optimize or tweak that
prompt, that string under the hood. And
because you've constructed your program
using the different modules, DSPI kind
of handles all of that for you under the
hood. So if you compose a program with
multiple modules and you're optimizing
against all that, it it by itself DSPI
will optimize the various components in
order to improve the input and output
performance.
And we'll we'll take it from the man
himself, Omar. You know, ESPI is not an
optimizer. I've said this multiple
times. it's it's just a set of
programming abstractions or a way to
program. You just happen to be able to
optimize it. Um so again, the value that
I've gotten and my team has gotten is
mostly because of the programming
abstractions. It's just this incredible
added benefit that you are also able to
to should you choose to to optimize it
afterwards.
And I was listening to this to Dwaresh
and and uh Carpathy the other day and
this kind of I was like prepping for
this talk and this like hit home
perfectly. I was thinking about the
optimizers and someone smarter than me
can can ple you know please correct me
but I think this makes sense because he
he was basically talking about using LLM
as a judge can be a bad thing because
the model being judged can find
adversarial examples and degrade the
performance or basically um create a
situation where the judge is not uh not
scoring something properly. um because
he's saying that the model will find
these little cracks. It'll find these
little spirious things in the nooks and
crannies of the giant model and find a
way to cheat it. Basically saying that
LM as a judge can only go so far until
the other model uh finds those
adversarial examples. If you kind of
invert that and flip that on its head,
it's this property that the optimizers
for DSpir are taking advantage of to
optimize to find the nooks and crannies
in the model, whether it's a bigger
model model or smaller model to improve
the performance against your data set.
So that's what the optimizer is doing is
finding finding these nooks and crannies
in the model to optimize and improve
that performance.
So a typical flow, I'm not going to
spend too much time on this, but fairly
logical constructor program which is
decomposing your logic into the modules.
You use your metrics to define basically
the contours of how the program works
and you optimize all that through um to
to get your your uh your final result.
So, another talk that this guy Chris
Pototts just had maybe two days ago, um,
where he made the point, this is what I
was mentioning before, where Jeepa,
which is, uh, you probably saw some of
the the talks the other day, um, where
the optimizers are on par or exceed the
performance of something like GRPO,
another kind of fine-tuning method. So,
pretty impressive. I think it's an
active area of research. people a lot
smarter than me like Omar and Chris and
others are are leading the way on this.
But uh point being I think prompt op
prompt optimization is a pretty exciting
place to be and if nothing else is worth
exploring.
And [clears throat] then finally metrics
again these are kind of the building
blocks that allow you to define what
success looks like for the optimizer. So
this is what it's using and you can have
many of these and we'll see examples of
this where again at a high level your
program works on inputs it works on
outputs the optimizer is going to use
the metrics to understand okay my last
tweak in the prompt did it improve
performance it did it degrade
performance and the way you define your
metrics uh provides that direct feedback
for the optimizers to work on.
Uh so here's another example, a super
simple one from that time entry example
I mentioned before. Um, so they can be
the metrics can either be like fairly
rigorous in terms of like does this
equal one or or you know some type of
equality check or a little bit more
subjective where using LLM as a judge to
say whatever was this generated um
string does it adhere to these you know
various criteria whatever it might be
but that itself can be a metric
and so all of this is to say it's a very
long-winded way of saying in my opinion
this is probably most if not all of what
you need to construct arbitrarily
complex workflows, data processing
pipelines, business logic, whatever that
might be. Different ways to work with
LLMs. If nothing else, DSPI gives you
the primitives that you need in order to
build these modular composable systems.
So, if you're interested in some people
online, um
there's many many more. There's a
Discord community as well. Um, but
usually these people are are on top of
the latest and greatest and so would
recommend giving them a follow. You
don't need to follow me. I don't really
do much, but uh the others on there are
are really pretty good.
Okay, so the fun part, we'll actually
get into some to some code. So, if you
haven't had a chance, now's your last
chance to get the repo.
U, but I'll just kind of go through a
few different examples here of what we
talked about. Maybe
Yeah. Okay.
Okay. So, I'll set up Phoenix, which is
from Arise, uh, which is basically an
obser an observability platform. Uh, I
just did this today, so I don't know if
it's going to work or not, but we'll
we'll see. We'll give it a shot. Uh, but
basically what this allows you to do is
have a bunch of observability and
tracing for all the calls that are
happening under the hood. We'll see if
this works. We'll give it like another 5
seconds.
Um, but it should, I think,
automatically do all this stuff for me.
Yeah. So, let's see.
Yeah. All right. So, something's up.
Okay, cool. So,
I'll just I'm just going to run through
the notebook, which is a collection of
different use cases, basically putting
into practice a lot of what we just saw.
Feel free to jump in any questions,
anything like that. We'll start with
this notebook. There's a couple of other
uh more proper Python programs that
we'll walk through afterwards. Uh but
really the intent is a rapidfire review
of different ways that DSPI has been
useful to me and others. So
load in the end file. Usually I'll have
some type of config object like this
where I can very easily use these later
on. So if I'm like call like model
mixing. So if I have like a super hairy
problem or like some workload I know
will need the power of a reasoning model
like GPD5 or something else like that,
I'll define multiple LM. So like one
will be 41, one will be five, maybe I'll
do a 41 nano um you know Gemini 2.5
flash, stuff like that. And then I can
kind of intermingle or intersperse them
depending on what I think or what I'm
reasonably sure the workload will be.
and you'll see how that comes into play
in terms of classification and others.
Um, I'll pull in a few others here. I'm
I'm using open router for this. So, if
you have an open router API key, would
recommend plug plugging that in. So, now
I have three different LLMs I can work
with. I have Claude, I have Gemini, I
have 41 mini. And then I'll ask
basically for each of them who's best
between Google Anthropic OpenAI. All of
them are hedging a little bit. They say
subjective, subjective, undefined. All
right, great. It's not very helpful. But
because DSPI works on Pyantic, I can
define the answer as a literal. So I'm
basically forcing it to only give me
those three options and then I can go
through each of those. And you can see
each of them, of course, chooses their
own organization. Um, the reason that
those came back so fast
is that DSP has caching automated under
the hood. So as long as nothing has
changed in terms of your uh your
signature definitions or basically if
nothing has changed this is super useful
for testing it will just load it from
the cache. Um so I ran this before
that's why those came back so quickly. U
but that's another kind of super useful
um piece here. Let's see.
Okay.
Make sure we're up and running. So, if I
change this to hello
with a space,
you can see we're making a live call.
Okay, great. We're still up. So, super
simple class sentiment classifier.
Obviously, this can be built into
something arbitrarily complex. Make this
a little bit bigger. Um, but I'm
basically I'm giving it the text, the
sentiment that you saw before, and I'm
adding that additional specification to
say, okay, lower uh is more negative,
higher is more positive. I'm going to
define that as my signature. I'm going
to pass this into just a super simple
predict object.
And then I'm going to say, okay, well,
this hotel stinks. Okay, it's probably
pretty negative. Now, if I flip that to
I'm feeling pretty happy. Whoops.
Good thing I'm not in a hotel right now.
U you can see I'm feeling pretty happy.
Comes down to eight. And this might not
seem that impressive and you know it's
it's not really but uh the the the
important part here is that it just
demonstrates the use of the shorthand um
signature. So I have I have the string,
I have the integer, I pass in the custom
instructions which would be in the dock
string if I use the class B classbased
uh method. The other interesting part or
or useful part about DSPI comes with a
bunch of usage information built in. So
um because it's cached, it's going to be
an empty object.
But when I change it, you can see that
I'm using Azure right now, but for each
call, you get this nice breakdown. and I
think it's from late LLM, but allows you
to very easily track your usage, token
usage, etc. for observability and
optimization and everything like that.
Just nice little tidbits uh that are
part of it here and there. Make this
smaller.
Uh we saw the example before in the
slides, but I'm going to pull in that
form 4
off of online. I'm going to create this
doc objects using attachments. You can
see some of the stuff it did under the
hood. So, it pulled out um PDF plumber.
It created markdown from it. Pulled out
the images, etc. Again, I don't have to
worry about all that. Attachments make
that super easy. I'm going to show you
what we're working with here. This case,
we have the form four. And then I'm
going to do that poor man's rag that I
mentioned before. Okay, great. How many
shares were were sold in total? It's
going to go through that whole chain of
thought and bring back the response.
That's all well and good, but the power
in my mind of DSPI is that you can have
these arbitrarily complex data
structures. That's fairly obvious
because it uses paidantic and everything
else, but you can get a little creative
with it. So in this case, I'm going to
say, okay, a different type of document
analyzer signature. I'm just going to
give it the document and then I'm just
going to defer to the model on defining
the structure of what it thinks is most
important from the document. So in this
case, [clears throat] I'm defining a
dictionary object and so it will
hopefully return to me a series of key
value pairs that describe important
information in the document in a
structured way. And so you can see here
again this is probably cached uh but I
passed in I did it all in one line in
this case but I'm saying I want to do
chain of thought using the document
analyzer signature and I'm going to pass
in the input field which is just the
document here. I'm going to pass in the
document that I got before. And you can
see here it pulled out bunch of great
information in the super structured way.
And I didn't have to really think about
it. I just kind of deferred all this to
the model to DSPI for how to do this.
Now, of course, you can do the inverse
in saying, okay, I have a very specific
business use case. I have something
specific in terms of the formatting or
the content that I want to get out of
the document. I define that as just kind
of your typical paid classes. So in this
case I want to pull out the if there's
multiple transactions the schema itself
important information like the filing
date
going to define the document analyzer
schema signature. Uh again super simple
input field which is just the document
itself which is parsed by attachments
gives me the text and the images and
then I'm passing in the document schema
parameter which has the document schema
type which is defined above and this is
the this is effectively what you would
pass into structured outputs um but just
doing it the DS pie where it's going to
give you um basically the the output in
that specific format. So you can see
pulled out things super nicely. Filing
date, form date, form type, transactions
themselves, and then the ultimate
answer. [clears throat] And it's nice
because it exposes it in a way that you
can use dot notation. So you can just
very quickly access the the resulting
objects.
So looking at adapters, um I'll use
another little tidbit from DSPI, which
is the inspect history. So for those who
want to know what's going on under the
hood, inspect history will give you the
raw dump of what's actually going on. So
you can see here the system message that
was uh constructed under the hood was
all of this. So you can see input fields
are document output fields or reasoning
and the schema. It's going to pass these
in. And then you can see here the actual
document content that was extracted and
put into the text and into the prompt uh
with some metadata. This is all
generated by attachments. And then you
get the response which follows this
specific format. So you can see the
different fields that are here. And it's
this kind of relatively arbitrary
response um basically format for the for
the names which is then parsed by the
pie and passed back to you as the user.
Um, so I can do okay response.document
schema and get the the actual result.
To show you what the BAML adapter looks
like, we can basically do two different
calls. So this is an example from uh my
buddy Pashant uh online again. So what
we do here is define pyantic model super
simple one. Patient address and then
patient details. Patient details has the
patient address object within it. And
then we're going to say we're going to
create a super simple DSPI signature to
say taking a clinical note which is a
string. The patient info is the output
type. And then note so I'm going to run
this two different ways. The first time
with the smart LLM that I mentioned
before and just use the the built-in
adapter. So I don't specify anything
there. And then the second one will be
using the BAML adapter which which is
defined there. Um so I guess a few
things going on here. One is the ability
to use Python's uh context which is the
the lines starting with with width which
allow you to basically break out of what
the global LLM um has been defined as
and use a specific one just for that
call. So you can see in this case I'm
using the same LM but if I want to
change this to like LM anthropic or
something
I think that should work. Um, but
basically what that's doing is just
offloading that call to the other
whatever LLM that you're defining
[clears throat] for that particular call
and something happened. And I'm on a
VPN, so let's kill that.
Sorry, Alex Partners.
Okay.
Okay, great. So, we had two separate
calls. One was to the smart LLM, which
is I think 41. The other one was to
Anthropic. Same. Everything else is the
exact same. The notes exact same, etc.
We got the same exact output. That's
great. But what I wanted to show here is
the adapters themselves. So in this
case, I'm doing inspect history equals
2. So I'm going to get both of the last
two calls. And we're going to see how
the prompts are going to be different.
And so you can see here the first one,
this is the built-in JSON schema, this
crazy long JSON string. Yeah, LLMs are
good enough to to handle that, but um
you know, probably not for super
complicated ones. Um uh and then you see
here for the the second one, it uses the
BAML notation, which as we saw in the
slides, a little bit easier to
comprehend. Um and on super complicated
use cases can actually have a measurable
u improvement.
Multimodal example, same sort of thing
as before. I'll pull in the image
itself.
Let's just see what we're working with.
Okay, great. We're looking at these
various street signs.
And I'm just going to ask it super
simple question. It's this time of day.
Can I park here now? When when should I
leave? And you can see I'm just passing
in again the super simple um shorthand
for defining a signature which then I
get out the the var the boolean in this
case and a string of when I can leave.
Um
so modules themselves it's again fairly
simple. You just kind of wrap all this
in a class. Good question.
>> So does it return reasoning by default
always?
>> Oh good question. Yeah. So when you do
>> can you repeat the question?
>> Yes. So for those online the question
was does it always return reasoning by
default? When you call DSPI.chain chain
of thought as part of the module where
it's built in. It's adding the reasoning
u automatically into your response. So
you're not defining that. It's a great
question. It's not defined in the
signature as you can see up here. Uh but
it will add that in and expose that to
you um to the extent that you want to
retain it for any you know any reason.
Uh but that's so if I ju if I changed
this to predict
you wouldn't get that same response,
right? You just you literally just get
that part.
Um so that's actually a good segue to
the modules. Um so module is basically
just wrapping all that into some type of
replic replicable uh logic. Um and so
we're just we're giving it the signature
here. We're saying selfpredict.
We're in this case is just a
demonstration of how it's being used as
a class. So I'll just add this module
identifier and sort some sort of counter
but this can be any type of arbitrary
business logic or control flow or any
database action or whatever it might be.
When this image analyzer class is called
this function would run um and then when
you actually invoke it this is when it's
actually going to run the the core
logic. And so you can see I'm just
passing in the So I'm instantiating it
the analyzer of AIE123
and then I'll call it.
Great. It called that and you can see
the counter incrementing each time I
actually make the call. So super simple
example. Um we don't have a ton of time
but I'll I'll show you some of the other
modules and how that kind of works out.
Terms of tool calling fairly
straightforward. I'm going to define two
different functions perplexity search
and get URL content. creating a bioagent
module. So this is going to define
Gemini 25 as this particular module's um
LLM. It's going to create an answer
generator object which is a react call.
So I'm going to basically do tool
calling whenever this is called and then
the forward function is literally just
calling that answer generator with the
parameters that are provided to it. And
then I'm creating an async version of
that function as well.
So I can do that here. I'm going to say
okay identify instances where a
particular person has been at their
company for more than 10 years. It needs
to do tool calling to do this to get the
most up-to-date information. And so what
this is doing and basically looping
through um and it's going to call that
bio agent which is using the tool calls
in the background and it will make a
determination as to whether their
background is applicable per my
criteria. In this case, Satia is true.
Brian should be false. Uh but what's
interesting here while that's going in
it uh similar to the reasoning uh par or
the reasoning object that you get back
for chain of thought you can get a
trajectory back for things like react.
So you can see what tools it's calling
the arguments that are passed in um and
the observations for each of those calls
which is nice for debugging and and
other obviously other uses.
Um I want to get to the other content so
I'm going to speed through the rest of
this. This is basically an async version
of the same thing. So you would run both
of them in parallel. Same idea.
Um I'm going to skip the JEPA example
here just for a second. Um I can show
you what the output looks like, but
basically what this is doing is creating
a data set.
It is showing you what's in the data
set. It's creating a variety of
signatures. In this case, it's going to
create a system that categorizes and
classifies different basically help
messages um that is part of the data
set. So, my sync is broken or my light
is out or whatever it is. They want to
classify whether it's positive, neutral,
or negative and the uh the urgency of
the actual message. It's going to
categorize it and then it's going to
pack all this stuff, all those different
modules into a single support analyzer
module. And then from there, what it's
going to do is define a bunch of metrics
which is based off of the data set
itself. So it's going to say, okay, how
do we score the urgency? This is a a
very simple one where it's okay, it
either matches or it doesn't. Um, and
there's other ones where it can be a
little bit more subjective and then you
can run it. This going to take too long.
Probably takes 20 minutes or so. Um but
uh what it will do is basically evaluate
the performance of the base model and
then apply those metrics uh and
iteratively come up with new prompts to
uh to create that.
Now I want to pause here just for a
second because there's different types
of metrics and in particular for Jeepa
it uses feedback from the teacher model
in this case. So it can work with the
same level of model, but in particular
when you're trying to use say a smaller
model, um it can actually provide
textual feedback. So, it says not only
did you get this classification wrong,
but it's going to give you some
additional um information or feedback as
you can see here for why it got it wrong
or what the answer should have been,
which allows it you you can read the
paper, but it basically allows it to um
iteratively find that kind of paro
frontier of how it should uh tweak the
prompt to optimize it based off that
feedback. It basically just tightens
that iteration loop.
Um you can see there's a bunch here. Um
and then you can run it and see how it
works. [snorts] Um but kind of just to
give you a concrete example of how it
all comes together. So we took a bunch
of those examples from before. We're
basically basically going to do a bit of
um categorization. So I have things like
contracts, I have images, I have
different things that one DSPI program
can comprehend and do some type of
processing with. So this is something
that we see fairly regularly in terms of
we might run into a client situation
where they have just a big dump of of
files. They don't really know what's in
it. They want to find something of uh
they want to maybe find SEC filings and
process them a certain way. they want to
find contracts and process those a
certain way. Maybe there's some images
in in there and they want to process
those a certain way. Uh [snorts] so this
is an example of how you would do that
where if I start at the bottom here,
this is a regular Python file. Um and it
uses DSPI to do all those things I just
mentioned. So we're pulling in the
configurations,
we're setting the regular LM, the small
and one we use for an image. As an
example, Gemini might Gemini models
might be better at image recognition
than others. So I might want to defer or
use a particular model for a particular
workload. So if I detect an image, I
will route the request to Gemini. If I
detect something else, I'll route it to
a 4.1 or whatever it might be.
So I'm going to process a single file.
And what it does is use our handy
attachments
um library to put it into a format that
we can use. And then I'm going to
classify it. And it's not super obvious
here, but I'm getting a file type from
this classify file uh function call. And
then I'm doing some different type of
logic depending on what type of file it
is. So if it's an SEC filing, I do
certain things. If it's a certain type
of SEC filing, I do something else. Uh,
if [snorts] it's a contract, maybe I'll
summarize it. If it's something that
looks like city infrastructure, in this
case, the image that we saw before, I
might do some more visual interpretation
of it. Um, so if I dive into classify
file super quick,
it's running the document classifier.
And all that is is basically doing a
predict on the image from the file. and
um making sure
it returns a type. Where is this
returns a type which would be document
type and so you can see here at the end
of the day it's a fairly simple
signature and so what we've done is
basically take the PDF file in this case
take all the images from it and take the
first image or first few images in this
case a list of images as the input field
and I'm saying okay just give me the
type what is this and I'm giving it an
option of these document types so
obviously say this is a fairly simple
use case but it's basically saying given
these three images the first three pages
of a document is it an SEC filing is it
a patent filing is the contract city
infrastructure pretty different things
so the model really shouldn't have an
issue with any of those and then we have
a catchall bucket for other and then as
I mentioned before um depending on the
file type that you get back you can
process them differently so I'm using
the small model to do the same type of
form4 extraction that we saw before um
and then asserting basically in this
case that it is what we think it is. Um
a contract in this case we're saying uh
let's see I have like 10 more minutes so
we can go we'll we'll stop after this uh
up to this file but for the particular
contract we'll go we'll create this
summarizer object. So we'll go through
as many pages as there are. We'll do
some uh basically recursive
summarization of that using a separate
DSPI function and then we'll detect some
type of boundaries of that document too.
So we'll say I want the summaries and I
want the boundaries of the document. Um
and then we'll print those things out.
So let's just see if I can run this.
It's going to classify it should as a
[clears throat] contract.
>> So is you're just relying on the model
itself to realize that it's a city
infrastructure.
>> Yeah. The question was I'm I'm just
relying on the model to determine if
it's a city infrastructure. Yes. I mean
this is more just like a workshop quick
and dirty example. It's only because
there's one picture of the street signs.
Um, and if we look in the data folder, I
have a contract,
some image that's irrelevant, the form
for SEC filing, and then the parking
too. Um, they're pretty different. The
model should have no problem out of
those categories that I gave it to
categorize it properly. In some type of
production use case, you would want much
more stringent or maybe even multiple
passes of classification, maybe using
different models to do that. Um but
yeah, given those options, at least the
many times I've run it, had no problem.
So in this case, I gave it um one of
these contract documents and it ran some
additional summarization logic under the
hood. So, if I go to that super quick,
um you can find all this in the code,
but basically what it does is use three
separate signatures to basically
decompose the contents of the the um the
contract and then summarize them up. So,
it's basically just iteratively working
through each of the chunks of the
document to create a summary that you
see here at the bottom. And then just
for good measure, we're also detecting
basically the the boundaries of the
document to say, okay, here's out of the
13 pages, you have the main document and
then some of the exhibits or the
schedules that are a part of it. So, let
me just bring it up super quick
just to show you what we're working
with. This is just some random thing I
found online. And you can see so it said
the main document was from page 0 to six
and the way and so we zero 1 2 3 4 5 six
seems reasonable. Now we have the start
of schedule one.
Schedule one it says it's the next two
pages. That looks pretty good. Schedule
two is just the one page 9 to9.
That looks good. and then schedule three
through to the end of the document.
And that looks pretty good, too. And so
the way we did that under the hood was
basically take the PDF, convert it to a
list of images and then for each of the
images pass those to classifier
um and then use that to
well let's just look at the code but
basically take the list of those
classifications
give that to another DSPI signature to
say given these classifications of the
document give me the structure and
basically give me a key pair of you name
of the section and two integers, a
tupole of integers that detect or that
uh determine the um you know the
boundaries essentially. Um so that's
what that part does.
Um [clears throat]
if we go back so city infrastructure,
I'll do this one super quick just
because it's pretty interesting on how
it uses tool calls. And while this is
running,
I should use the right one. Hold on.
>> [clears throat]
>> Yeah,
>> good question. The second part like when
you generated the list of like my
documents from 0 to six, did you have
like original document as an input or
no?
>> No. Uh so let let's just go to that uh
that was super quick. So
that should be boundary detector.
So, there's a blog post on this that I
published probably in August or so that
goes into a little bit more detail. The
code is actually pretty crappy in that
one. It's it's going to be better here.
Um, but basically what it does is
this is probably the main logic. So, for
each of the images in the PDF, we're
going to call classify page.
We're going to gather the results. So
it's doing all that asynchronously
pulling it all back saying okay all
these you know all the different page
classifications that there are and then
I pass the output of that into a new
signature that says given tupil of p I
don't even define it here given tupil of
page and classification
give me this I don't know relatively
complicated output of a dictionary of a
string tupil integer integer and I give
it this set of instructions to say just
detect the boundaries. Like this is all
very like non-production code obviously,
but the point is that you can do these
types of things super super quickly.
Like I'm not specifying much not giving
it much context and it worked like
pretty well. Like it it's worked pretty
well in most of my testing. Now
obviously there is a ton of low hanging
fruit in terms of ways to improve that,
optimize it, etc.
Um, but all this is doing is taking that
signature, these instructions, and then
I call react. And then all I give it is,
uh, the ability to basically
self-reflect and call um, get page
images. So, it says, okay, I'm going to
look at this boundary. Well, let me get
the the page images for these three
pages to and make sure basically that
the boundary is correct. And then it
uses that to construct the final answer.
And so it's really this is a perfect
example of like the tight iteration loop
that you can have both in um building it
but then the you can kind of take
advantage of the model's introspective
ability if you will to use function
calls against the data itself the data
it generated itself etc to kind of keep
that loop going. question.
>> So under the hood, the the beauty of ESP
then is that it enforces kind of
structured output on a on a model.
>> I mean yes, I think that's probably
reductive of of like its full potential,
but generally that's that's correct. I
mean yes, you can use structured
outputs, but you have to do a bunch of
crap basically to coordinate like
feeding all the feeding that into the
rest of the program. maybe you want to
call a model differently or use XML here
or use a different type of model or
whatever it might be
um to to do that. So absolutely yeah I'm
not saying this is the only way
obviously to kind of create these
applications or that you shouldn't use
Pantic or shouldn't use structured
outputs. You absolutely should. Um, it's
just a way that once you kind of wrap
your head around the the primitives that
DSPI gives you, you can start to very
quickly build these types of
arguably uh I mean these are like
prototypes right now, but like if you
want to take this to the next level to
production scale, you have all the
pieces in front of you to be able to do
that.
>> Um,
any other questions? I probably got
about five minutes left. Go ahead. Can
you talk about your experience using
optimization
and just
>> Yeah. Yeah. So Jeep and actually I'll
pull up uh I I ran one right before
this. Um this uses a a different
algorithm called my row but basically um
the optimizers as long as you have well
structured data. So for the machine
learning folks in the room, which is
probably everybody, obviously the
quality of your of your data is very
important,
um you don't need thousands and
thousands of examples necessarily, but
as long as you have enough, maybe 10 to
100 of inputs and outputs.
[clears throat] And if you're
constructing your metrics in a way that
is relatively intuitive and and that,
you know, accurately describes what
you're trying to achieve, the
improvement can be pretty significant.
Um, and so that time entry corrector
thing that I mentioned before, uh, you
can see the output of here. It's kind of
iterating through. It's measuring the
output metrics for each of these. And
then you can see all the way at the
bottom once it goes through all of its
optimization stuff.
You can see the actual performance
on
um, the basic versus the optimized
model. In this case, went from 86 to 89.
And then interestingly, this is still in
development, this one in particular, but
you can break it down by metric. So you
can see where the model's optimizing
better, performing better across certain
metrics. And this can be really telling
as to whether you need to tweak your
metric, maybe you need to decompose your
metric, maybe there's other areas within
your data set, or the the basically the
structure of your program that you can
improve. Um, but it's a really nice way
to understand what's going under the
under the hood. And if if you don't care
about some of these and the optimizer
isn't doing as well on them, maybe you
can maybe you can throw them out, too.
So, it's it's a very kind of flexible
system, flexible way of kind of doing
all that.
>> Yeah. What's the output of the
optimization? Like what do you get out
of it and then how do you use that
object, whatever it is?
>> Yeah. Yeah. So the output of the
optimizers is basically just another um
it's almost like a compiled object if
you will.
>> So DSPI allows you to save and load
programs as well. So the output of the
optimizer is basically just a module
that you can then serialize and save off
somewhere
>> or you can call it later uh as you would
any other module
>> and it's just manipulating the phrasing
of the prompt. So like what is it
actually like you know what's its
solution space look like?
>> Yeah. Yeah. under the hood, it's
literally just iterating on the actual
prompt itself. Maybe it's adding
additional instructions. It's saying,
"Well, I keep failing on this particular
thing, like not capitalizing the names
correctly. I need to add [clears throat]
in my upfront criteria in the prompt an
instruction to the model to say you must
capitalize names properly." And Chris uh
who I mentioned before has a really good
way of putting this and I'm going to
butcher it now, but like the optimizer
is basically finding latent requirements
that you might not have specified
initially up front, but based off of the
data, it's kind of like a poor man's
deep learning, I guess, but like it's
learning from the data. It's learning
what it's doing well, what what it's
doing not so well, and it's dynamically
constructing a prompt that improves the
performance based off of your metrics.
And is that like LMG guided like is it
like about like capitalization?
>> Yeah. Yeah. Question being is it all LLM
guided? Yes. Particularly for Jeepa it's
using LLM to improve LLM's performance.
So it's using the LLM to dynamically
construct new prompts which are then fed
into the system measured and then it
kind of iterates. So it's using AI to
build AI if you will.
>> Thank you.
>> Yeah
question. Why is the solution object not
just the optimized prompt?
>> Why is the solution object not what?
>> Not just the optimized prompt. Why are
you using
>> Oh, absolutely is. You can get it under
the hood. I mean, you can The question
was why don't you just get the optimized
prompt? You can absolutely. Um,
>> what else is there besides
>> the the So, what else is there other
than the prompt? The DSPI object itself.
So the module the way things um well we
can probably look at one if we have
time. Um
>> if I could see a dump of what gets you
know what is the optimized state that
would be interesting.
>> Yeah. Yeah sure. Let me see if I can
find one quick. Um but fundamentally at
the end of the day yes you get an
optimized prompt a string that you can
dump somewhere if you if you want to. Um
actually
um
>> there's a lot of pieces to the
signature, right? So it's like how you
describe your feels in the doc.
>> This is a perfect segue and I'll I'll
conclude right after this. I was playing
around with something I was well I was
playing around this thing called DSPIHub
that I kind of created to create a
repository of optimized programs. So
basically like if you're an expert in
whatever you optimize an LLM against
this data set or have a great classifier
for city infrastructure images or
whatever kind of like a hugging face you
can download something that has been
pre-optimized
and then what I have here this is the
actual loaded program this would be the
output of the optimized process or it it
is and then I can call it as I would any
anything else and so you You can see
here this is the output and I used the
optimized program that I downloaded from
from this hub. And if we inspect maybe
the loaded program,
you can see under the hood, it's a
predict object with a string signature
of time and reasoning. Here is the
optimized prompt. Ultimately,
this is the output of the optimization
process. this long string here.
Um, and then the various uh
specifications and definitions of the
inputs and outputs.
>> Have you found specific uses of those?
Like to his question like what is it?
What can you do with that?
>> It's up to your it's up to your use
case. So if I if I have a so a document
classifier might be a good example. If
in my business I come across whatever
documents of a certain type, I might
optimize a classifier against those and
then I can use that somewhere else on a
different project or something like
that. So out of 100,000 documents, I
want to find only the pages that have an
invoice on it as an example. Now sure
100% you can use a typical ML classifier
to do that. That's great. This is just
an example. We can also theoretically
train or optimize a model to do that
type of classification or some type of
generation of text or what have you
which then you have the optimized state
of which then lives in your data
processing pipeline you know and you can
use it for other types of purposes or
give it to other teams or whatever it
might be. So it's just up to your
particular use case. um something like
this like hub who maybe it's not useful
because each individual's use case is so
hyper specific I don't really know but
um yeah you can do with it kind of
whatever you want last question yeah
>> is generally you know like using DSP
something where people kind of do
replays just to optimize their prop or
is there a way to sort of do it in real
time given delays
What I mean by delayed is okay chat GPT
gives you your answer and you can thumbs
up or thumbs down. You know that thumbs
up comes you know 10 minutes later, 30
minutes later, a day later, right?
>> So is the question more about like
continuous learning like how would you
do that here?
>> You can be the judge.
>> Well, how are you feeding back delayed
metrics to optimize it? Why why would it
need to be delayed? Because you know
usually the feedback is from the user,
right? Like delayed.
>> Yeah. Well, then
>> yeah, that's right. You it basically be
added to the data set and then you would
use the latest optimize and just keep
keep optimizing off of that
>> ground truth data set.
>> That's right.
>> You will collect the outputs of your
optimization and feed it back and the
loop hits.
>> Yeah. But that Why you're trying to do
offline optimization, right?
>> Yes.
>> But I'm I'm asking, can you do this
online where with the metric feedback?
>> If you're good if you're a good enough
engineer, you probably do it. But
>> I'm not I'm not recommending replacing
ML models with like optimized DSPI
programs for particular use cases. Maybe
like classification is a terrible
example. I recognize that. But for other
other are other in theory, yes, you
know, you could do something like that.
Yes.
But for for particular LLM tasks, I'm
sure we all have interesting ones. If
you have something that is relatively
well defined where you have known inputs
and outputs, it might be a candidate for
something worth optimizing. If nothing
else, to transfer it to a smaller model
to preserve the level of performance at
a lower cost. That's really one of the
biggest benefits I see.
All right, last last question.
I've heard that uh DSPI is can be kind
of expensive because you're doing all
these LM calls.
>> Um so I was curious your experience with
that and maybe relatedly like if you
have any experience with like large
context in your optimization data set
ways of shrinking those.
>> Yeah. So the question was do can BSI be
expensive and then for large context
kind of how have you seen that? How have
you managed that? The expensive part is
totally up to you. If you call a
function a million times asynchronously,
you're going to generate a lot of cost.
I don't think DSPI necessarily maybe it
makes it easier to call things, but it's
not inherently expensive. It might, to
your point,
add more content to the prompt. Like,
sure, the signature is a string, but the
actual text that's sent to the model is
much longer than that. That's totally
true. I wouldn't say that it's a large
cost driver. I mean it again it's
ultimately it's more more of a
programming paradigm. So you can write
your compressed adapter if you want that
like you know reduces the amount that's
sent to the to uh to the model. Um in
terms of large context I it's kind of
the same answer I think in terms of if
you're worried about that maybe you have
some additional logic either in the
program itself or in an adapter or part
of the module that keeps track of that.
Maybe you do some like context
compression or something like that.
There's some really good talks about
that past few days. Obviously, I have a
feeling that that will kind of go away
at some point where either context
windows get bigger or context management
is abstracted away somehow. I don't
really have an answer just that's more
of an intuition. Um, but DSP again kind
of gives you the tools, the primitives
for you to do that should you choose.
Um, and kind of track that state, track
that management over time.
So, I think that's it. We're going to
get kicked out soon. So, thanks so much
for your time. Really appreciate it.
[music]
[music]