Context Engineering: Connecting the Dots with Graphs — Stephen Chin, Neo4j

Channel: aiDotEngineer

Published at: 2025-11-24

YouTube video id: LLuKshphGOE

Source: https://www.youtube.com/watch?v=LLuKshphGOE

Hello everybody and welcome to my
session at a engineer code summit and
I'm going to talk a bit about how you
can connect the dots with graph
technology and solve problems like
context engineering um improving
retrieval patterns and also agentic
memory. So we're going to have a lot of
fun. My name is Stephen Chin. I'm VP of
developer relations at Neo Forj and you
can find me at all the different social
media outlets with my handle Steve on
Java. So excited you're all here to join
for the session today. And I think this
is how a lot of us have felt the past
couple years as AI technology has
basically taken our jobs away. We've
become slaves to um AI programming, to
prompt development, to building things
off AI models. Now, it's not all bad. I
mean, we we have a lot more time to to
play games, to hang out in the matrix,
but what we really want to be doing is
we want to be doing things which are
higher value. So this is where context
engineering comes in the picture and
transforms what we've traditionally been
doing with kind of oneshot clever
phrasing prompt engineering to get
different results out of the AI and
we're evolving that to have a more
dynamic and and wider scope of things
which we're feeding the AI as context
which gives us much better results. So
um this allows us to feed the desire of
agents to get even more context and
information to do things together. Um to
have more dynamic models and
applications to make our applications
goal driven. Um selectively curate the
information for the relevancy of the
particular domain which we're working
in. So if you're working in a um an
enterprise domain, if you're working
with a lot of business context, this is
particularly important. And then we can
structure the input and get a lot more
signal over all the noise of what's
being entered into the model which is
one of the biggest problems with models
today. Um huge context windows but very
little attention focus and simply not
looking at the right parts of the
context to give us good results. And
this allows us to think not like prompt
engineers but like information
architects where we're building the
model context which actually gives us
superior results coming out of the AI.
And this evolves us from being um you
know your traditional trapped to matrix
worker to being superheroes. So this is
this is where we want to be. We want to
be in control of our destiny. We want to
be able to give the agents all of the
information all the context which they
need to perform the task and to do
exactly what we want to get for results
out of it. And there's a lot of tools at
our disposal now which allow us to
manipulate, control the context and
really um feed the AIS all the
information which they need to be
successful. So in the kind of the scope
of context engineering
there's a whole bunch of things which
are clearly um part of the domain that
this now encompasses. So one is prompt
engineering. Of course, we need to
design engineer good good prompts. Um,
make sure that the AI actually has the
right instructions, the right
information and the right grounding
which it needs to do its job well. But
we also need to pull in from different
data sources by using things like
retrieval, augmented generation. So um
rag is still very relevant for the
ability to pull in data from enterprise
context from different business contexts
and then supply that as additional
information to the AI that it can use to
make decisions. Um pulling in state and
history as well. So now we actually want
our models want memory um both
short-term memory so they can
collaborate with each other and also
long-term memory so they remember the
conversation state the history and um
they can do more effective long-term
operations
and we also want to be able to structure
the output in a meaningful way so we can
actually feed into not only other
applications but other tools and things
which we need to collaborate with and
integrate our um context with. And when
you put this all together, this is kind
of the the scope and domain of context
engineering. Now, one of the big focuses
of this is all about memory. So, it's
all about how we capture the AI memory
and what we're able to do with it. So,
um there's kind of two main
categorizations of memory. One is
short-term memory. So, this is what the
AI is currently working with on the
current tasks. Um, we want to compress
as much information as possible into the
short-term memory and give it relevant
results which are high up in the search
window. Um, be able to integrate tool
results into this as well, although you
know not give it too much information
from tools, especially from previous
exchanges where the tools might have
dumped a lot of output or information
which will fill our context window. Um,
and in addition to this, we also need to
mix in long-term memory. So things which
you've learned over a long set of
conversations which might be episodic.
Um we need to figure out the semantic
and the structural meaning of past
conversations. Um kind of pull this out
into things which can either be used as
instructions for the AI and also for
procedures and operations which we can
use to to guide and plan the artificial
intelligence. And um when we put this
together, this helps us pull the more
relevant context higher up into the
context window, fill in the gaps, and
then avoid a lot of the noise, which
gives us bad results or um
hallucinations or other problems coming
from our AI applications.
And um memory is really the core of what
we need to accomplish. um you know if
you're plugging yourself up to the
matrix this is where you si synergize
all of your memories all the things
which you want to get into the AI
together with um your own own bind your
own neural network that you want to um
express and it's extremely important
right now because LMS are only as good
as the quality of the response that
they're getting from the data. So if you
give them bad data if you get them
garbage then you are going to get
garbage back out again. So, we needed to
give them the the right information in
the context window and kind of limit and
give it um move it up as high as
possible. Be able to do more dynamic
prompting with things like DSPY and
BAML. Um ability to do more reasoning um
so we can do internal context
engineering on top of our data. This
will turn us from human developers into
agents where we're actually using more
agent technology to fuel our
applications to build things. Um, and
then this allows us to focus more on the
time which we're doing our tests rather
than just focusing on the time which
we're training our models. Um, so
together when we have more context now
it allows us to do better things but
then it's still important that we really
have structured information relevant
inputs and this improves the reliability
and the explanability of our models that
come out of it. Um so one of the ways
which we can do this is by leveraging
knowledge graphs. Knowledge graphs are a
technology which has been around for a
while but they're very applicable for AI
because they fill in that gap between
the AI which is very good at um creating
things building things kind of pulling
from different sources it has but um
structured information knowledge graphs
are a structured representation to
understand a bit about how a knowledge
graph is constructed.
It's typically built with facts which
are are nodes about people, places,
events or things. Those are linked
together by relationships or um um lines
between them which reference how those
things are related. It's very easy for
both humans and LLMs to read knowledge
graphs. So both acts as a um organizing
concept, but also a way which you can
understand what your AI is doing and
actually look at some of the data behind
it. And um it can be also very useful as
a digital twin of your organization, of
your supply chain, of um a whole bunch
of different processes in your
organization. And the basic construct of
a knowledge graph is um nodes which
represent different people in the
situation, relationships, and then you
can attach properties to these nodes. So
this is an example of a knowledge graph
where you have two people, they know
each other, they live with each other.
Well, one person lives with the other.
So I guess technically it's Ann's house
and they both drive a car. Now the car
is owned by Dan or or by an driven by
Dan as well. So they both have a
relationship with the car and they have
a relationship with living each other.
And you can see the there's attributes
on this. How long has Dan lived or
driven the car, the type of car? So it's
a Volvo. Um it's a um model V70. some
information about it and also some
embeddings. So we can also encapsulate
embeddings on the graph as well. So we
can do vector lookups and this allows us
to do fairly complex things as we build
larger knowledge graphs to capture all
this information. And what knowledge
graphs gives the benefit of is all of
that knowledge context and enrichment
that we can build into a representation
of knowledge in addition to LLMs which
have kind of that language reasoning and
creativity and when we put them together
we can do really powerful things.
Um so we talked a bit about rag being an
essential part of context engineering
and a even better way of doing rag is
graph rag. Now what is graph rag? So
graph rag is any retrieval pipeline
which also uses graphs as part of the
retrieval process. And so um an example
of this is a user asks a question
um it goes to the LM and it does a
search and it asks for if there's any
relevant information which will go as a
query out to a knowledge graph. This
then gets passed in as additional
context to the LLM when it's answering
the question and then the LM gives an
enriched answer which is more relevant.
So it's it's a will give you more
relevant results than just a vector
similarity search because you also have
information about relationships about
nodes about community grouping more
context. So you can now get domain
information factual information
structured knowledge on your subject.
You can explain what the LM is actually
doing because you can see the part of
the knowledge graph which got passed to
the LM. And you can also evolve the
knowledge graph over time. And you can
now start to implement overlays like
role-based access. So you can say only
these people get access for example in a
um patient information system only the
doctor would have access to the
diagnosis but only the person who um
handles the administrative information
would have access to phone numbers or
addresses or other personal information
about the patient. So it allows you to
kind of overload overlay that role based
access directly on the knowledge graph
and then instruct the LM on what
information it's allowed to respond with
and um knowledge graphs allow this sort
of explainable AI. So in a in a large
graph with a lot of nodes and a lot of
information now you can store the
learnings from the user and agents at
the interactions in the graph context.
You can start to visualize conversation
flows with the addition of reasoning.
You can analyze the context data of
agent systems about performance
um identify opportunities for
improvement over time of the um either
the um the quality of the results which
you're passing in the relationships um
removing duplicate nodes so that you get
better quality results coming out of it.
So it gives you a lot of control over
the application and the ability to
modify and control what the AI is
answering kind of like you're you're
training in a in a dojo. So I think you
know in the in the film Neo spends a lot
of time doing virtual training improving
his skills with different programs he's
loaded up and um this is how we're able
to do a lot of amazing things like this
demo which I'm going to show you. So the
first demo we're going to show is a um
graph rack demo using the LLM knowledge
graph builder. So I've already set up a
Neo Forj aura instance. This is the um
um online free version of Neo Forj. You
can see I have a a running instance with
a bunch of relationships loaded up. And
to load up those relationships, I use
the knowledge graph builder. The
knowledge graph builder is a very simple
web application. It's open source and it
lets you do a couple things. So, it lets
you upload files. So, you can drag and
drop different files into the user
interface. Before the presentation, I
loaded up a couple representative files
of a um supply chain use case. One is a
supply chain document and as you can see
here it has a whole bunch of information
about different artifacts
um and the digital signatures of them
and the relationship of them. And the
second one is the more interesting one.
This is a a VEX document which is a
security standard and it talks about
some vulnerabilities
um in this case inside the Jackson
library and talks a bit about um how to
remediate with it, which versions are
affected
um which commits fix it and all that
good stuff. So um we have quite a bit of
information which we loaded up and then
what I've done is I've already dropped
those into the knowledge graph and we
can take a look at what got generated by
the LM. So it takes this through an
ingest phase um where the LM actually
builds out a knowledge graph and then we
can see that some of these nodes
represent different parts of the um VEX
document. Um here we can see some
information about um um who found the
vulnerability, information about the um
vulnerability database URL and um all
this stuff is connected with different
relationships and this allows us to
query, navigate and traverse this
information to build better responses
for the LM. So what we're going to do in
this demo is we're going to take this
knowledge graph which we built and then
we're going to run an LLM which does a
two-pass process. The first pass it's
going to do a vector lookup and find a
similarity search to find related nodes
in the knowledge graph. And the second
pass it's going to take those nodes
which are related to the result find
related nodes and then pass those in as
context to the LM. And ideally what we'd
what we'd like to get from the LM is
that um it will answer questions with
information it has from the knowledge
graph and then it won't be able to
answer questions or refuse to answer
questions with things which are outside
that knowledge um pool. So let's ask it
[gasps and sighs]
um about vulnerabilities
in the in the Jasper library. So Jasper
is another um Java library that's very
commonly used. It wasn't actually
referenced in the VEX document. So,
we're in this case, we're hoping to get
a no response. Okay, so that's amazing.
I I made a typo. I should have said
Jackson.
Let's see what we get when we um ask
about the Jackson library, even with the
typo, because LMS know that humans are
imperfect and they're very good at
fixing our mistakes. And um here we can
see that it actually pulled out
information about the Jackson databind
library with an XML injection attack. It
knows a bit about the vulnerability,
what version it's in, um whether it's
fixed and at which version it's fixed
and um all this information is is pulled
from and aggregate off the knowledge
graph. So um it gives us quite a lot of
information um very detailed and very
focused because it's rounded in a um
data which is um very um complete
um it's finite and it's something we can
edit modify and change the response over
time. So knowledge graphs are a very
powerful tool. It allows us to do things
like this where we can um get better
responses and better answers. And now
with knowledge graphs at our disposal,
now we can we can go and we can fight
the um the evil agents. Actually, it's
kind of ironic that the um the agents of
the Matrix film were um the bad guys,
but actually they operated and acted a
lot like modern agents where we're
having LMS collaborate, pull together,
and um cooperate on different
algorithms. And even the agents in the
film had different personalities and um
different types um kind of like
individual agents. So um we need to
power up and and get our um matrix and
graph skills up to a level where we can
go tackle the agents with new tools like
memory retrieval. So we we talked a bit
about graph retrieval and um graphs are
also a great tool and mechanism which
you can use to do um memory retrieval as
well. So we can do search
um in in memory use graph memory
retrieval tools. We have an open- source
MCP server which does a lot of this for
graph retrievalss. And now you can query
the graph not only for knowledge graphs
but also vectors as we did in the last
um example. And we can also use graph
data science algorithms like um
community groupings or k nearest
neighbors or different graph algorithms
which allow us to get um pull some
insights out of the relationship and the
structure of the graph. Pull back
relevant information and then pass this
as additional context um either for
short-term or long-term memory into the
agent loop. um where now we're feeding
the agent with additional information
and context from either um a short-term
memory source about the current
conversation or knowledge pulled in like
kind of what we showed in the previous
example from a graph or from a long-term
structure of memory where we memorize um
previous conversations give them
temporal information and then structure
those into a memory that can be
retrieved from the graph. And um now
we're able to use graph memory to
capture knowledge in the form of
entities and relationships between them
where some nodes have the relevant
properties such as text details
embeddings time and location on top of
them. So this is kind of a visual
representation of our of our graph our
memory graph. Some of these properties
get vector embedded and this enables us
to do vector-based semantic search. So
now we can do semantic search on the
graphs via projections into vector
space. But then we can also use
algorithms like K approximate nearest
neighbors, community groupings,
um page rank algorithms on top of the
graph to answer different types of
questions and to kind of bubble up the
most relevant results into the context.
This gives us quite a lot of power
because we can do both the vector
embeddings, but we can also
do additional graph algorithms on top of
it.
Okay, so now we have our our superpower
with our our graph where we're able to
do amazing things which aren't possible
just with um vector embeddings and
similarity searches kind of like like
the bullet time and the Matrix films. Um
this will allow us to do amazing stunts
and to evade the um um the agents. But
let's give a quick example of how this
would actually work in practice. So
let's say my question to the LM was
let's update this presentation from the
last time I presented with Sid who's my
colleague in India. Um so we can now
search this information in the graph and
there's two relevant people for this
right so it's it's me um VP of Devril
it's Sid who's a community manager and
the event and the last time we presented
it was at an event called GIDS um which
is an event in Bangalore awesome
developer conference so um now we have
kind of that temporal relationship with
the two people and then an event and we
can add to this the the memory record at
a particular time of the presentation.
So now we're pulling back information
about this presentation at a specific
point in time and we can pass this in as
context to the LM. So when we ask it to
update the presentation we both have the
context of who presented where they
presented and the time period in which
they presented for the LM to build
additional information on top of it. And
this is only possible because we can do
this um multi-stage query with graphs.
Graphs excel in use cases where you are
able to pull in multiple facts which are
related um but don't get pulled back in
a single query. If you can do it in a in
a one shot or you can get a a single
similarity search um standard vector rag
is is fine for those sorts of use cases.
It's where the relationships are two or
more where you get the real value from
doing um graph rag and graph memory.
And um this allows us to do a whole
bunch of different types of graph
retrievers. So um we could do explicit
retrieval queries where we have
pre-anned retrieval queries with
different entry points and retrieving
some context. So this gives us some
great information from the graph but we
can do better by doing text decipher. So
fine-tuning the LM with schema
generating a query for the question and
then we can kind of take this to the
next level with a genetic traversal
where we iteratively navigating over the
graph collecting information until we
answer it and using an appropriate set
of tools. And to show an example of this
um we're going to use the same knowledge
graph which I loaded up again in our
demo number two but this time we're
going to query that knowledge graph
using clawed code. So what I've done is
I've hooked up claude code using the um
Neoraj MCP cipher um MCP server which is
an open source extension. You can say
new forj cipher mcp server which I've
already configured with the database
settings and now when we talk to cloud
and we ask it a question it can answer
with that additional graph context that
it can tell us things. So, I put a few
keywords into the MCP server like um
graph and database and we can ask it um
what do you know about the Jackson
vulnerability
uh based on your graph database.
And so now in addition to you know
pulling in information from its standard
knowledge sources it's going to use the
NeoRaj MCP server and then query it to
get additional information. And you can
see that it's doing this multiplestep
plans um search of the graph. So first
it gets back the schema of the graph so
it can understand the relationships and
how the graph is structured. Now that it
understands the schema of the graph, it
can go back and it can make a bunch of
queries to get information about the
particular vulnerability. So it's firing
off a bunch of different cipher queries.
Cipher is the um graph query language
for Neo Forj and most graph databases
support it. It's also now a standard.
The GQL standard that's ratified by ISO
is um basically a subset of Cipher. And
now that it got back information about
the vulnerabilities, it's pulling back
some of the text chunks to get
additional context which are hanging off
of those nodes. And this way it can give
us a a very complete answer with as much
information and context as possible from
the graph. And the the main difference
between this approach where compared to
the previous one is the previous
approach was relatively fast but you the
level of detail it gave us on the CV was
limited. When we give a an an agent in
this case we're giving the clawed agent
the ability to kind of have at it for
the graph do traversals get information
do more traversals you can see that it
gives us back very detailed information
about the vulnerability. So it figured
out the CV number, the affected
vulnerability, the type of attack, the
severity, and a technical description of
the attack. So it's a lot more detailed
than what we got before. And it tells us
specifically what versions we need to
upgrade to remediate the attack um and
gives us some advisory information as
well about this. So um if we were trying
to develop a um vulnerability report or
something to kind of explain how we
should as an organization um address
this vulnerability um using the sort of
agentic multi-step um MCP retrieval
approach is quite powerful because you
can see that it gives us um the best
possible response since it's able to go
back and keep pulling additional
information from the knowledge graph
that it needs. Okay, so we've seen a few
different ways which we can apply
knowledge graphs to solve and improve
the context of our AI applications. So
now that we know that we need graphs, we
need we need a lot of graphs. And the
best place to find information about
graph technology and getting a lot of
different use cases for graphs is graph
academy. It's a entirely free resource
to learn about um both cipher queries.
It has courses on cypher queries, has
courses on graph rag with examples in
both Python and TypeScript. Um we have
some more advanced um graph G gra
courses coming up as well. So um a lot
of information which is all free and
very hands-on for you to get started and
actually build your first application
kind of like the ones which I showed
here in the presentation. Um now if we
want even more knowledge kind of a wider
span we can then go to nodes AI 2026
which is our free online virtual
conference. Um this is following up the
amazing nodes conference we had last
week with um over 13,000 registrants. So
it was a huge event and Nodes AI is all
about AI for um an entire day with AI
focus sessions. The CFP is open right
now if you'd like to submit and it's
free to attend and watch all the
sessions.
And if we want to really get down and
you know beat the architect at his own
game, then we need a lot of deep
research and information. And the best
place for that is graphrag.com which is
a community site which we support um
where it has all of the latest research
on different graph approaches um how-to
guides and conceptual information about
how to implement graph rag and just a
general resource which will help you to
uplevel your ability to apply graphs to
different problem domains um with a
whole bunch of of the cutting edge
latest research which is coming out. So,
um, exciting stuff. Thank you very much
for joining me for the session today.
And I hope you learned a little bit more
about how you can use graphs to connect
the dots and improve your context
engineering for your AI applications.