AI Consulting in Practice – NLW, Superintelligent, @AIDailyBrief⁩

Channel: aiDotEngineer

Published at: 2025-12-18

YouTube video id: ehQFj6VmuI8

Source: https://www.youtube.com/watch?v=ehQFj6VmuI8

[music]
Today I'm excited to talk uh about
something a little bit different. So
right now uh there's been the last
couple of months have been an
interesting time in AI. There's been a
sort of surge in the air uh the
narrative of an AI bubble. A lot of it
driven by dubious studies uh like the
MIT report. And so what I wanted to do
today is get into not so much the
practice of consulting and transforming
but what organizations are actually
finding value in right now. So for those
of you who don't know me, uh there's
kind of two contexts I bring to this
conversation. The first is as the host
of the AI daily brief which is a daily
uh news analysis podcast about AI. The
second is as the CEO of super
intelligent which is an AI planning
platform. So the different perspectives
are sort of very high level macro
thinking about the news that's happening
and then a much more kind of ground
level view where we're spending a ton of
time interviewing executives about
what's going on inside their
organizations. And what we're going to
talk about is sort of one kind of
briefly in the first part just the
status of enterprise adoption uh as it
currently stands. And two, um, and the
more interesting part is we've been live
with a study in in the market for about
a month now collecting self-reported
information about ROI around different
use cases. And [snorts] this will be the
first time uh, this week was the first
time I did some analysis on it. And so
I'm going to share what people have uh,
what people have told us around the
first kind of 2500 or so use cases that
they've shared. Um, so it should be
pretty pretty interesting stuff. talking
about kind of enterprise AI adoption
first. I'll go through this pretty
quickly because it's um pretty
well-known stuff. Uh the short of it is
enterprises are adopting AI uh in in a
growing fashion. Um pretty much everyone
is using it at least a little bit. Uh
and increasingly they're using it a lot.
Uh this year I will need to tell none of
you that there is a major inflection
around um specifically adoption in the
uh coding and software engineering.
Right? You saw a huge huge uptick in
this. Um there's a lot that's
interesting about that from an
enterprise perspective because it wasn't
just with the software engineering
organizations. Other parts of the
organization are also now thinking about
how they can communicate with code,
build things with code. Uh but that's a
huge huge theme of this year coming into
2025. One of the big sort of thoughts
that many people had was that this would
be the year of agents inside the
enterprise, right? That big chunks of
work would get automated away. And on
the one hand, I think it's pretty clear
that we didn't see some sort of mass
shift towards automation uh at large
across different functions in the
organization. But when you dig into the
numbers, there has been actually pretty
significant uh shifts in the patterns of
of agent adoption. So this is from
KPMG's quarterly pulse survey. And it's
a measure of how many enterprises that
are a part of their survey, which is all
companies over a billion dollars in
revenue, have uh actual sort of full
production agents in deployment. This
isn't pilots, this isn't experiments.
This is where they consider uh some
agent that's actually doing doing kind
of work in a in a full way. And it's
jumped from 11% in Q1 of this year to
42% in their most recent study for Q3.
So, you actually are seeing pretty
meaningful uptake of of agents inside
the enterprise. In fact, I would argue
based on our conversations that people
have that it's moved more quickly
through the pilot or experimental phase
than people might have thought. um so
much so that you're actually seeing now
a big shift in the emphasis around kind
of the human side of agents and how
humans are going to interact with agents
and it's involving a shift in upskilling
and and uh in enablement work. Um you're
seeing a decrease in the sort of
resistance to agents as people start to
actually dig in with them. You're seeing
more experiments like these sandboxes
where people can interact with agents.
So this is a big theme even if it wasn't
necessarily the dominant theme that some
thought it might be coming into this
year. [snorts] At the same time, it is
absolutely the case that many many if
not most enterprises are broadly
speaking stuck inside sort of pilot and
experimental phases. There is a lot of
challenge around moving from some of
those first exciting experiments to
something that's more scaled. Um, so
this [snorts] is from McKenzie state of
AI study which came out I think a couple
weeks ago now and you can see only 7% of
the organizations that they talk to
claim or sort of see themselves as as
fully at scale with with AI and agents.
and it's something like 62% are either
still experimenting or piloting.
Interestingly, big organizations are on
in general a little bit ahead in terms
of uh the organizations that are scaling
as compared to small organizations. This
has been a a thing that we've noticed
kind of throughout the trajectory of uh
of AI um adoption over the last couple
of years that you would think that
perhaps smaller, more nimble companies
uh would be more kind of quick to adopt
these things, but in fact, it's often
been the opposite with the biggest
organizations making the biggest
efforts. You can also see from the chart
on the bottom that there's very sort of
jagged patterns of adoption, right?
You're starting to see uh from you know
last year if you looked there's very
similar kind of rates of experimentation
across lots of different departments.
You're starting to see some pretty big
breakouts now uh with for example you
know IT operations kind of jumping out
ahead of other functions.
I won't spend too much time on this sort
of high performer piece, but I think the
thing to note because it comes back in
and in and some of the stuff that we
found with our ROI study is that you are
also starting to see a pretty
significant bifurcation between leaders
and laggers when it comes to AI
adoption. And one of the things that
tends to distinguish the companies that
are leading is that they are just doing
more of it and they are thinking more
comprehensively and systematically about
AI and agent adoption. So they are not
just sort of doing spot experiments.
They're thinking about their strategy as
a whole. They're doing multiple things
at once. And importantly, they're not
just thinking about sort of the very
kind of first tier time savings or
productivity types of use cases, they're
also thinking about how do we grow
revenue, how do we create new
capabilities, how do we create new
product lines.
Overall, it's very clear that despite
what is sort of, you know, the the the
concerns in the media that spend is
going to do nothing but increase on
this. Um, to the bottom is the KPMG
pulse survey again. And this is a an
estimation of the amount of money that
these organizations intend to spend on
AI over the next 12 months. The
beginning of the year was 114, which by
the way was up from like 88 in Q4 of
last year. It's now up to in their last
study 130 million is what they expect to
spend uh in the in the year ahead, which
obviously the the total magnitude
doesn't matter as much as the change. Um
you also see the green charts are from
Deote and you can see 90% plus of
organizations intend to increase their
spend uh on AI in the next 12 months and
as part of that I think that you're
going to see a much more determined
conversation around impact and ROI uh
which is a particularly thorny topic but
interestingly
there has been an increase in optimism
over the course of this year around the
realization of AI. So this is from a
different KPMG study, their annual CEO
survey, which interviews tons and tons
of CEOs. And if you look at the 2024
numbers, 63% of those pled thought that
it would take between 3 and 5 years to
realize ROI from their AI investments.
20% said 1 to three and 16% said more
than five. This year in that same
survey, the number that said 1 to 3
years had gone up to 67%. There were now
19% who said 6 months to 1 year. uh and
3 to 5 years was down to just 12%. So
huge huge kind of pull forward of
expectations of of ROI realization. The
challenge is that ROI is really tough.
So this is back to the poll survey. 78%
of those pled in that in that survey
said that they thought that ROI is going
to basically become a bigger
consideration in the year to come. Uh
but also 78% said that traditional
impact metrics and measures were having
a very hard time keeping up with the
with the new reality that we were living
in. And this is something that I've
heard constantly over and over from CIOS
and other people who are in charge of
these investments that the the the ways
that we have measured impact of previous
technologies and just previous
initiatives are kind of falling flat
with AI. And so that got us thinking
about the the the overall need that we
have to just have more information. I'm
not even talking about good systematic
information, just more information
around what ROI looks like, what impact
looks like, and you know, I've got this
great podcast audience. They're super
engaged. And so, we just decided, screw
it. We're going to ask them, we're just
going to ask them to report on what ROI
they're finding from their use cases.
So, this went up at the very end of
October. Uh like I said as of this
morning or when I looked last looked
we've had over a thousand submissions uh
a thousand individual organizations
rather submit something like 3500 use
cases and um this is uh some some of the
first observations that we had around um
kind of the first 2500.
So the impact categories the way that we
divided things was into sort of eight
broad categories of impact um which will
all I think be very intuitive to you
guys. time savings, increased output,
improvement in quality, new
capabilities, improved decision- making,
cost savings, increased revenue, and
risk reduction. So, basically, it was
trying to think of like kind of a a
broad simple heristic for uh for for
kind of dividing or subdividing the
different the different ways that people
are thinking about ROI. And TLDDR is
that people are finding uh ROI right
now. Um, now again, the caveats are that
this is a highly infranchised audience.
they're listening to a daily AI podcast
and they are voluntarily sharing this.
So, I think that, you know, there's
there's some caveing there, but you have
44.3% saying that they're seeing modest
ROI right now. And then you have another
37.6% seeing high ROI. For the purposes
of a lot of these stats, high ROI will
be significant plus transformational. Uh
only 5% or so are seeing negative ROI.
And keep in mind, negative ROI doesn't
mean that they think programs are
failing. It just means they haven't
they've spent more than they've gained
uh in terms of how their their
perception is. More than that,
expectations are absolutely skyhigh. 67%
think over the next year they will see
uh increased and high growth in their
ROI. [snorts] So we have a really
optimistic sense from the ground view of
where ROI is going to be in AI. Um you
even have the teams that are currently
experiencing negative ROI. 53% say that
they're going to see high growth. So
very very optimistic. Um as [snorts] you
might imagine, time savings is the
default. It's the starting point for so
many organizations. It represents about
35% of the use cases. After that,
increasing output, quality improvement,
basically all those things that you
would imagine around productivity are
sort of like the dominant categories
when it comes to these uh when it comes
to these use cases. When it comes to the
specifics around time savings, you see a
real cluster between 1 and 10 hours,
especially right around 5 hours. And I
think this is interesting to call out
because it's so obvious to all of us who
are inside building these things uh
whether you are a developer or an
entrepreneur or just someone sort of in
and around it how the the vast breadth
of opportunity that AI represents new
capabilities things unimagined yet. It's
hard to or it's easy to forget that if
you save 5 hours a week or 10 hours a
week you're talking about winning back 7
to 10 work weeks a year. Uh and that's
very very powerful. And when it comes to
a lot of these enterprises, that is a
very meaningful thing, even if it's not
what they're ultimately in it for.
Interestingly though, it's very clear
that the story, although it might be uh
have a concentration in time savings, is
about much more than time savings. So
this is the ROI distribution category uh
ROI distribution by organization size.
And this starts to get really
interesting where you can see that there
are some differences in where different
size organizations are focused. So for
example, the organization size between
200 and a,000 people has a higher
portion of their use cases concentrated
in increasing output. Now we haven't
taken the time yet to really figure out
exactly what this means or even
speculate on on what this means. But I
think it's interesting that this is a
category of organization that has often
reached a certain scale but is still
very much striving for more and so seems
to be focused more on use cases that
expand their capabilities.
Same thing with uh when you start to
divide things by role you see real kind
of variance where for example seuitees
and leaders uh are less focused on those
time savings use cases and more focused
on other things like increased output
and uh and new capabilities
in general we're finding that sea
leaders uh and just sort of seuite and
and leaders in general are even more
optimistic and excited and seeing
transformational impact than people who
are in more junior positions. Now, some
of this might be sort of selection bias
in terms of um what types of use cases
you are focused on. If you are in that
seuite, you're thinking about things
that inherently if they work are more
transformational. Uh but it is notable
that 17% of uh of the use cases that
that people in those leadership
positions have submitted uh they say
have transformational impact and ROI
already.
[snorts] Uh I'm going to skip this
because there's we don't have time for
too much. um you're seeing interestingly
uh a concentration um where the smallest
organizations are getting more of that
transformational benefit early. Um one
of the things that I want to do
following this study is maybe do a sort
of second round where we dig into what
this 1 to 50 person uh size really looks
like. I actually think that whereas
there might be a lot of similarity
between a 1000 and a 2,000 person
organization, there could be a wild
difference between a threeperson, you
know, small company and a 40 person
company. And so I'd really like to dig
into that more. But you are definitely
seeing a a lot of impact in those sort
of more small nimble moving
organizations.
Uh as you might expect, coding and uh
and software related or uh use cases
have a higher ROI than average and a
lower negative ROI than average. Um one
really interesting kind of you know
pulling on a specific category of use
cases. Risk reduction is our lowest
category in terms of the percentage of
use cases that that that was their
primary benefit. So when you're filling
out the survey, which is by the way at
roervey.ai AI if you want to check it
out. Uh you basically only get to pick a
primary ROI benefit. We didn't want it
to be super sort of um we wanted you to
pick and and hone in on the thing that
was uh seemed most important or most
significant. And so only 3.4% have risk
reduction as their primary benefit uh in
terms of ROI categories. But it is by
far those use cases are by far the most
likely to have transformational impact
as as the as as their outcome. It's at
25%. So a full quarter of those uh have
transformational ROI. And interestingly,
I was having this conversation with a
couple of my friends who work in sort of
back office and compliance and risk
functions and this has been their
experience as well where there are a lot
of uh a lot of the the the challenges
for those organizations involve sheer
volume and quantity uh in ways that that
AI can be really helpful for.
We also are finding some interesting
patterns among organizations. And again,
this is where we get into some of the
limits of this just being a whoever
walks through the door of my listeners.
We have a pretty heavy concentration
among technology, as you might expect,
industries and among professional
services, but we still have fairly
decent sample sizes for some others. And
in both healthcare and manufacturing,
the use cases are meaningfully higher
impact on average uh than the average
across all organizations. Um, which I
think is uh it was kind of worthy of
further study.
Last sort of part of this as I wrap up,
you know, a lot of these use cases as
you saw have to do with that sort of
first tier that most enterprises are
going to be in. Uh, increasing the
amount of content that you output,
increasing the quality of that content,
just finding ways to win back, you know,
your 5 hours a week. Um but increasingly
there are automation and agentic use
cases and we are absolutely seeing that
where those are the the focus where
those use cases mention certain types of
automation or they mention agents they
wildly outperform in terms of the
self-reported ROI from them that's both
on automation and it's on agents and I
think that that's sort of a a trend
towards where we're headed with sort of
the next layer of more advanced use
cases.
The last thing that uh from this sort of
first first look of observations is
there is clearly benefits and this goes
back to to what we saw with that
Mackenzie study as well of thinking
about AI and agentic transformation in
systematic cross-organizational
cross-disciplinary types of terms. um
effectively pretty much uh directly the
more use cases that a person or an
organization submitted that the the
better they tended to see uh ROI for.
Now there's lots of reasons for that but
I do think it speaks to that that core
idea that once you move beyond kind of
your single spot experiments there's a
lot of opportunity uh to to sort of grow
grow the impact of the organization. So,
like I said, that is the the first look.
Uh, it's kind of the first twothirds of
these uh of these use cases. We'll be
open for another week and then we'll
have the full study out at the beginning
of December. Um, I'm really excited, I
think, heading into next year to see how
we move from sort of generic
conversations about impact uh and our
gut senses about impact to a lot more
random experiments like this to figure
out where the impact really is and uh
and where we go next. So, look at that.
I'm going to end 27 seconds early and
really throw off the time, but
appreciate you guys all being here. Uh,
and again, if you want to check this
out, it's roicervey.ai.
[music]
[music]