Microsoft's Cloud & AI Head on the AI Buildout's Risks and ROI — With Scott Guthrie

Channel: Alex Kantrowitz

Published at: 2025-10-01

YouTube video id: g6vFyuCZrVs

Source: https://www.youtube.com/watch?v=g6vFyuCZrVs

Microsoft's head of cloud and AI joins
us as we ask, is this AI buildout going
too far? That's coming up right after
this. Welcome to Big Technology Podcast,
a show for coolheaded and nuanced
conversation of the tech world and
beyond. We're joined today by Scott
Guthrie. He's the head of cloud and AI
at Microsoft. And he is the perfect
guest to give us some context on the
massive and some would say insane
buildout of AI data centers taking place
today. What does it mean? Is it going
too far? What will it lead to? Scott,
I'm so thrilled to have you on the show.
Welcome.
>> It's great to be here, Alex. Thanks for
having me.
>> All right, let me take you through the
headlines over the past couple weeks.
It's crazy that this has just been over
the past few weeks, but uh here we go.
Nvidia agreed or announced that it would
invest up to hundred billion in OpenAI,
uh starting out with 10 billion. Oracle
announced it would invest 30 billion uh
in OpenAI or a $30 billion uh build out
with with the company. Anthropic raised
13 billion. So, you know, we're just
talking about a cool 143 billion. No big
deal. Uh, is this crazy? Is this
overinvestment?
>> Well, I think there's uh it's a great
question. I'm I'm sure that's top of
mind for everyone. I think, you know,
stepping back for a moment, I would say
if you look at AI and the impact I think
it's going to have in the economy, uh,
it's going to be, I think, the most
profound technology shift in in our
lifetimes. And so I I think if you look
at um the long-term trend uh
I I don't see I I don't worry about
overinvesting. I think that you know
there will be a question on the horizon
of different companies are making
different strategies in terms of their
investment and how they get their return
in the one two three year horizon. So
you know am I going to say that every
company is perfectly timed? I don't I'm
not going to make that assertion. But at
the same time, I do think the long-term
secular trend of AI is going to be that
we're going to need more infrastructure.
There's going to be more ROI from it and
it's going to be more widely used. And
so, you know, I think directionally from
an industry perspective, um the
investments do make sense and and will
ultimately yield pretty profound um
results.
>> You think so? You think that this level
of buildout is healthy?
I I think we definitely are not nearly
at the point at which there is too much
AI infrastructure given I think the
number of AI workloads that are coming
for the world. Um and um you know I
think we're seeing on a over the last
couple years as people use AI they get
value they use it more the models get
better and people then use it even more
for new use cases and and I think at
this point across the industry with AI
we're still more supply constrained than
we are demand constrained and I think um
I you know I expect that to continue
over the next couple years as the
technology continues to evolve and as
people start to integrate AI into more
and more workflows.
>> Okay. So, let me put it bluntly then.
Um, Microsoft has a partnership with
OpenAI, has invested 13 billion
thereabouts, has the capacity to build
big data centers. Um, why did Microsoft
make the decision not to do the hundred
billion level buildout with OpenAI uh or
even the $30 billion that the company is
doing with Oracle um and leave it to
other partners to do that? Well, we we
have a great partnership with OpenAI and
it's gone back many many years and and
continues going forward and we are
building out and doing uh uh a lot of
projects with OpenAI and um uh across
the Microsoft cloud we're building out
AI data centers all over the world. uh
and you know at the same time you know
we are balancing um you know our
investment uh to make sure that it
maximizes um the AI infrastructure for
both our first party Microsoft offerings
uh that we're investing in our customers
uh AI offerings that we're investing in
and obviously OpenAI's uh offerings um
that we're deeply enabling. So, you
know, we are very invested. I don't
think it's a binary. Are we are we uh
building out for OpenAI or not? We
definitely are building out for OpenAI.
Uh and at the same time, the way our
partnership works is we're supportive of
of others u participating in that as
well.
>> Okay. Okay. But I I just want to put a
fine point on it because u again like if
if you believe that uh this technology
is going to be massively transformative
which you stated uh and that we're not
at the uh at the sort of optimal point
of the AI buildout yet that there's room
to continue to do more. Um and again
it's the partnership with what is the
consensus leader in the space. They
needed more infrastructure. There must
have been some calculation within your
group or your company to say uh is it
worth it for us to be the one that goes
out and builds this massive massive
footprint you know in partnership with
them or somebody else. So I definitely
understand there's there's multiple
stakeholders but what made Microsoft
pause on that front? Well, we we have a
balanced view and so it's we and we take
both a a long-term view in terms of
making sure that we're building out in
all the locations that we want to build
out that we're being um uh thoughtful in
terms of kind of the investment spend
and the infrastructure that we're
building and also recognize that we
don't have to do it all. And so I think
we're always trying to kind of take a a
continually balanced view of that and
and as you've seen from our capex and as
you've seen from our earnings calls, we
are investing a lot in infrastructure
and building out like crazy. Um but
again, at the same time, you know, we're
we're always constantly re-evaluating
and watching closely, you know, which
data centers in which markets to what
specifications and making sure that that
we keep, you know, good discipline as
we're doing it. um that optimizes for
both the long-term, near-term, and
midterm uh horizons.
>> Okay. I I'll just ask one more follow-up
and then we can move on. Um good
discipline. What about uh this would
have been undisiplined to have gone uh
to this level.
>> Well, I don't think it's so much the
volume level. I think it's it's uh we
one of the things that we do when we add
new data data center capacity or AI
infrastructure is you know making sure
that that we can use this infrastructure
for a variety of different AI use cases.
I think one of the things that's really
going to differentiate
um AI infrastructure companies in the
future uh is that ability to kind of
maximize yield on the infrastructure
like how are you driving down the cost
of you know tokens per watt per dollar
and um you know part of what makes the
Microsoft portfolio so unique is the
fact that we have a lot of our own AI uh
products Microsoft 365 copilot GitHub
copilot
um the work that we're doing with nuance
and and dragon and healthcare. Uh we've
got the world's largest consumer
application with chat GPT uh that runs
on top of Azure and we have thousands uh
hundreds of thousands and millions of
businesses that are also building their
own AI applications on top of us. And so
as we think about like what market are
we going to build a new data center? Is
it for training? Is it for inferencing?
Um and uh you know how do we make sure
that that infrastructure is going to be
maximally used? You know we we feed in
kind of each of these different customer
scenarios into our calculus and there
are certain tranches of capacity that
we're happy to build out because we can
see very clear line of sight in terms of
how we're going to maximize the usage
and the revenue from it. And there's
others that were maybe less uh uh likely
to see the immediate or the ROI that
we'd like. And so we try to be
disciplined about it. Um as we've kind
of shared in our blog post, you know, we
do kind of look at every request first
and we do have an opportunity on that.
And and as you've seen from our capex,
uh we are swinging in a lot of
opportunities. Um but that doesn't, you
know, we're not going to be undisiplined
and say blanket we're going to do
everything. you know, we we know that um
you know, some opportunities will have
uh more certain returns than others, and
we're trying to make sure that we
maximize our focus around those.
>> You know, as as we're talking, I'm kind
of laughing at myself because, you know,
my my question is basically boiling down
to you've like talked about your capex.
Isn't Microsoft expected to spend like
80 billion on infrastructure this year
or in the neighborhood?
I think that's what we shared in our
last
>> and I'm like well why aren't you doing
another 100red billion um but but the
fact that you're not as actually very
interesting and um and it goes to a
point that you that you just made um and
and I'm trying to read behind the line
between the lines and you tell me if I'm
if I'm getting this right. uh you think
you're you're talking about where you
invest and where you're pretty sure
you're going to get an ROI and um to me
if I'm sitting in your shoes the
question I would be asking is uh is it
worth spending you know all that money
on training where there where there's
been a a lot of um noise about
diminishing returns of training larger
models with these unbelievably massive
data centers now you have the startups
uh like OpenAI and uh Anthropic, you
know, their their belief in the scaling
laws seems unabated. Uh and so the
numbers get bigger and bigger and uh
they seem to believe that they'll
continue to get an exponential return
from training these bigger models. But
is your decision in terms of being
disciplined based on a belief that
you're not you're not sure if if scaling
up will continue to work and therefore
it's too big of a risk to make such a
large bet on uh training an even bigger
model in a bigger data center.
>> Well, I think there's there's a couple
different elements of that. I think one
is um recognizing that you want to have
the best models. So, you know, training
is super important. uh because if you
don't have the best models then you know
your actual ability to monetize AI goes
down you know at um uh you know then and
you know part of what makes our
partnership with OpenAI unique is the
fact that we do have access to the best
models frankly whether they're trained
on our infrastructure or anywhere else
um you know that's part of our um
partnership that's really important um
and I also think when you think about
training
training is evolving uh from maybe where
uh simplistically we'd think of training
a couple years ago of you you do
training in one place and then you do
inferencing where you are uh executing
the models and building applications.
You know there's now multiple types of
training. There's pre-training, there's
post-raining, uh there's reinforcement
learning, there's fine-tuning. Um
there's a lot of new techniques that uh
both sometimes require lots of
contiguous infrastructure and sometimes
requires lots of infrastructure but
sometimes it's smaller sizes uh that can
be used for very specific tasks. Uh and
so when we think about the investments
of our infrastructure we're trying to
think about all of this and compose it
all end to end. Um, you know, for us
that means for example, we want to make
sure that we have lots of inferencing
capacity because that ultimately is how
customers pay us. Um, and how ultimately
you make money from any product AI
product that you build. Um and I think
increasingly on the uh inferencing side,
you know, one important element is the
geopolitics of the world have gotten
complicated over the last many years.
And uh you know, customers in Europe
want to make sure that their AI is in
Europe and the customers in Asia are
going to care about their AI in Asia.
Obviously, the customers in North
America and the United States are going
to care about their AI being delivered
uh uh in in North America. And so, you
know, even as we build out our
infrastructure, we want to think about
it not just narrowly as we want to have
one giant pool all in the US. We need to
kind of be distributed around the world
to kind of meet those uh geopolitical
needs and to make sure that our AI is as
close to the customers that are going to
be using the AI as possible and can can
meet all of the data residency and data
sovereignty needs. And so even if you
look at our infrastructure builds around
the world, we have regions in more
countries in more locations than any
other right infrastructure provider. And
um again, as we balance out the
investments we're making on AI info, you
know, we're trying to keep that in mind
versus narrowly put it all in one
location.
>> I totally understand that, but I have to
go back to the diminishing returns of
training uh question. What where do you
stand on that? Well, I think if you look
at training broadly, um, I think you're
going to continue to see more value from
the models by doing more training. But
kind of going back to my answer earlier,
I don't know if that's always going to
be pre-training. I think increasingly
lots of post-training activities are
going to significantly change the value
of the model. And so by post training I
mean take the base model and how do you
add financial data or healthcare data or
something that's very specific to an
application or a use case. What's nice
about post- training is that you don't
have to do it in one large data center
in one location. And so part of the the
technique that we've been focused on is
how do we take this inferencing capacity
around the world and a lot of it is idle
at night as people go to sleep. you
know, how are we doing increasingly
post- trainining in a distributed
fashion across many many different
sites? Uh, and then when employees come
to work in the morning, we serve the
applications. And so having that kind of
flexibility and being able to
dynamically schedule your AI
infrastructure so that you're maximizing
revenue generation and training ideally
in a very swappable dynamic way. I think
is one of the things we're investing in
heavily and I think is one of the
differentiators for Microsoft.
>> Okay. Okay, but you'll forgive me for
going back to this uh scaling
pre-training question. Um
>> I'm just trying to see what you believe
here and uh you haven't said it
outright, but from your answers, it does
seem to me like you believe that uh
spending wildly on scaling pre-training
is a bad bet.
>> I wouldn't necessarily say that. I think
we've definitely seen as the the scale
infrastructure for
um pre-training has gotten bigger, we're
we are seeing the models continually
improve and and we're investing in those
types of pre-training
um sites and infrastructure. You know,
we recently, for example, announced our
fairwater uh data, you know, data center
regions around the US. We have multiple
fair waters um and you know we we did a
blog post recently of one of our new
sites in Wisconsin and these are you
know hundreds of megawws
um uh hundreds of thousands of the
latest GB200s and GB300 GPUs and are um
yeah we think the largest contiguous
block of GPUs anywhere in the world um
in one giant training infrastructure
that can be used for pre-training um and
So we're investing heavily in that uh as
you could see kind of from the the the
photos uh from the sky uh in terms of
massive infrastructure. Um and you know
we do continue to see the scaling laws
improve. Now will the scaling laws
improve linearly? Will they improve at
the rate that they have? I think that is
a question that everyone right now in
the AI space is still trying to
calculate. Um but do I think they'll
improve? Yes. Um, and the question
really around what's the rate of
improvement on pre-training? And I do
think with post- training, we're going
to continue to see dramatic
improvements. Uh, and that's again why
we're trying to make sure we have a
balanced investment both on pre-training
and post-training infrastructure.
>> And and yeah, and just to parse your
words here, it's uh you can see
improvement by making by doubling the
data center, but that's why I use the
word bet because are you going to get
the same return if it doesn't improve
exponentially? and just improves on the
margins and that I think is the big
question right now right
>> it's a big question and and you know the
thing that also makes it um uh the big
question is uh it it's not like a law of
nature that's immovable and so
there could be one breakthrough that
actually changes the scaling laws for
better and and there could be a lack of
breakthroughs that means again things
will still improve but do they improve
at the same rate that they historically
did uh from a raw size and scale
perspective and that is the trillion
dollar questions.
>> Okay, great. I do want to get to the ROI
of of uh AI spend uh in in a moment. Um
you know, it's always great to have a
chance to speak with someone who's in a
position like you are within Microsoft
because we get a chance to like take
some headlines and uh which might paint
a portion of a story and then ask you
what the truth is. Uh there were some
stories over the past year talking about
Microsoft had like cancelled options to
build data centers in certain locations
and people took those headlines and they
read into it that there was no demand
for AI or that um uh it wasn't going as
well as Microsoft's telling us. Uh but
but what is the what is the reason for
um why those those data centers that
there were the options and they were
canceled? What what happened there?
Well, we're we're constantly
um well, I think in general the
headlines were focused on things that we
canceled as opposed to all the things we
signed. Um and so if you look at a given
>> It's amazing how news works that way,
right? If it bleeds, it leads. So
>> if you um if you look at kind of the
overall investments and certainly if you
look at the overall capex spend, um it
has been going up uh and up and up and
so um as has again the revenue that
comes from it. And so I think um I would
kind of focus on the overall picture as
opposed to individual tanches or
individual projects that we potentially
made decisions on. Now you know the
thing that we did do and we continually
do is look hard at every single
investment decision we make and we don't
take this level of investment and this
level of project and infrastructure
lightly. It's critical that we uh invest
wisely. Is it critical if we invest that
we make it successful and that we bring
it to market on time with the right
quality and the right security? Um, and
it's critical that we have the right go
to market to monetize it. And so, you
know, part of our calculus that we do as
a leadership team is constantly looking
at the variables for all of those. And
there are places and times when we slow
down or pause projects and there are
times when we accelerate projects
somewhere else. Um and kind of going
back to my comment around the world, you
know, the also the
um you know regulation geopolitics of of
how AI is going to be used going forward
has changed quite a bit and what Europe
thinks about where GPUs can be based has
evolved quite a bit I'd say in the last
12 to 18 months and I think it's going
to continue to evolve around the world.
And so even as we think about the
investments we're making, we're also
being very very thoughtful in terms of
where geography based are we investing
so that we can again maximize um the uh
AI tokens we can serve in in real
production applications. Uh and then
ultimately use that maximization to uh
ensure that we're delivering a good
return on investment for every capital
dollar we spend.
>> Okay. And I I have some technology
questions for you, but um just to keep
on speaking about the financing of this
stuff because it's so important. Um so
there has been some interesting
reporting about how uh the AI
infrastructure buildout has begun to be
funded by debt, not just profits. Great
story in the Wall Street Journal this
week. It says, "Debt is fueling the next
wave of the AI boom." I'll read the
beginning. In the initial years of the
AI boom, companies uh comparisons to the
dotcom bubble didn't make sense. Three
years in, growing level of debt are
making them ring truer. Early on,
wealthy tech companies were opening
their wallets to out joust each other
for leadership in AI. They were spending
cash generated largely from advertising
and cloud computing businesses. Uh there
was no debtfueled splurge on computing
and networking infrastructure like the
one that inflated the bubble two and a
half decades ago. Uh however uh that is
starting to happen. Now uh OpenAI's deal
with Oracle um is uh has been uh pushed
Oracle to start taking on debt. Um they
say this is according to the story.
Analysts at Key Bank Capital Markets
estimated in a recent note that Oracle
would have to borrow 25 billion a year
over the next four years. Um, obviously
you guys are not Oracle, but um, you
know, you're you're watching this happen
as it plays out and see the parallels to
the dot boom, I'm sure, is not not fun.
You've been at Microsoft for I think 27
years,
>> 28 years,
>> 28, sorry, I don't want to miss that
that last year there. Um, so you've seen
it. Uh, Scott, uh, is this this seems to
be an issue at least from the outside.
What do you think about it being on the
inside?
Well, I think I um I think uh obviously
there's a tremendous amount of spend
from lots of different companies and I I
would say um yeah, the thing I can speak
most to is what we're doing. Um and kind
of uh you know per my comments earlier,
I think we're trying to make sure that
we have a smart um investment play uh
and a long-term u strategic play that
allows us to ride the AI revolution that
we think is going to transform the world
and do it in a way that leverages some
of the strengths that we have at
Microsoft which is we have a very good
cash flow. We have a very diverse
portfolio of businesses in particular in
the commercial enterprise space. Um
whether it's cloud infrastructure,
productivity applications, business
applications, security etc. All of them
are going to be transformed by AI. uh
and you know if you look at say to your
comment earlier on the Wall Street
Journal post I think if you even read
further in the post you know it does
show the ratios for different companies
and there are some companies that are
400% debt to equity ratios and
>> that would be Oracle
>> uh and then there are other companies
that are are much smaller and that would
be Microsoft and I think you know we
want to make sure that we're not um and
I think again based on our capex spend
and uh uh and the rate at which our
capex spend is going up. You know, we're
not going to sit on the sidelines and
not be bold as we invest. And at the
same time, you know, I think the thing
that our investors expect and ultimately
I think every investor of every company
will expect is to see that revenue
growing um in terms of AI services and
products that are being delivered in
terms of net revenue recognized in a
quarter and making sure that the
proportionality of that to the spend and
in particular to the obligations that
maybe are being undertaken with debt are
balanced. And um yeah, I can that's the
thing that we've been focused on. I
think um you know if you look at our
last quarterly earnings I think people
were pretty pleased with the getting the
balance right there and you know every
every quarter going forward people are
obviously going to be looking at making
sure that that balance is right so that
they see us investing for the long term
and going to win and at the same time
doing it in a way that um is sustainable
and allows us to kind of ride through
um
uh you know the ups and downs that
inevitably will happen over the next
many years as this technology you know
transforms the world.
>> What are the consequences if this goes
wrong with the debt?
>> Uh well
>> you're obviously not taking on the same
amount of debt. So there's a rationale
behind it. What happens if Yeah. it
breaks?
>> Uh well I mean we have the ability we're
not constrained. I mean our borrowing
costs ironically right now are low.
>> Yeah.
industrywide, big picture, industrywide,
not Microsoft specifically.
>> Well, I think the thing that um uh that
we
as an industry, I think, you know,
again, you need to have that thesis of
how you're going to use the
infrastructure and is it do you have I I
would focus less on the megawws
um that sometimes get reported in the
press and more at where are those
megawatts and what are you going to do
with those megawws? Is it going to be
ultimately capacity that you can use to
serve customers? Is it to build better
models that help you serve customers?
And you know what is the line of sight
in terms of the the product services and
revenue that comes from it? And I think
that's a place where again between
chatbt which is the number one AI app in
the world, between Microsoft 365 which
is the number one enterprise AI app in
the world and between GitHub which is
the number one developer AI app in the
world. I feel good that we have
applications using our infrastructure
and maximizing it. And I feel good about
the investments we're making in terms of
capital spend and buildout in the right
locations to kind of continue to do
that. And I think not every company
probably has that
um level of
um game plan and I don't think that
maybe not every company is probably
doing the same level of thoughtfulness
of that and you know at some point you
know different companies will probably
be hit by it but you know we're very
focused on what we do and how do we make
sure that we stay um aggressive yet
disciplined and make sure that we get
that balance right. All right, I want to
take a quick break and then I'm going to
ask you a couple uh technology questions
about the state of the buildout, GPUs,
uh custom silicon, and uh and then maybe
we can get a little bit into this ROI
question. In fact, we will we have to
talk about the ROI of AI. We'll do that
right after this. And we're back here on
Big Technology podcast with Scott
Guthrie, the head of cloud and AI at
Microsoft. Uh Scott, we have a Discord
here at Big Technology and I asked some
of our members, you know, what they
would ask you and we got a flood of
excellent questions. Uh and I think I
think the they were great because they
focused on the technology. Some
questions that I don't think you hear
too often um in the common conversation
about this technology. So, uh you're the
perfect person to ask. I'm going to ask
them to you. One of our members asked,
"What is the working life of a GPU and
how long until they burn out?" they are
there use cases for GPUs once they are
no longer top of the market right we
hear often about okay well unlike the
the um the laying of the fiber the GPU
depreciates after a couple years so I
think this is a a pretty important
question can you can you hand that
tackle that for us
>> yeah I think kind of going back to the
comments we had earlier on balance I
think as you think about your GPU
buildout one of the things that we think
about is the lifetime of the GPU you and
how we use it. I think you know what you
use it for in year one or two might be
very different than how you use it in
year 3, four and five or six. Um and so
you know I think that is something where
um you know so far we've always been
able to use our GPUs even ones that we
deployed multiple years ago for
different use cases and and get positive
ROI from it and that's why our
depreciation cycle for GPUs is what it
is. Um but I do think that's you know as
we build out um our infrastructure we
are definitely consciously thinking
about that because uh you don't want to
have your entire fleet in two years
suddenly have to be replaced because um
you know that that that would be
expensive and so you know we are very
thoughtful on that um and again that's I
think I talked earlier about um
different training I also think even as
you think about training we we often in
the past used to monolithically call
training training. There's lots and lots
of different training use cases now.
There's pre-training, there's synthetic
data generation that goes into training,
there's post-raining with RL and
fine-tuning and and other different
techniques and um you know having
infrastructure that's very fungeable and
that you can use for a variety of
different training infra scenarios and
at the same time be used for inferencing
where you ultimately enable an
application to perform a query or
perform an AI uh invocation.
um is key and I think that's uh
uh does go goes beyond just the the GPUs
even though I think people often
narrowly focus on that. It also is
around the data center architecture.
It's around the storage and the compute
that's near the GPUs. And it also really
comes into play with the network because
if you are for example building one
large data center that only does
training and it's not connected to a
wide area network around the world
that's close to the users, it's hard to
use that same infrastructure for
inferencing um because you can't go
faster than the speed of light. And so
someone elsewhere around the world that
wants to call that GPU if you don't have
the network to support it um
uh you can't use it for those
inferencing needs. And so again going
back to kind of some my comments earlier
about how we're trying to be very
thoughtful about where we place
infrastructure and how we um maximize
the utilization. We're definitely
thinking of that not just for this year
or this quarter, but thinking about it
on that four or five or six year horizon
for how we want to basically leverage
and and use it.
>> Okay. Uh here's another question. Are
there any cool technological
breakthroughs that would change the
economies of data centers as we know
them now? GPUs started as uh graphics
processing units for video games. Uh are
there resources you found that might do
as well but with fewer constraints?
Well, I think one of the biggest changes
that's happening right now from a data
center perspective, and you're seeing
this with the latest Nvidia GPUs, and I
think you're going to see this in a more
profound way over the next um two years,
is the shift from air cooled data
centers where you use, you know,
effectively giant air conditioning units
or chillers um to a uh liquid to liquid
cooled facility water where you're
actually pumping in water in order to to
cool the equipment uh in a in a closed
loop circulating system. So in other
words, you feed in cold water, you run
it over the GPUs effectively, extract
the water, cool it down again, and then
do it again throughout the building.
That's a massive technology change. Uh
and it does mean that older data centers
um that are air cooled, you know, they
can't just drop in liquid cooling uh and
be effective. And so that is something
that I think everyone that's in the AI
space is designing for and needs to be
thoughtful of again with their
infrastructure projects to make sure
that they're ready for that technology
shift. It also is going to have a big
difference and big impact in terms of
the staffing. Um when you have a
aircooled data center you'd have very
few employees often per server. when you
all start to involve water and liquid,
you know, it's not massively more, but
at the same time, it does change
staffing because there are more things
that break when you have pipes that are
actually uh continuously flowing liquid
uh into a data center. So, it's there's
a lot of technology shifts that are
happening right now behind the scenes
beyond the GPUs. And then obviously GPUs
are the things that dominate the press
in terms of innovations both in terms of
the silicon
but also in terms of the network because
at the end of the day if you have a chip
that can process a lot more information
but you don't have the ability to get
that information to the chip and extract
it or have it communicate with other
chips you know then you don't get the
the yield out of it. And so um I think
it's fascinating right now in technology
the pace at which so many things are
evolving so fast both with the GPUs to
the question but then also the data
centers even the power and cooling
infrastructure for the data centers and
the network and um you know as a
technologist it's it's exciting times
>> right you mentioned staffing so I want
to ask a follow-up on that front u I
think for those who don't live those who
are not in this uh deep into it. There
is a perception that data centers um
they're placed uh near communities in
some cases. They use up a lot of water.
They don't provide a lot of jobs. Is
that a misconception?
>> I think it's a misconception. Um uh I
mean
>> like some numbers to sort of flesh out
what they actually bring to a community
that they appear next to.
>> Yeah. Yeah, I mean, we we've talked
about with our Wisconsin uh fairwater
site that we did some press on recently
and um uh talked about uh including with
the governor of Wisconsin and and others
that were attending. You know, it's it's
thousands of jobs that we've created on
the construction of the site. Um uh I
think we've we've shared over 3,000 um
jobs and these are these are very
skilled jobs. These are um you're
talking about electricians, you're
talking about plumbers, you're talking
about welders, you're talking about
skilled tradescraft
uh and um uh and you know highquality
jobs and and I think you know if if you
look I mean we have phenomenal work
site, phenomenal workers there and a
phenomenal safety culture which has
allowed us to attract some of the the
best workers to work on that project. I
think what people are missing sometimes
when they say, "Okay, but when the
project's done, how many people are
going to be in the data center um and
there will be, you know, hundreds of
people that will be in the data center."
What people are missing is the fact that
right next to that data center, we're
building another data center. And so
those thousands of people that have been
working on the first Fairwater data
center, we just announced are now going
to be starting work on the second one.
Uh and then after the second one, we
will do a third one. And and if you look
at the land and you look at the power
we've accumulated in that area, it's
multi- gigawatts of land um or multi
gigawatts of power and it's it's an
awful lot of land. And so you you're
going to see us continue to employ
thousands of very skilled trades
um uh workers in that community. And as
each one of those data centers comes
online, we're going to add net new
employees that will actually operate it
and uh and manage it. So, you know,
that's would be an example, I think, of
a community and we, you know, we have
over 400 data centers around the world.
So, they're not all that size obviously,
but um replicating that and I think as
more infrastructure gets built out,
you're going to continue to see um not
just jobs created, but well-paying jobs
that really require real uh real uh
trade craft. Do you feel what do do you
feel the um I don't know the right word
to put it the pressure of competing with
China um because from my understanding
China has a much looser regulatory
approval process and they're just
stacking you know data centers they have
abundant electricity uh in the United
States in particular um I imagine Europe
is the same way uh that that is not the
case
so what's it like oh
>> certainly I think the world has a very
different regulatory approval process. I
mean I think one thing that when I talk
to people and they say how can you build
data centers faster
um you know there's obviously things
that we can do from a technology and are
doing from a technology and from a
manufacturing perspective but you know
candidly here in the US the longest
part of building a data center is
getting permitting. It's not actually
the construction. It's it's making sure
that you you know uh get permitting
approval for all the steps that you want
to take. And you know, different states
and different parts of the country have
different regulatory environments. And I
think even if you look at a sort of a
heat map, if you will of where data
centers are being built in the US, you
definitely see pockets. And I would say
some of that approximates to where there
is land and where there's power and some
of it really, you know, closely
correlates with where it is easier or
faster to kind of complete the
permitting process. Um, you know, in
Wisconsin, we had a, you know, a
phenomenal partnership with the governor
and the local county. Um, we were able
to to purchase some land and power that,
um, a manufacturer was previously going
to use and they they pulled out of a
project. And so, you know, I think that
the the the local communities recognized
if they weren't able to work with us,
you know, they were gonna lose jobs and
and have, you know, impact on the
community and they leaned in with us
and, uh, you know, can't say enough
positives in terms of the speed with
which, you know, we went through all the
process. We got all the approvals. It
was a very thorough process, but it was
streamlined so that we could move fast
and that we could actually help ensure
that jobs weren't lost and that instead
they were created in the community. And
I think there's more opportunities for
public private partnership like that
that we'd certainly welcome as part of
it.
>> Okay, another Discord question. Uh what
time frames are you looking at to get
ROI on these investments? So when would
you stop investing if prior investments
weren't showing returns? And how do you
know when it's time to stop building?
Well, I think you know at the end of the
day I think you know every every quarter
we we share our revenue growth and we
share our capex spend and you know to
some extent I think the you know markets
keep um uh companies honest in terms of
um uh that balance and u you know
sometimes markets can be slightly
irrational at times but in the limit the
markets keep you honest and
>> um yeah that's a big part of why we
focus so much on making sure we get that
balance right. Make sure we're we're
again investing for the long run. I
don't think anyone if you look at our
capex spend and our our commitments and
our investments would say that we are um
uh not being bold. But at the same time,
you know, we have a report card every
quarter where we need to kind of
demonstrate and uh prove not just with
press releases, but you know, here's how
much revenue growth we had. You know,
last quarter we grew Azure at 39%
year-over-year on a very large number
and a lot of that was driven by AI and
then also driven by the other systems
that come with AI. um because there are
databases and there's compute and
there's storage sold with that AI and
you know I think investors were happy
with the
um both the spend and the aggressiveness
that we were building out but also the
return and you know I think that's going
to be true um you know forever and you
know making sure you get that balance
right and and again as part of that
balance in markets want to know you're
investing to win the long run uh and at
the same time they want to make sure you
have some level of discipline And and I
think our portfolio the the balance that
we have both across the the products we
build but then also the fact that we
have the largest AI product in the world
called Jad GBT running on top of our
cloud. Um gives us a unique opportunity
to get that balance and that growth and
that investment right.
>> Uh is is chat GPT by the way going to
stay on Azure even though OpenAI is
making these partnerships with uh Nvidia
and Oracle?
>> Uh yes.
>> Okay. All right. It's good to get
something definitive on that. Um, you
know, you mentioned your 39% Azure
growth and you know uh I'm uh I'm
looking at your quarterly numbers every
every uh quarter and uh often talking
about them on CNBC and the numbers are
are massive. And the other side of it
though is so that's spend coming from
clients, right? And there have been
multiple studies that have come out
recently that have talked about how
enterprises aren't getting the ROI that
they've anticipated on their AI projects
yet. Uh when you see those studies, do
they ring true to you? How do you react
to them?
>> Well, I think um uh I think when you say
AI in general, it's a very broad
statement. Um, and
>> this is gener it's this is obviously I
mean not obviously it's in large part
this is generative AI where companies
everywhere have tried to adopt LLMs and
try to put some version of that into
play in there and it's not recommener
engines basically.
>> Yeah. But I think what you need to do is
double click even further from Genai to
GitHub copilot or healthcare or uh
Microsoft 365 copilot or security
products built with geni. I do think
ultimately, you know, the closer you can
kind of doubleclick on is this really
delivering ROI, then then you have much
more precise data because I do think a
lot of companies have dabbled or done
internal kind of I'll call it proof of
concepts and some of them have paid off
and some of them haven't but um um yeah
and and but I think ultimately a lot of
the solutions that are paying off that
we we continually hear from our clients
and our and our customers is um uh you
know a bunch of the applications for
example that we've built um I think
similarly you know a bunch of the
applications that our partners have
built on top of us and you know
ultimately the Azure business is um you
know we get paid based on consumption um
it is it's a consumption based business
meaning if people aren't actually
running something we don't get paid it's
not like they're they're pre- buying a
ton of stuff um you know we recognize
our revenue based on when it's used and
so you know the Good news is when you
look at our revenue growth, it is um you
know it's not a bookings number. It's
actually a consumption number and uh you
can tell that people are consuming more
and um you know you know the last two
quarters our our revenue growth's
accelerated on a big number and that
that is um you know a statement of the
fact that I think people are getting a
lot of ROI at least with the projects
that they're running on top of our
cloud.
>> Yeah, I think that's an important point
to to bring home. It is consumption
based. Um, so you you talked a little
bit about water cooling versus air
cooling. I love the term for the air
cooling. It's called chillers and that's
what my friends in high school called
ourselves, you know, back in the day.
Um,
and um, I want to end on the GPU side of
things or the the silicon side of
things. Um, what do you think the
potential is for for custom silicon in
the AI world? I mean, like we talked
about previously, GPUs were designed for
gaming. they happen to do parallel
processing actually ended up being
really good for uh you know large
language models the training and the and
the inference. Um what's your
perspective on on whether this industry
is going to continue to run on that type
of chip uh and what the potential is for
custom silicon?
I think um a couple things I think one
is it um I think that increasing the
um
the the number of tokens you can get per
watt per dollar is going to be the game
over the next couple years and and
maximizing
u the ability of our cloud to deliver
the best
uh
volume of tokens for every watt of power
for every dollar that's spent where the
dollar is spent on energy, it's spent on
the GPUs, it's spent on the data center
infrastructure, it's spent on the
network and it's spent on everything
else is is the thing that again we're
laser focused on and uh it is you know
there's a bunch of steps as part of that
GPUs being a critical component of it.
Um, and you know, one of the things that
our scale gives us the ability to do is
to invest for uh kind of nonlinear
improvements in that type of
productivity and that type of yield. You
know, if you've got, you know, a million
dollars of revenue on couple hundred
GPUs, you're not going to be investing
in custom silicon. Um, when you're at
our scale, you will be. Um, and you're
not just investing in custom skeleilican
for GPUs for pre-training or for
inferencing. You're you're looking at
what could we be doing for synthetic
data generation with silicon. What can
we be doing from a network compression
perspective with custom silicon? What
can we be doing from a security
perspective? And we have bets across all
of those, many of which are now in
production and are actually powering a
lot of these um AI experiences. In fact,
I think every GPU server that we're
running in the fleet right now um is
using custom silicon at the networking
compression storage layer that we've
built. Um now the GPUs themselves are
also going to be a prize that people are
going to try to optimize like the actual
instructions um for doing um the GPUs.
Nvidia is a fantastic partner of ours.
Um we're probably one of if not the
biggest customer in the world of theirs.
Um and um uh we partner super deeply
with Jensen and and his team. You know,
at the same time and and partly why
they're so successful is they're
executing incredibly well. Um you know
at the same time if you look at the
history of silicon
um not every silicon company or it's
rare to have a silicon company that
every single year is is doing the
absolute u perfect work that's
differentiated and and kudos to Jensen
for what he's done and I know he's going
to keep trying to do it going forward
but you know there will be other
opportunities from other companies where
people are going to look for a niche
that's going to be big enough in this AI
space to be truly differentiated versus
what Nvidia is delivering and then we're
doing our own silicon investment in
house. So, because we're going to be
going after those same opportunities and
and ultimately the way we've tried to
build our infrastructure,
uh, none of our customers know when
they're using Microsoft 365 or GitHub or
or any open models what silicon they're
running on. And we're going to be
constantly tuning the use cases based on
the applications.
>> And if we find ways that are
breakthroughs, we're absolutely going to
be taking advantage of them for those
use cases. And again, at our balance of
scale and our balance of use cases, I'm
very confident that we're going to find
use cases where custom silicon will make
a difference. And I'm also very
confident we're going to continue to be
a great partner to Nvidia and others in
the world that are going to be selling
us great solutions.
>> All right, Scott, I want to end on this
because I'm always I've always been
curious about the human aspect of this.
like you're going out and working on
designing your own chips that are trying
to be you know better than GPUs for
certain parts of this AI uh uh
application layer uh and training and
then you're you said one of Nvidia's
biggest customers if not its biggest
customers. So is this like a situation
where like you go to Jensen and you're
like we're gonna just both give a shot
at building the stuff and may the best
chip win and it's friendly like friendly
competition or is there any awkwardness
in there cuz you're like kind of
building the thing that is making them
the most valuable company in the world?
>> Well, I think probably different
companies handle that differently. I
think the nice thing about Microsoft is
a we've been around a while and I think
also we're we're you know we compete
almost in every market in some way shape
or form. So like there's none of my
partners that I'm not also a competitor
with. I think is probably a true um form
and the important thing is I think you
have that enterprise maturity to be able
to recognize you know I want Jensen to
do the best possible work because it's
going to benefit me. Um and we've leaned
in. We were the very first cloud to
deliver uh live GB200s, you know, which
is a massive architectural shift for
Nvidia. That's the first of their liquid
>> gracewell.
>> Yeah. And we were the first
>> the first one running, first rack
running, the first cluster running, the
first data center running of any cloud
or neo cloud provider in the world. And
so, you know, that's an example where we
really leaned in and moved at the speed
of light together. And we're going to
continue doing those types of projects.
and at the same time, you know, he
recognizes and understands we're going
to be doing lots of things. And I also
recognize he's going to work with other
providers as well. So, I think the
ability to kind of keep a complete
thought and recognize it's not zero sum
on every single decision. And that at
the end of the day, um, you know, it's a
market, we're all going to compete and
we're also going to partner and, um, you
know, I think we have the maturity at
Microsoft to do that. Again, the balance
I keep I think I've said balance
multiple times. I do think balance in
life but especially in business um and
especially in technology that is the
devil's in the detail but if you can get
that right and do it consistently those
are the companies that win and those are
the companies that um really you know
have the ability to set the agenda and
and that's what we're focused on.
>> Well Scott, uh I just want to say thank
you for taking the time. I know you
don't do this often, so I appreciate why
why did you uh say, "Okay, hey, I want
to come out and speak about this today."
>> Uh well, a bunch of people internally
said, "Hey, are you going to talk to
Alex?" And so
>> that is always a good advice to follow.
>> And so I Okay. And um so it's uh it's
fun to fun to get a chance to do and uh
really I really enjoyed the
conversation.
>> As did I. Yeah. Thank you again for
taking the time again. I know it's rare
for you to come out and speak about
these things. uh you're running a
massive massive and fast growing
business and so it was great to be able
to speak with you and get into peak get
a peak into it today and a look as to
what the rest of the industry is doing
and your perspective on that. So thanks
for coming on the show Scott. Appreciate
it.
>> Thanks for having me Alex.
>> All right everybody, thank you so much
for listening and watching. We'll be
back on Friday to break down the week's
news with Max Ze of Techrunch. It's
going to be a great episode and we hope
to see you there. Thanks again and we'll
see you next time on Big Technology