I do not believe that enterprises will accept a pure third-party-hosting cloud model any time in the near future, if ever. I think that cost and data security concerns are too great, and there’s no clear and credible way to address them. I also think this is true for AI. If enterprises want to depend on AI, then the considerations that inhibit “moving everything to the cloud” will also constrain the way they adopt/host AI. Cost and “security”, meaning the security of critical business data, constrain the use of “cloud AI” services in business-critical missions, and it’s these missions that would be most likely to build an AI business case. Thus, self-hosting AI seems to most enterprises to be a foregone conclusion.
AI vendors would love to see enterprises decide to host AI. The sale of GPUs would skyrocket, and AI networking would drive major changes and opportunities in the switching space. A lot of vendors at the chip and switch level already bet on enterprise AI hosting, and the software opportunity that it would generate hasn’t even gotten started at this point.
But suppose an enterprise wants to host AI. I got 284 comments from enterprises on self-hosting the LLM form of AI this year, and only 46 enterprises indicated they were “confident” they had a handle on it. An additional 58 thought they understood how to go about it, which means less than 40% believed they were in a position to plan for deploying LLM AI in house. What this group thinks, though, might drive the whole AI space in the future, since it also represents the only enterprises who have actually gotten AI projects approved, and the only ones who have made an LLM AI business case successfully, or even believed they were on track to do that. Let’s look at the views offered by these 104 enterprises on self-hosting LLMs.
How much AI is likely to get hosted? The measure most cite is the number of GPUs they believe would be used, and 88 responded. Of that group, only 2 saw themselves deploying over a thousand GPUs, 36 thought they’d deploy between 500 and a thousand, 41 between 200 and 500, and only nine less than 200 GPUs. The number the remaining 196 who hadn’t deployed or planned deployment explicitly came up with averaged just under 250.
Only 11 of 104 enterprises who had deployed or planned explicitly to deploy LLM hosting had a single application in mind, and 6 of this group said their mission was customer support chatbots and 5 business analytics. 86 said they had justified LLM AI with a starting single application but then applied it to multiple missions, and five said that they started with multiple missions. Among 122 enterprises with deployments or mature plans for application of LLM AI, the most popular initial missions were customer-facing chatbots, business analytics and operations support (with netops leading).
Operations missions seem to generate the smallest deployments and the slowest rate of increase in the size of deployment as new missions are added. This seems to come about because the tools and plans associated with operations use of LLM AI don’t readily expand to other missions. Both customer-facing and business analytics AI seem somewhat symbiotic, and so either mission preps enterprises to consider the other.
Most (99 out of 104) enterprises planned for LLM AI to form an autonomous cluster in a data center, and of 88 who had deployed or were about to, 87 used/planned Ethernet connectivity within their GPU cluster. In all cases but 2, these AI clusters were in the main data center, and those that were not were in a separate facility for reasons of power and cooling.
What’s in such a cluster? Obviously, GPUs, but obviously standard servers, database systems, network switches, software, security…a lot of stuff. In fact, 27 companies said their AI clusters were mini-data-center elements, and that they required different tools for management, deployment, and security. This group also said that training requirements were “very different” from the requirements to run a model, and they wished there was a way to securely augment their AI resources during training. Cloud AI, for security reasons, wasn’t considered an ideal answer.
Of 59 enterprises who had selected a source of AI software, 31 said a vendor was the driver/integrator of the project and had provided the LLM. Only 8 selected a licensed “commercial” LLM, and 20 picked an open-source LLM. Interestingly, over 70% of enterprises not fully committed (and not among the 59 above) favored a vendor or licensed source; open source was less popular. This seems to indicate that most companies confident in deploying an open-source LLM had already done so, or that some form of evangelism on the part of a supplier was essential in driving the project. Supermicro was the big driver of open-source LLM, and IBM the AI “vendor” most often cited. IBM was also classified as the most proactive in promoting AI in their accounts.
I believe that the majority of the operations AI missions are really ML rather than LLM, but I’ve tried to exclude the ML applications because they’ve tended to be product features rather than generalized AI. They’re also more confined to the operations personnel using them, which means that unless these specific workers comment to me on their applications, I don’t see them.
There are still issues with LLM AI, both in the general case and the case of self-hosting by enterprises. Every enterprise is experimenting with several “copilot” models of AI-as-a-service, but while most say workers like it and that it improves work quality or productivity, most still find it difficult to create a formal justification for it, and to work out how the current tools could be extended to create something that would actually improve operations in a quantifiable way.
One thing that does appear to be true, which is that the great majority of enterprises believe that LLM AI would have to be self-hosted to be used in the two missions (customer-facing chatbot and business analytics) that are most likely to actually generate a business case. That means that to shift LLM AI focus from the “hobby” mission of personal productivity to a real, successful, profit-expanding mission will require self-hosting, and that likely means that sponsorship in some form is critical.
The question of mission expansion also seems critical. A small group of CIOs have commented that they believe the notion of a customer-facing chatbot could be expanded to create a virtual entity that could “collaborate” as a senior partner in a number of applications, and say that code review in development teams is an example they’ve already validated. Something like this could create a broad internal mission for LLM AI that would surely require self-hosting, and at least this small CIO cadre believes they can make a business case for it.
Overall, it seems to me that enterprises are increasingly likely to turn to “sponsors” for LLM AI, meaning they want a solution rather than a set of products they have to combine themselves. Right now, the sponsor candidates are dominated by computer vendors (IBM, Supermicro, Dell, HP Enterprise) or AI companies (OpenAI, Microsoft, Google), with emphasis on the former. Whether either of these groups expands, or another group or groups emerges, may be a factor in AI growth, if the emergence of new players offers buyers something distinctly different and valuable.
