OK, you know that I believe AI is over-hyped. You may have seen some recent stories in even non-tech media that show others hold the same view. Lets get to things we need to know…like whether the AI hype problem is with AI overall or just generative AI, and whether AI might actually be better off without the hype.
In the last month, the number of enterprises who told me they believed that generative AI would have a “significant” impact on their business dropped from 191 to 111, with all but 8 of those who changed their view doing so because of early trials. They repeated a comment I’ve already noted; because generative AI was more approachable by non-technical types, it got management all gaga over it. Reality was setting in now. But what is “reality”? If I digest the views of the 72 who had failed experiments, I find a number of interesting observations.
The first is that generative AI failed to save money, at least at the rate needed. That was because it didn’t actually improve productivity much. In fact, 13 enterprises said it didn’t improve it at all, and 7 said that it actually interfere slightly.
The second observation was that generative AI didn’t offer any real insights. While users expected it to do better than searching or than non-augmented human work, it failed to do that. In fact, of 60 enterprises who used it in lieu of search, 44 said search was better.
That relates to the next point, which was that generative AI “hallucinations” were excessive and difficult to deal with. All 7 enterprises who said the productivity impact of generative AI was negative related this to the need to check its outputs. But outright mistakes weren’t the only reason to check output; 14 companies said they were afraid that generative AI might plagiarize another source, one it trained on. In nine of those cases, the company said that phrases in a generative AI composition, when searched on, hit Internet text exactly.
Of the 72 companies who found problems with generative AI, 48 tried it in writing code, and all said the results were marginal at best. It could handle simple tasks, the stuff given to junior programmers, but even there it needed more code review, and as one development manager put it, “It never got any better, improved with experience, learned from mistakes.”
The biggest overall complaint uses offered was that writing a prompt to get generative AI to solve a complex problem was almost like programming, and resulted in bad results all too often. The next-most-cited problem was the inability of generative AI to produce results on company-specific data. Obviously neither of these would be insurmountable, but both created more issues than expected.
None of the 72 companies were saying they were walking away from generative AI forever, but all said they’d take a breather in adoption until things improved, which on the average they believed would take over a year. They did say (by a margin of 48 to 24) that they had scaled back their expectations for generative AI. The only area of application they remained hopeful for was the private-model, trained on local data.
In the group of 72, there were comments from 37 people who I believe are at the tip of the AI understanding spear. This group believes that the big problems with generative AI comes down to two words, generalization and accessibility. The tools are designed to try to do too much, and to be too “conversational”. One expert pointed out that having a tool to analyze a form or spreadsheet would be more useful than one designed to answer questions in general. In short, experts want task- or job-focused AI.
This view is spreading to the management level, particularly to CIOs. Their problem is the same hype that’s infected the generative AI space, and AI overall. Nobody, they tell me, is doing enough to make AI a realistic asset. Well, at least very few are. Those served by IBM account teams like IBM’s approach a lot, and report good results from it so far. Two of the companies have been trialing their own AI deployments with open-source tools, but these two are having problems getting up to speed. One additional company tried an external firm to lead them through their efforts, but the initiative was unsuccessful.
All 72 of the companies believe that hype has hurt AI. They think way too much attention has been focused on trying to create a human intelligence rather than on creating a specialized intelligence. Of the 72, 58 said that the tools they were aware of were difficult to apply to their own business needs, partly because they seemed to work only on public knowledge and partly for data security reasons.
What about the 111 enterprises who remain confident (at least to a degree) that AI will improve their business. Only 73 of them are actually in AI trials, so for the rest we have reason to wonder if their views are more than wishes. Of the 73, 35 are playing with generative AI in ways that those who had on-staff expertise suggest will fail to make a business case. Another 12 are using “integrated AI”, and whether all of these actually qualify as AI is difficult to determine. That leaves only 26 who might be on a path to using AI for meaningful changes.
I think this all leads to a simple conclusion, which is that enterprise AI progress is stalled by lack of understanding of the path forward to success. What they read and hear is simply not preparing them for the future. And only 39 of the 111 enterprises who still believe in their own AI success say they have confidence they could obtain, or even recognize, AI job candidates they need. AI has an education problem that won’t be easy to fix, because we lack ready information and human resources to fix is. Just how that will be corrected, and who will do the job, is a question I can’t answer, because the industry can’t…for now.
