As those who read my blog know, most network operators and many enterprises undertake a more-or-less-formal technology planning cycle between mid-September and mid-November. One of the areas where both enterprises and network operators are focusing attention is on network operations, or more broadly on IT operations. I’ve followed these sessions for over two decades, and they’ve always proved interesting. More recently, over the last five years, the sessions have shown some subtle shifts in emphasis, and this year is shaping up to be an even greater change. This is a good time to take a look at the early signs, many of which are focusing on operations.
While nobody says a lot about this, the focus of operator changes in netops has been on reducing costs by reducing staff. Operators told me 15 years ago that they were already cutting headcount in “craft” or outside plant personnel and in customer support and contact roles. These showed a quick return, but at this point most of the available cuts have been made. There have been attempts to reduce headcount in network operations centers and technology center support personnel, but these have delivered only modest benefits.
Those modest benefits have been largely offset (and sometimes totally obliterated) by higher operations costs associated with computer server and software technology. Of 28 mobile operators I’ve chatted with, 23 said that their operations costs for data center technology has been higher than expected. A dozen made a point of saying that a virtual function was more expensive to support than the equivalent network device. The reasons for this are complicated.
The top problem cited by these 28 operators is the old-like OSS/BSS versus NMS division. In the structure of telcos thirty years ago, the “plant” or network infrastructure was maintained by a team of technical people and there was a network operations center (NOC) that handled errors and problems with the technology. The billing and customer support tasks were largely managed by business support systems (BSS), and operations support systems (OSS) had a sort-of-intermediary role between BSS and the NOC. Just what that role involved varied from operator to operator. The OSS/BSS systems were (and still largely are) under control of the CIO, the NOC under the COO, and technology evolution under the CTO. However, there were plenty of operators who had NOCs part of the OSS processes. The way the NOC is run remains a point of tension, in fact.
In many cases, roughly a quarter of all operators, the CIO has retained a lot of influence on the evolution of network infrastructure because the only personnel who had any server/software experience were in that organization. Operators today say that they believe some of the problem with the operations costs for server/software technology stems from the fact that hosting infrastructure is a COO element, and in most cases had to be staffed up quickly from outside hires or CIO transfers. The people thus had relatively little experience with the role of hosting in network operations.
The second problem cited by operators is the overall management framework. The big Tier Ones all recognized that when hosted features/functions were introduced, they would have to be somehow integrated with the management of legacy network devices. This was one reason why there was a drive, when Network Functions Virtualization was evolving, to take a virtual-to-physical device mapping as a given. The goal was to present the management of virtual functions in the same way as the management of an appliance was presented, particularly with regard to the OSS functions.
ONAP or Open Network Automation Platform, was originally developed as ECOMP (Enhanced Control, Orchestration, Management, and Policy) by AT&T, and turned over in open-source form to the Linux Foundation. ECOMP reflected that same general vision of virtual-to-physical mapping, and also reflected legacy views of OSS and NMS tools. That meant that it was really designed more as an evolution to open device management than as a mechanism for managing virtual functions and hosted features, and from an architecture view it fails to conform to what I’ve said all along NFV management should have been designed to do. I stopped taking briefings on ONAP evolution because it wasn’t changing to adapt to what operators say they wanted in an operations framework.
This year, operators tell me that they are hoping to (to quote one) “transform our NOC and infrastructure management to conform to cloud developments.” This operator’s hope is that the Nephio project (also from the Linux Foundation) will be the basis for the future of network operations and will also “bring the OSS into the cloud.” That may happen down the road, but at the moment that kind of sweeping change is out of the scope of the project. What Nephio is really doing is making it possible to deploy both real and virtual elements using Kubernetes; management of networks is still a bit orthogonal to that. Still, operators overall are very interested in Nephio and more than half of the 77 who had comments on this said they expected to “evaluate” or “deploy” it starting by around 2H24.
On the enterprise side, things are even more confusing. Of 185 enterprises who offered views on this, 58 said they had a “unified” model of network operations and the remainder said they still relied most on vendor management tools. The management focus for enterprises is on reducing errors rather than on lowering costs (except where cost management is achieved in part through reduction of errors). There is some interest in integrating “the cloud” into network operations, largely arising from the use of either cloud-resident SD-WAN elements or SASE, and this appears to be growing. However, there is still a sharp difference between enterprise network management and cloud management, which falls under IT operations. It remains to be seen whether gradual absorption of network features by the cloud will drive integration at the management level, and if so how that integration will be achieved.
One thing that enterprises and operators agree on, with qualification, is that it’s important that network operations and management consider a different approach. Operators almost universally believe that it needs to be “event driven”, which is also how they characterize the steps they believe to be needed in modernizing OSS systems. Enterprises who have any view on basic NOC changes agree, but of the 185 enterprises I got info from, only 98 had a view on how NOCs should change. Even this group tended to characterize what was needed as an avoidance of a “polling” model of management to an “alert” model, though they acknowledged that they did get alerts already, which suggests that they may not have thought through all the details.
The big question, then, is whether anyone will actually make any new operations-specific tech plans in their fall cycle, and thus drive changes at the budget level in 2024. Operators, I think, are more likely to do that because there’s evidence they actually push vendors on issues, but the problem I see there is that there really are no strong products out there for them to pick. On the enterprise side, where the buyer tends to be driven by seller solicitations, I’d be surprised if there was a lot of movement. We won’t know till November, and I’ll report back then on what happened.
