The involvement of vertical industries and enterprises in the ongoing deployment of 5G networks has increased the need for performing test and measurement (T&M) – solid and extensive trials as well as validation T&M for vertical-specific applications. The trial environments should mirror the conditions and configuration that the vertical applications will face at the time of their launch so as to verify whether these can be considered 5G ready. By early testing of their innovative use cases over a standards-based full-chain 5G facility, a wide range of vertical industries can make timely and well-informed business decisions on launching their services with guaranteed performance levels and higher chances of business success. To this end, the evolving 5G ecosystem needs to accommodate frameworks that can easily test, measure and demonstrate the key performance indicators (KPIs) of the new softwarised 5G network carrier-grade environments. However, there is a gap between the end-to-end (E2E) and network functions virtualization (NFV) characterisations, and performance evaluation in the new 5G heterogeneous infrastructure. T&M procedures, tools and methodologies need to be agreed upon to ensure proper functioning of the deployed networks.
Role of 5G KPIs
Viable performance assurance
Prior to any development and testing activities, a set of standards and well-defined 5G KPIs must be considered for carefully assessing the potential feasibility of its use case over a standard 5G network. More often than not, in 5G literature, listed 5G KPIs are associated to their values for maximum theoretically achievable performance. If we take the case of the KPI user experienced data rate (UL), the value of 50 Mbps is the maximum performance that could be achieved with a standard 5G deployment. As a result, a theoretical use case demanding a minimum sustained user data rate (UL) of 64 Mbps shall not be supported by standard 5G deployments, and further development and testing should be aborted under such assumptions. That type of preliminary assessment must be performed and logged in detail by a vertical, for each KPI including the user data rate, peak data rate, capacity, E2E latency, mobility, reliability and availability.
Moving from theory to practice, from the whiteboard to the planning of costly development and testing activities, the verticals should identify their best partners among the 5G actors whose facilities can attest beforehand the delivery of performance KPI levels in line with the set expectations. For all these reasons, the 5G Infrastructure Public Private Partnership (5G PPP) infrastructure projects are planned to increase the level of 5G KPIs at their network facilities, deploy and upgrade their 5G infrastructure periodically, perform systematic measurements, testing and validation for determining and demonstrating the best achievable performance, and make that information public for consultation by innovating vertical firms that might be interested in partnering with them.
Predictable performance assurance
The variety of standard 5G service slice types (eMBB, URLLC, mMTC), architectural options and alternative configurations must be taken into account since they condition the actual feasibility for a standard 5G environment to support the specific set of 5G KPIs demanded by the application. This does not imply that verticals should become deep experts in 5G architecture/ technology; instead they must collaborate with 5G domain counterparts for actual testing activities. Here, the decisive focus should be on KPI monitoring and assessment, considering a variety of application scenarios and a selection of 5G network configurations and conditions. By following this systematic approach, the verticals can not only benefit from assessing and, hopefully, proving the technical viability of a service, but also from deriving a model for predicting the service performance in operations for a wide range of foreseeable conditions. That will provide the verticals with a strong foundation for making a well-informed business decision on whether, when, and how they may successfully launch their services.
A commercial 5G network is, unfortunately, not the type of environment that the verticals need for completing this stage of development and testing. The real insight into behaviour and performance needed by the verticals for completing their innovation cycles can only be provided by an experimentation facility, equipped with the tools and processes to carry out their T&M activities. The 5G PPP infrastructure projects aim at filling that gap in the 5G ecosystem and providing that kind of added value services to the verticals. These infrastructures will help in processing the KPI requirements of the verticals for deriving and automatically synthesising as well as launching several test cases over their 5G facilities. The data generated by the execution of all those relevant test cases are gathered, analysed and summarised for the vertical users to help them characterise the behaviour of their 5G-compatible applications and end-user devices under a variety of internal and external conditions.
Plug-and-play performance assurance
The ultimate ambition of most verticals is to launch their applications successfully worldwide and increase their return on investment. The variability of 5G commercial network deployments and service offerings on locations around the world can turn out to be overwhelming, creating a complexity perception that may slow down or even cause dropping the initial launch plans beyond the first pilot. In such a scenario, 5G KPIs, along with the insights obtained when addressing the predictable performance challenge, can be the lifeline for the verticals involved in this journey. At this critical stage where the verticals have to transit from a successful development and testing cycle to putting their systems into actual business operation over a variety of 5G networks, it would indeed not be a wise approach to start the full testing and characterisation cycle all over again at each of those 5G networks. This is not realistic either, as commercially rolled-out networks do not really serve that purpose; they just deliver end users with the performance agreed to in service-level agreements (SLAs).
Further, the negotiation of the necessary SLA for 5G KPIs with the network provider is a requisite condition for moving forward, and it involves time and cost. Valuable information logged at the extensive trials performed in the previous stage, relating to network architectures, configurations, test cases and monitored KPIs, can be leveraged by the 5G network service provider for coming up, in a short time and at a reasonable cost, with a proposal for creating the service that can guarantee the vertical’s required SLA. The time and effort saved by the mobile network operators (MNOs) at this stage translate into faster time-to-market of the vertical services roll-out.
Testing and monitoring
Testing (or active testing) provides greater observability due to active control over the type and intensity of traffic that is pushed through the network and through the subsets of the network elements. This provides a higher degree of freedom in selecting what can be tested and measured (for example, scalability or security resilience). Monitoring, meanwhile, is a generally passive process that provides metrics from various components/layers of the 5G network. While testing is normally applied throughout network roll-out, during the onboarding process of new virtualised network functions or new software updates, and for the validation of newly deployed network services, monitoring is constantly active during network operations and is a key enabler of the network management actions and processes.
Testing as a service
The advent of 5G brings along a variety of parameters that can affect overall network performance. Constant software updates and optimisation of NFVI parameters are only two examples of the huge number of possible use cases. Testing becomes an even more important tool in the hands of network management and operations engineers for guaranteeing SLAs signed between the MNOs and their vertical customers.
Test-as-a-service (TaaS) plays an important role in reducing the effort that MNOs’ engineers need to put in for testing the 5G infrastructure and components. By simplifying the testing operations and providing an interface to connect to the continuous integration of the MNOs, TaaS promises a stable performance delivery while controlling (or even reducing) the opex. TaaS is expected to become an essential component of the zero touch philosophy that is currently pursued in standards such as ETSI ZSM (zero touch network and service management) or open source communities such as OPNFV (Open Platform for NFV) and ONAP (Open Networking Automation Platform).
TaaS is an automation and interface layer that allows connection among all the T&M tools for validating and verifying the 5G system, from individual components up to the E2E service. Automation makes it possible to abstract the complexity with a series of either standard or custom test cases. All the aspects ranging from conformance to security, and from performance to quality of experience can be tested through TaaS system, making it a powerful tool for network equipment vendors, MNOs and vertical customers. s
Based on a white paper by the 5G PPP, a joint initiative between the European Commission and European ICT industry
