Modern network bugs require fast customer-centric solutions, not patches

Modern network bugs require fast customer-centric solutions, not patches

Software has become essential to hardware solutions within an increasing number of IT environments. Nowhere is this trend more apparent than in networking. Between 2013 and 2017 the software-defined networking (SDN) market exploded from $406 million to over $6.6 billion, and the IDC forecasts that number to reach $8 billion by 2021.

The motivation for this growth is clear. Overlaid software solutions cut through decentralization and provide single points of network control while simultaneously improving performance. SDN delivers programmability, network segmentation, improved traffic handling and the potential for increased security.

But, this reliance on software brings with it a risk that network vendors and service providers now must learn to face — software bugs. This issue has only been made worse by the competition from low cost “white box” technologies and software-only network solution suppliers that is pressuring established hardware vendors. Everyone is rushing to bring new solutions to market.

Why network firms need to take software reliability seriously

Software bugs create unstable and insecure solutions that leave customers vulnerable and unhappy. Solving defects with devices already in production is costly, if not nearly impossible using traditional methods. Engineers often lack the data they need to understand a fault, and the processes involved in collecting and assessing the necessary information is so slow that customer satisfaction cannot be delivered. Defects result in reputational damage, lost customers and potentially the failure of a product line.

In addition to service failures, bugs can create security vulnerabilities of which suppliers and customers can remain unaware until it is too late. Firmware that sits on hardware components is often a source of cyber infiltration because the devices (routers particularly) are rarely turned off, and firmware is both built on open source code and infrequently updated. Cyber attacks cripple businesses. Customers are unlikely to forgive network providers if they are proven to be the weak link that left them exposed.

The extent of the problem: networks and software ripe for failure

There’ve been several recent reports in the press about vulnerabilities in routers. In September 2018, Cisco reported that it had exposed 13 vulnerabilities in its IOS and IOS XE switch and router operating software, which the company said should be patched within weeks.

This is not an isolated incident. A few weeks following Cisco’s internal revelations, the American Consumer Institute issued a report on router safety that demonstrated a staggering 83% of home and office routers have vulnerabilities that could be exploited by attackers — assessing routers by HPE, TP-Link, ASUS, Linksys, NetGear, and others. Of those vulnerable, over a quarter are ‘high-risk’ or ‘critical’. These types of issues are likely to impact every major vendor in the network industry including Juniper, Citrix, F5, HPE, VMware and more.

Are perfect network solutions even possible?

Every one of those 13 devices in which Cisco identified vulnerabilities had been given a high impact security rating. We need to catch as many bugs as possible before products come in contact with customers. But, the number of different contexts in which network devices and software solutions are used makes it very unlikely that every single defect can be identified and remedied during the quality assurance stage of development.

With added pressure to demonstrate rapid evolution in innovation to cope with the rollout of 5G, ever-increasing cloud adoption, and hyperscaling storage solutions, development is only accelerating. The introduction of software only vendors and white box manufacturers has degraded any existing centralized quality control capabilities that existed within the industry. Perhaps more dramatically, it has increased the pressure on traditional equipment suppliers to maintain their market share and incentivized the rapid deployment of new solutions.

Without more time, it is unreasonable to expect network service providers and vendors to drastically improve their quality assurance in an ever increasingly complex technology environment. Even without the need for capacity expansion, the existing number of protocols embedded in complex architectures across network stacks makes debugging what is already deployed a monumental task.

An information-driven solution to debugging networks and software reliability

To confront the demands of modern networks, the industry needs to rethink how it handles software quality control. A new solution is needed to aid developers in resolving issues prior to production, and innovative debugging technology is critical to solving problems in the field. The use of software to reinvent networking has delivered the rapid developments required to meet current and future demands. But, using traditional software development strategies will slow things down. Speed is essential to success.

The challenge for service teams when attempting to debug software in a live customer environment is a lack of data and poor visibility into the factors that led up to a network failure. When devices are on-site at a customer location, engineers lack a precise record of the incident. They don’t know what went wrong; much less how it should be solved. Weeks can be wasted trying to re-trigger a failure and capture the data needed to start developing a fix. This is the source of tremendous customer dissatisfaction.

The likely solution is Software Flight Recorder technology. Just like the black box on an airplane provides aerospace engineers with the flight information needed to identify the source of a system failure or crash, Software Flight Recorder technology delivers software engineers all the data they need to quickly diagnose the problem so that they can quickly deliver a solution. The technology is based on a program execution record & replay engine which captures bugs ‘in the act’ and turns sporadic or non-deterministic failures into 100% reproducible test cases.

The same recordings can be used to step backwards and forwards in the code, allowing engineers to rapidly identify the root cause of failure. This emerging technology delivers improvements to customer-facing debugging operations and development teams, allowing for dynamic and fast solutions. Record & replay provides more information than general purpose tools such as printf, logging or core dump analysis, and more versatility than static analysis tools. Critically, this technology offers a solution to the most challenging software failures which cannot otherwise be solved.

Software Flight Recorder technology: delivering a network and software reliability solution that counts

With the growth of software-based network architecture, the need to ensure software reliability has never been more important. The competitive success of established hardware-based network vendors and service providers depends on that outcome. Customers deserve secure solutions and expect providers to deliver. Growing competition from white box and software-only networks demands innovative solutions from enterprise network vendors.

The leading enterprise software developer SAP has already introduced Software Flight Recorder technology into their HANA database management system, allowing them to catch the most challenging bugs before they hit production. The same technology is poised to answer the deficiencies faced by network vendors attempting to deliver clean software solutions to customers and rectify problems in the field at speed. Both outcomes are central to customer expectations.

IoT and AI create even more pressure to eliminate software failures prior to product releases. Gartner forecasts that 20.4 billion connected things will be in use worldwide by 2020. In a world of interconnected business, network failures are a disaster. Customers count on their infrastructure to work and hold up against cyber-attacks. They will not forgive network partners that let them down. The risks and opportunities have never been greater, and software is finally delivering a solution to hardware complexity in network management. The reality is that developing the ability to quickly solve software defects in the field has become critical to network vendor success.

To do this right, network companies need to be prepared. Software Flight Recorder technology delivers the capabilities needed to tackle disruption and provide world-class service solutions for the 21st century.