Few industries have more to offer and to gain from big data than telecommunications. For decades, communications service providers (CSPs) have transported and captured huge volumes of information about customer calling patterns, wireless data usage, location, network bandwidth statistics, and even the individual apps and webpages accessed by mobile devices.
Until recently, much of that data was discarded. There simply was no efficient way to mine value from it and storing it was expensive. However, this is all changing. Big data technologies, when combined with streaming analytics and analytics at scale, are enabling telecommunications companies to uncover significant new insights about their infrastructure and their customers. These insights are leading to massive changes in the ways they do business. In addition, with new legislation that opens the door for internet service providers to sell data about their customers’ online behavior, data may become a significant new revenue source.
As the range of telecommunications services has expanded with the adoption of mobile data, so have the potential applications to improve efficiency and generate new business. Providers can leverage this information to better understand their own business. For example, usage pattern data can guide bandwidth allocation and the positioning of equipment such as cell towers. It can also help identify new services to offer customers and even open up new revenue streams in areas such as targeted advertising.
Big data technologies are revolutionizing telecommunications. Tools like Apache Hadoop, streaming analytics, and machine learning are opening up new opportunities for CSPs to gain insights from data sets that were previously unwieldy.
This industry guide looks at the trends driving the big data revolution in telecommunications, how different segments of the industry are being affected, and how big data is being put to work in the field to change the competitive equation. This guide outlines a number of issues the industry faces and discusses how these trends are driving new solution areas. It also looks at use cases that show how big data and analytics are yielding game-changing breakthroughs for telecom providers today.
The Motivation for Big Data
With the rapid expansion of smart phones and other connected mobile devices, communications service providers (CSPs) need to rapidly process, store, and derive insights from the diverse volume of data travelling across their networks. Big data analytics can help CSPs improve profitability by optimizing network services/usage, enhancing customer experience, and improving security. According to McKinsey, the potential for Telcos to profit from applying data science effectively is substantial. Examples include:
- Predicting the periods of heaviest network usage, and targeting steps to relieve congestion
- Identifying the customers most likely to defect, and targeting steps to prevent churn
- Identifying the customers most likely to have problems paying bills, and targeting steps to improve the recovery of payments
Big Data Use Cases In Telecom
Telecommunication companies collect massive amounts of data from call detail records, mobile phone usage, network equipment, server logs, billing, and social networks, providing lots of information about their customers and network, but how can telecom companies use this data to improve their business?
Most telecom use cases fall into these main categories: customer acquisition and retention, network services optimization, and security.Customer 360
Using data science in order to better understand and predict customer behaviour is an iterative process, which involves:
- Data Discovery and Model Creation:
- Analysis of historical data.
- Identifying new data sources, which traditional analytics or databases are not using due to the format, size, or structure.
- Collecting, correlating, and analysing data across multiple data sources.
- Knowing and applying the right kind of machine learning algorithms to get value out of the data.
- Using the Model in production to make predictions
- Data Discovery and updating the Model with new data.
In order to understand the customer, a number of factors can be analysed such as:- Customer demographic data (age, marital status, etc.)
- Sentiment analysis of social media
- Customer usage patterns, geographical usage trends
- Calling-circle data
- Browsing behaviour from clickstream logs
- Support call centre statistics
- Historical data that shows patterns of behaviour that suggest churn
With this analysis, telecom companies can gain insights to predict and enhance the customer experience, prevent churn, and tailor marketing campaigns.
The architecture below shows how batch processing on different data sources can be used to build and update a model, which can then be used for real-time predictions on streaming data.
INDUSTRY TRENDS
To say the telecommunications industry is in transition is an understatement. Once a highly regulated business with fixed prices, monopoly market control, and little customer choice, the telecom market has been upended by digitization and mobility.
Customers now expect access from any device, anywhere. Their bandwidth needs have expanded to include video and high-speed data. Customers also have plenty of carriers to choose from, ranging from traditional phone service providers to cable companies to VOIP services. Thanks to full mobile phone number portability, customers can move their service from one carrier to another with virtually no disruption, enabling them to play carriers against each other.
In short, a once-staid industry is now hypercompetitive. This has created pressure on the large incumbents and opportunities for nimble niche players, though there’s opportunity for the big players as well. Telecom firms are aggressively seeking new lines of business, ranging from advertising to cloud hosting to original programming. They are leveraging their infrastructure to deliver new high-margin information and entertainment services to mobile customers, homes, and businesses. Smart phones and programming are profitable new revenue sources. Although competition has increased, customers are writing bigger checks to telecommunication service providers than they ever have.
Some of the priorities today’s telecom providers must address include the following:
- System-wide cost take out and optimization
- Improving customer loyalty, acquisition, and retention
- Monetizing data and analytics
- Mass personalization
System-Wide Cost Reduction and Optimization
Customer demand for data is growing at an accelerating rate, putting pressure on all service providers to optimize infrastructure. Operational expenses typically consume 30-40% of revenue, and network operations account for about 45% of that cost.1 A single cell tower can cost $250,000 to build, and equipment must be continually maintained and enhanced to support new protocols and services. For example, the cost of upgrading existing networks to the new 5G network infrastructure is expected to cost more than $100 billion over the next 10 years.2 As a result, service providers are always looking for ways to squeeze more capacity out of existing infrastructure, reduce the cost of expansion, and find new ways to leverage existing infrastructure profitably.
They are also seeking to better understand customer behavior in order to maximize margins. Just 1% of mobile users consume half of the world’s bandwidth.3 Carriers want to identify these heavy users and charge them appropriately.
Two factors that will influence their planning are the move to software-defined networking (SDN) and the surge of new bandwidth demand associated with the growth of the Internet of Things (IoT). SDN promises to lower costs and increase flexibility, but there are big migration expenses involved. IoT will create new pressure on capacity planning, but also yield attractive new sources of revenue. Both technologies promise to create massive structural change in existing networks, and they will require careful planning and ROI analysis.
Improving Customer Loyalty, Acquisition, and Retention
The consumer market is essentially saturated with more phones on the planet than people. This means growth must come from selling more products and services to existing customers and stealing customers away from competitors. Factors such as service quality, price, speed, and customer service are key variables in this equation.
Research shows that customers perceive little difference between telecommunication service providers. This means the ability to understand customer needs at a fine-grained level, provide fast and friendly service, and customize service plans for businesses and individual customers is crucial for success.
KEY INDUSTRY STAKEHOLDERS
Telecommunications industry diversification has created a host of new stakeholders beyond carriers and their customers. Each has different information needs.
Carriers
Service providers need to capture detailed information about infrastructure, service quality levels, demand patterns, and other operational data in order to optimize resources and deliver high-quality user experiences. They also need to mine customer behavior data in order to fine-tune their product offerings and take advantage of new opportunities like advertising and paid information services.
Customers
In a market with a profusion of service offerings, many of them based on use, customers appreciate having up-to-the-minute data on account status and warnings of additional charges they may incur. CSPs can enhance customer satisfaction by offering complete reports and recommendations for account changes that can optimize each customer’s spending.
Suppliers
The high cost of infrastructure gives service providers plenty of incentive to understand the ROI of the dollars they spend on equipment and service. Sharing this information with suppliers can help them negotiate more favorable contracts and ensure that suppliers are delivering on service-level agreements.
Companies that supply end-user devices such as handsets and accessories want detailed sales information as well as recommendations for promotions and other incentives that can improve their bottom lines.
Regulators
Government overseers demand full transparency about service levels, rates, customer satisfaction, and other metrics. They also want evidence that service providers are staying within customer privacy and confidentiality guidelines. CSPs must not only capture this data, but tag and index it for rapid access since compliance audits may require a response in 48 hours or less.
Content Providers
This new but important constituency provides programming and information services that open up new revenue streams. Content providers require information on how their content is being catalogued, promoted, and consumed as well as details on royalties, licenses, and other forms of compensation.
Advertisers
Advertisers are another new and promising business opportunity. Ad buyers want clickstream data such as views, click-throughs, downloads, registrations, view times, and other engagement metrics. They also may demand this information within the context of location, time of day, app use, and other variables.
USE CASE EXAMPLES
All these resources have been used in real-life scenarios to change the telecommunications game.
Customer Loyalty and Acquisition
Once a highly regulated industry, telecom is now a virtual free-for-all. With the market nearly saturated, carriers focus much of their efforts on stealing customers from each other. And thanks to phone number portability and the decline of service agreements, it’s never been easier for customers to switch.
The telecommunications industry suffers the second lowest customer satisfaction ratings after government.4 In the U.S., turnover rates for the major carriers typically run around 20% per year. Globally, the churn rate is much higher.5 At the same time, the cost of acquiring a single new telecommunications customer has been pegged at more than $300.6 With more than 17 million customers signing up for new plans or switching carriers each year, acquisition costs add up to more than $5 billion annually.
This has made customer service a critical competency for all providers.
Big data and analytics tools help carriers understand customer behaviors and interests at a fine level. For example, CSPs can use a variety of internal and external metrics for churn analysis, which alerts companies to signals that a customer may be about to defect. Evidence might include declining usage, repeated calls to the customer support center, or frequent dropped calls. Social media also presents valuable new ways to detect at-risk customers. By monitoring online sentiment and matching usernames to customer accounts, telcos can pinpoint dissatisfied customers and extend individualized retention offers. Social media is also a valuable feedback mechanism for new products and services. Customer reactions to new equipment, service plans, or offers can be captured almost immediately and used to adjust pricing or marketing plans proactively.
Churn analysis, driven by big data and analytics, enables telecom companies to identify potential defectors quickly and to target their retention strategies more selectively. For example, a company can look at a large pool of recent lost customers and cross-tabulate the data with other characteristics, such as marital status, age, location, volume of use, or payment delinquency. The same analysis can be performed on customers who have increased spending with their providers. This analysis yields “buckets” of customers that can be categorized according to their likely future spending patterns. Offers and incentives can then be targeted to groups of customers. Likely defectors can be intercepted, even if they have expressed no explicit plans to leave. This is important since research indicates that only 5% of dissatisfied customers ever overtly express dissatisfaction and about 80% of defectors give no reason for leaving.
The impact of churn analysis can be substantial. McKinsey cited one telecom provider’s use of machine learning to combine sociodemographic data, information from customer calls, and social media interaction network usage data to identify the customers who were most likely to defect or have trouble paying their bills. It cut churn by three percent and improved payment recovery by 35%.8
A key element of customer retention is delivering individualized service and promotions. Multi-channel call center automation software now makes it possible for companies to create unified views of their customers composed of feedback from phone calls, email, social media, and even in-store visits. By applying analytics to these rich profiles, telecom companies can group customers by category and customize marketing programs and offers to them. For example, heavy data users may be presented with bonus bandwidth or coupons for free streaming video, while customers with modest data needs may be offered discount upgrade incentives to get them to the next level. These tactics work; half of the business-to-business customers surveyed by Forrester Consulting rated personalized recommendations as the feature they would most like suppliers to offer.9
Capturing data from multiple sources in a reference database, as illustrated below, makes it possible for that information to be used in a variety of use cases ranging from searchable customer records to model training for machine learning applications.10 The ability of Hadoop and NoSQL databases to combine and perform analytics on a mix of both structured and unstructured data makes applications practical that were previously impossible.
Customers also vote with their clicks, and this activity can be captured and analyzed to understand customers’ content needs. For example, customers who upload a lot of photographs or streaming video may be offered free accounts on media-sharing services or cloud storage space. Frequent music listeners may be offered gift cards for streaming music services. The cost of these giveaways is a pittance compared to the cost of acquiring a new customer.
Social media sentiment analysis cuts both ways, and it can be useful in identifying at-risk customers of competitors. By monitoring negative comments about rivals, telecom companies can offer timely incentives to make the switch. Online real estate transactions or openings of new businesses
can also trigger offers to customers who are new to the area or those who are leaving but can be retained. The same profiling techniques used to extend retention offers to customers based upon demographic characteristics can also customize offers to potential new customers.
Big data permits much finer levels of targeted marketing. Instead of creating direct mail according to geography, companies can segment customers by combinations of demographic and behavioural data. Email and display advertising campaigns can be customized based on the characteristics of individual customers, and tactics like online A/B testing quickly provide feedback on the most effective offers and messages.
Customers increasingly expect to be treated as markets of one. Big data and analytics is enabling this goal to be realized.
Product and Service Quality
Telecommunications is a capital-intensive business with global CSP capital expenditures expected to total more than $2 trillion by 2019.11 Providers are under intense pressure to minimize dropped calls and data dead zones, which are among the biggest sources of customer dissatisfaction. Breadth of coverage is also a competitive advantage, so telecoms need to maximize reach while constantly monitoring their networks for outages, capacity thresholds, and other service quality issues.
The amount of data generated by data equipment is enormous and getting bigger. The advent of 4G mobile networks alone increased the data volumes from mobile devices about tenfold,12 and the arrival of 5G networks over the next two years promises to do the same. Other growth factors include location-based data, streaming media, IPv6 addressing, and the arrival of an estimated 50 billion connected IoT devices by 2020. Much of this data will need to be analysed in real time, both for service-level compliance and to recognize the promise of new revenue sources through services like location-based and contextual marketing.
Security and Compliance
Telecommunications is a regulated industry, and service provider networks are prime targets for attackers. Again, big data has an important role to play.
Service providers must comply with many standards in areas like service levels, availability, pricing, and coverage. The penalties for failing to capture this information can be steep, and audit demands may carry deadlines of 48 hours or less. Historically, responding to regulatory requests has involved digging through racks of data archived on tape. Thanks to Hadoop, much of this data can now be stored online for rapid retrieval. Telecom providers can also use the technologies of big data to better understand their own operations, flag potential regulatory violations, and correct them.
Security is a perpetual cat-and-mouse game in which analytics is playing a growing role. For example, security information and event management (SIEM) is a growing class of real-time analytics tools that monitors security alerts generated by network hardware and applications. It constantly compares network activity to normal traffic patterns and flags anomalies that may indicate penetration or fraud. The concept has existed for more than a decade, but the new breed of machine learning and predictive analytics tools, combined with large data stores, promises to make this technology far more effective.
Telecom providers have long had the ability to capture all the data that streams across their networks, but they can now do so much more affordably thanks to big data and streaming analytics. The potential bottom-line impact is clear. The industry loses about $38 billion to fraud each year,14 or about 1.7% of total revenues.
The benefits of strong security go beyond just revenue impacts. As CSPs increasingly expand into cloud hosting, software-as-a-service, and managed services, their ability to secure their networks will be an increasingly important factor in customer satisfaction. The combination of real-time analytics and predictive security enables government employees to access internet information without worrying about malicious payloads. It has also made reporting more timely. Reports that used to take 14 days now take only two hours.15
CONCLUSION
In a recent survey of 273 global telecom companies,16 McKinsey identified a strong appetite for big data projects, but relatively little use. While nearly half of respondents said their companies are considering investments in big data and analytics, only 30% had actually made them. Many of those reported disappointing results, with little incremental profit improvement. However, these results were mostly blamed on poor data quality, lack of talent, and under-investment.
In contrast, a small group of telecom providers had achieved “outsized benefit” from their investments. For example, one had used analytics models to predict the periods of heaviest network use resulting from customer video streaming. It was able to take steps to relieve congestion and reduce its planned capital expenditures by 15%. “The potential for companies that apply data science effectively is substantial,” McKinsey researcher Jacques Bughin wrote.
Effective use of big data requires commitment, a clear understanding of goals, and an investment in skills and technologies. There can be no question that telecommunications companies have many potential use cases that can significantly improve their understanding of customers and their own infrastructure. The best approach for early adopters is to identify projects with measurable short-term opportunities then deploy a scalable platform that can adapt to a wide variety of data types and tools.
