The Data Center
A Central Cog in the Digital Economy
Abstract
The Internet content providers, still young companies, have annual revenues in the tens of billions of dollars. Their profits are generated using data centers that deliver services over the Internet. Not surprisingly, telecom operators and enterprises are keen to adopt such practices to grow their own revenues. The Internet content providers own multiple large-scale data centers, each housing more than 100,000 servers, and they are building more, even larger ones. Scaling data centers is needed for the Internet players to continue to grow their businesses and deliver new, bandwidth-hungry services such as video. But such scaling is challenging: data centers are becoming costlier and consuming more energy. Systems innovation is needed, and silicon photonics, a core technology, can play a key role. But before detailing silicon photonics’ role in the data center, this chapter details the businesses enabled by data centers and the associated challenges. Data centers are core assets in the information age, and their importance will only grow with time.
Keywords
Cloud computing; energy; server; digital economy; data center builds; Internet content providers; telcos; enterprise; hybrid cloud
We are no longer in a linear one-dimensional industry, we are in a dynamic multi-faceted one. The fluidity at the moment is all around the data center, open networking, and how the unconventional network operators are influencing the market.
Brandon Collings, CTO of Lumentum
6.1 Internet Content Providers Are Driving the New Economy
The combination of the Internet, social media, and mobile devices has created an economy that offers entrepreneurs new ways to make money. Fig. 6.1 is a snapshot of the broad spectrum of Internet applications and the pervasiveness of their usage [1]. Internet content providers such as Google, Facebook, Amazon, Microsoft, and Apple are leading companies in this new economy, competing fiercely for the business of consumers and enterprises.
Data centers play a fundamental role in generating revenues for the Internet content providers. They are the ultimate modern-day factories that take in raw material (information in the form of digital data) and use labor (server-based computing and information processing) to manufacture products (services and data) that are distributed in the marketplace (via the Internet).
Data centers, packed with servers, must support annual double-digit traffic growth to ensure the delivery of services efficiently to users and businesses. Moreover, the Internet content providers’ ability to capture market share depends on their capability to fine-tune their data centers to maximize efficiencies. For now, revenue growth of the Internet content providers shows no sign of abating.
Internet content providers are disrupting several markets. After its first 20 years, spending on Internet advertising in the United States outpaced radio and television advertising during their first 20 years (normalized to 2015 revenues). The Internet accounted for $60 billion in advertising revenues in 2015 in the United States alone. Internet content providers Google and Facebook accounted for over three quarters of these revenues, and their share is growing [2].
E-commerce is another market segment where the Internet is having a huge impact, with growth forecast to continue. Internet sales accounted for 12% of all US retail—$340 billion—in 2015 [2].
The ability of the Internet content providers to reliably and cost-effectively deliver services to the user is key to this financial success, and data centers and associated networking are the enablers. For this web economy to continue to grow, it is important to offer new services to a broader community of customers while ensuring their experience is a good one.
The data center and its growing network bandwidth requirements offer silicon photonics the best near-term market opportunity. Silicon photonics promises to deliver cheaper, more compact, lower power high-capacity optical circuits—core infrastructure enablers needed to continue to fuel this new economy.
This chapter highlights the digital economy opportunities and the consequences and challenges resulting from the growth in the number and size of data centers. Silicon photonics is a technology suited to enable data centers to continue scaling, as highlighted in Chapter 5, Metro and Long-Haul Network Growth Demands Exponential Progress, and expanded upon in Chapter 7, Data Center Architectures and Opportunities for Silicon Photonics.
6.2 Cloud Computing: Another Growth Market
Cloud computing—offering IT and telecom services over the network—is another growth business opportunity for the Internet content providers. Cloud computing provides services for businesses, from small enterprises to multinationals, and for consumers. Cloud computing services can be offered over the same data center infrastructure used for existing services such as Internet searches for Google or social networking for Facebook. The infrastructure can be adapted to make available computing, applications, and storage resources over the Internet, creating an additional revenue stream.
Amazon Web Services offers a glimpse into this revenue potential. Amazon Web Services is the recognized leader in the cloud-based services market [3], and its 2013–15 annual revenues are illustrated in Fig. 6.2. Annual sales nearly tripled in 2 years, and if the trend continues revenues will exceed $10 billion in 2016.
Amazon Web Services has been building data centers worldwide to support its cloud business. The company operates 38 “Availability Zones” in 14 geographical regions, where each zone consists of one or more data centers [4]. Amazon plans to introduce nine more zones in four more regions throughout 2017, and it continues to expand its business in Europe and Asia.
In effect, Amazon Web Services is developing a scale of operation needed to deliver secure and low-latency cloud services. Amazon understands what is needed to support customer demand, with its data centers playing a key role.
6.2.1 Microsoft, Google, and Others Seek to Grow Their Cloud Businesses
Amazon Web Services’ nearest competitor in the cloud service business, Microsoft, is trying to grab more market share for its Azure cloud service. Financial analysts estimate Microsoft’s Azure revenues were $500–$700 million in 2015, a tenth of Amazon Web Services [5]. Microsoft, famous for its personal computer operating systems and software, boasts 1.5 million servers housed in 100-plus data centers globally [6,7].
Google is also seeking to win cloud computing business with its Google Cloud Platform, promising customers use of the same infrastructure that powers Google search and YouTube services. Google has captured from Amazon some of Apple’s cloud business, a significant feather in its cap [8].
The Internet content providers also use each other’s data centers to buttress their own offerings. Apple, e.g., was already a customer of Amazon Web Services and Microsoft Azure before signing on with Google [9]. Apple is rumored to be building data centers to expand its cloud offerings and reduce its dependency on sourced support.
The expansion of data centers to grow cloud businesses is not confined to the North American Internet content providers. Tencent, an important Chinese Internet content provider, announced that it is spending $1.6 billion to build data centers in China, Hong Kong, and North America to expand its cloud computing business [10].
Moreover, Tencent is collaborating with Intel on its Rack Scale Architecture—which disaggregates the servers’ computational, storage, memory, and network functions—to demonstrate the technology’s performance. While Tencent continues its assessment of the server architecture, it has reported better resource utilization and lower total cost of ownership [11].
6.2.2 Telcos Offer a Mixed Story Regarding Cloud Computing
Telcos are also active in growing their cloud service businesses and winning consumer and enterprise customers.
The data center strategy of the telcos has changed, however. Mergers and acquisitions ruled the day in 2011 for North American carriers seeking to expand their cloud offerings and credentials. Verizon’s acquisition of Terremark for $1.4 billion [12] and CenturyLink’s merger with Savvis for $2.5 billion were marquee transactions then [13].
More recently, telecom service providers have investigated selling these assets in what appears to be a reversal of their initial strategy. For example, AT&T’s data centers were rumored to be for sale in 2015, and eventually the operator struck a deal with IBM to take over managing its hosting service [14]. IBM is acquiring the equipment used to host the services. Both CenturyLink [15] and Verizon [16] are also considering selling their data centers.
One reason for this strategy switch is that the North American telcos have significant debt, a consequence of purchasing radio spectrum to expand their mobile services [17]. The telcos recognize they need not own and operate data centers to continue offering colocation and cloud services (colocation refers to offering third parties a presence in a data center). The telcos have also sold other assets to service their debt: Verizon, e.g., sold part of its wireline business to Frontier Communications for $8.6 billion [18], and AT&T and Verizon are selling their cell towers and leasing them back to support their mobile networks [17].
While using hosting services to support cloud services is an approach taken by some telcos, this is not a universal rule. Japan’s NTT, e.g., has acquired data centers globally. NTT sees its cloud business as central to its plan to grow its overseas business and as a key pillar of its overall strategy [19].
Telcos have central offices that can be rearchitected as data centers. These are not large-scale data center venues but are valuable resources to have. Central offices are used by telcos to house generations of equipment developed for their telecom services. Telecom carriers can own thousands of central offices—AT&T alone has between 4000 and 5000 [20]—that serve tens to hundreds of thousands of customers and enterprises. But they are not tooled to address today’s emerging opportunities based on software-defined networking and network functions virtualization, which are being used to support telecom service implementation on generic servers. In effect, telcos have seen the merits of running operations as the Internet content providers do, and they are looking to do the same for their own services.
AT&T is driving a high-profile program to transform its central offices to data centers. The program, known as Central Office Re-Architected Datacenter or CORD [21], is focused on the access network, delivering services on a passive optical network, e.g., to consumers and enterprises alike. And since these data centers are distributed and located at the edge of the telcos’ network, they provide an excellent opportunity to offer services close to customers.
6.2.3 Traditional Enterprises Use a Hybrid Cloud Model
To manage costs and deliver services to many regions, enterprises use data centers from providers such as Equinix and Digital Realty that allow connection to multiple telecom service providers. Enterprises prefer to use a hybrid cloud—i.e., a combination of private or on-premises and public or third party–based cloud services. This model allows them to have distributed data centers without the cost of building and managing them in all regions.
An announced deal between Shaw Communications and Microsoft in 2016 exemplifies such a relationship [22]. Customers can colocate servers in the Shaw data center while having the option of a public cloud service such as Azure.
6.3 The Expansive Build-Out of Data Centers
Internet content providers, telcos, and enterprises are all building data centers as well as seeking extra capacity through third parties as they expand their scale and offerings. Fig. 6.3 shows Facebook’s two Prineville, Oregon, data centers, and a third building, 40% larger than the first two, has been announced [23]. These buildings are distributed, and the growing bandwidth requirements they demand benefit silicon photonics, as outlined in Chapter 5, Metro and Long-Haul Network Growth Demands Exponential Progress.
6.3.1 Demand for Video Adds to Bandwidth Pressures
One trend helping to drive data center expansion is the growth of video. Video is a well-known network bandwidth hog, with Netflix’s streaming video service being the biggest culprit followed by Google’s YouTube [24]. Leading social media providers are also getting in on the act, introducing and extending services based on user-generated video.
The growth in daily video views on Facebook is shown in Fig. 6.4. It increased two and a half times during the period shown, from 3.2 billion views of at least 3 seconds duration at the end of 2014 to 8.0 billion views in the third quarter of 2015. And this does not include Facebook Live, a service launched in April 2016 that allows users to share live video and is driving further usage.
Meanwhile, Snapchat (since renamed Snap), another social media site, reported 10 billion video views a day in the first quarter of 2016 [25].
Facebook and Snap users are generating video that is streamed over the network, requiring service providers to support video-level bandwidth with storage and networking while maintaining the ability to rebroadcast clips on demand. Such traffic growth will increasingly stretch the networks of both the Internet content providers and the telcos. These services are in their infancy and will have a significant impact on networks and data centers.
Not surprisingly given these trends, Internet content providers are spending billions of dollars on building new data centers. In many cases they are building campuses where hundreds of thousands of servers are spread across several buildings. Microsoft, e.g., already has two data centers in San Antonio, Texas, and is said to be spending $1 billion to build a third, covering multiple buildings for a total floor space of 1.2–1.3 million square feet [26]. Breaking ground in 2016, the target completion date is between 2021 and 2023. And Apple, Google, Facebook, and Amazon are all building large-scale data centers to support their cloud services.
6.3.2 Facebook’s Data Center Builds: A Case Study
Facebook operates four campuses in the United States, each made up of multiple data center buildings, each totaling over 100,000 square feet. And Facebook is bringing data centers online around the world. Fig. 6.5 shows Facebook’s operational data centers and those announced, along with data center locations and approximate size (where data is available).
Building multiple data centers in one location is a clear trend, as illustrated in Fig. 6.5, and is a practice being followed by many Internet content providers. Typically, campus building connections are up to 2 km, distances that can be supported by 100 Gb/s silicon photonic transceivers that are commercially available today. And, as discussed in Chapter 5, Metro and Long-Haul Network Growth Demands Exponential Progress, new direct detection dense wavelength-division multiplexed modules are being developed to offer high-capacity links between buildings.
Fig. 6.6 highlights the increasing square footage of data centers. The data in Fig. 6.5 was used to construct Fig. 6.6 with the assumption that announced data centers will be operational 2 years after construction starts.
The increasing size of data centers comprising several buildings comes at a cost. Rising power consumption is one sensitive issue because it impacts the cost of doing business: operational cost.
Like other Internet content providers, Facebook is sensitive to its data centers’ environmental impact and energy demands. It has chosen locations like Lulea in Sweden where it can use the cold climate’s air to cool its servers. Wind power and evaporative cooling are being used in other locations, and Facebook is developing facilities that do not require traditional air conditioning.
Facebook also partners with local power authorities and is building renewable energy nearby for its data centers to drive down their operational cost and environmental impact. The issue of power consumption is now discussed.
6.4 Energy Consumption Poses the Greatest Data Center Challenge
Large-scale data centers consume vast amounts of power. Facebook’s three data centers in Prineville, Oregon, use some 70 MW [27], about two-thirds the power used by all the homes in the rest of the Oregon county where the data centers are located [28]. For reference, 100 MW can power 80,000 US homes [29].
Many Internet content providers use renewable energy including solar, wind, and natural cooling. According to environmental group Greenpeace, Google is the leading Internet player in purchasing renewable energy, with an estimated 35% of Google’s operations using clean energy [27].
In addition to using renewable energy and natural means of cooling, data center operators also seek greater efficiencies from their powered equipment. Server usage has been increased through software innovations and network sharing. Data center managers also replace equipment on much shorter timescales, from every 5–7 years previously to 2–3 years today. New equipment is more energy-efficient while delivering higher-bandwidth performance [30].
Energy remains a key issue when building large-scale data centers, but despite technological advances, the rate at which the power consumption requirements of the data centers is growing is a major concern. Electricity consumption from data centers and their networking was about 700 billion kWh in 2012 and is expected to increase to over 1200 billion kWh by 2017, according to Greenpeace [27].
Within the data center, the server is the most costly item (Fig. 6.7A), and it is the most power-demanding equipment class, accounting for nearly half of the power consumption as illustrated in Fig. 6.7B; IT equipment is dominated by the server. The second largest power-hungry item is the heating, ventilation, and air conditioning (HVAC) needed to keep the room temperature consistent so the IT equipment such as servers and the switches that connect them operate correctly.
As bandwidth demand continues to increase, and as Moore’s law ends, data center operators need to develop power-efficient strategies and reduce overall data center size while driving up the bandwidth that can be supported.
6.5 Silicon Photonics Can Address Data Center Challenges
Notwithstanding improvements in design and operation, Greenpeace finds data center power consumption the fastest growing among three important segments: end-user devices, networking, and the data center. More bandwidth is needed to support the growth in video and the increasing number of consumers using broadband. What is needed are ways to deliver services using more compact, power-efficient systems. This is an opportunity for systems innovation. How can silicon photonics help? This is the topic of Chapter 7, Data Center Architectures and Opportunities for Silicon Photonics.