7.3.1 The Leaf-and-Spine Switching Architecture

Data center managers have adopted a standard hierarchical arrangement of network switches that they replicate across the data center to connect more and more servers and accommodate the traffic flows between them. A commonly used building-block switch architecture is known as leaf and spine [6].

Leaf and spine is a layered architecture. Leaf switches typically reside on the top of a server rack, hence they are also known as top-of-rack switches. The spine switches, typically having a higher switching capacity, link several leaf switches to enable them to talk to each other.

A data center manager wants the switch architecture to connect the end devices (e.g., typically servers, but it could also be storage units) as efficiently as possible. Efficiency refers to performance metrics such as latency—the time it takes for a data packet to travel from one server port to another—which ideally should be as low as practicable. Efficiency also means using the least amount of switch hardware to meet the networking goals: more switches mean more floor space, more power consumption, and more transceivers and cabling, all of which add cost. If a data center manager is going to use multiple leaf-and-spine arrangements to connect hundreds of thousands of servers, the basic leaf-spine unit had better be as efficient as possible.

Fig. 7.3 shows a simplistic leaf-and-spine arrangement to highlight how switch count and interconnect grow as more servers are added.

The basic leaf switch shown is a four-port device that can support up to four servers. Doubling the server count to eight increases the number of switches from one to six and sends the number of interconnects linking the switches up from nil to eight.

Fig. 7.4 shows the exponential impact of increasing the number of servers further. Note the interconnects grow more rapidly than the switch count. The cost of scaling a leaf-and-spine architecture is thus a function of the number of switches and the interconnects they use. From Fig. 7.4, what can also be learned is that an individual switch with a high port count is desirable.

Table 7.1 further shows the effect on a network of scaling the number of servers. The number of switch tiers needed increases, which complicates the cabling and increases cost. The number of servers that can be linked is determined by the switch port count in the spine. Adding servers to meet end-user requests require a significant increase in switches and ports—all expensive items.

In practice, data center networks are far more complicated than the example above. For a start, other networking elements are required besides servers and switches. One is edge routers that support Internet Protocol (IP) routing to handle external connectivity to the backbone and wide area network (WAN)—i.e., communication beyond just the leaf and spine. Another is the load balancer that, as suggested by the name, has a view of workloads and ensures that one cluster of servers is not overworked while another is relatively idle.

Fig. 7.5 shows a schematic of a network described by Facebook [4]. Consider a 128-port leaf switch where 96 devices such as servers are connected using 10-Gb links. The remaining 32 leaf switch ports are used for uplinks. For example, ports can be grouped in units of four to create eight ports, each at 40 Gb/s, to connect the leaf switches to the spine, or fabric, switches. A three-tiered architecture is used in this example, with the second spine used to enable more end devices to be connected on the network.

Top-of-rack and spine switches, with 96 ports each, enable a total of 221,184 ports to be used to connect servers and other appliances. How is this number arrived at? The number of devices that can be attached is a function of the edge or top-of-rack port count, P_e, and the core or spine switches ports, P_c.

For a three-level design, the maximum number of devices or servers that can be connected is P_e × (P_c/2)^(L − 1), where L is the number of layers [7].

The issue with such a topology, says Facebook, is the huge number of cables and optical transceivers needed. In this example, there are 442,368 links between the switches (where the number of links equals P_c × P_c/2, as described above, where P_c is the number of core switch ports, in this case 96).

Not surprisingly, interconnect is a considerable part of the overall cost of scaling leaf and spine architectures. Moreover, the need for 100-Gb links and the need for single-mode transceivers affect the interconnect cost independently. It is this rising cost that is driving the need for cheaper interconnect.

The attraction of using the leaf-and-spine architecture is that it is scalable, predictable, and resilient. The switches are built using merchant silicon, and while these are complex chips, they are relatively inexpensive. Adding leaf switches and appliances does not require software changes as long as there are free spine ports. Multiple paths and spine-switch routing algorithms provide predictable transit times between the servers, ensuring good service for end users. And spine-switch failures can be circumvented with minor network performance degradation. All these factors are positives as far as data center managers are concerned.

The distance between the server and the top-of-rack switch is typically under 10 m, enabling copper cabling to be used. The link between the top-of-rack switch and the spine can be a few 100 m in length, but link lengths less than 50 m are typical. Although optics is used, the link cost is modest due to inexpensive optical transceivers and the short lengths.

As the network size increases and the number of servers and switches grows, so does the distance between the fabric (the spine 1 switches in Fig. 7.5) and the cluster (spine 2) switches, requiring more single-mode fiber and more costly single-mode transceivers.

Indeed, data center buildings can be distributed tens of kilometers apart to serve a metro area. Considering that redundancy is often desirable, such data center clusters can require a large number of links. Microsoft talks of distances up to 70 km; optical engineers are developing solutions for this application. In this case, tiered spine switches need to talk to each other over such distances to make the servers in the distributed data center buildings appear as a single shared resource (see Section 5.5.1)

7.3.2 Higher-Order Radix Switches

One way to scale data center networking is to increase the number of ports—or radix—of the building-block switch. Using a high-radix switch reduces the overall interconnects, switch tiers, and switch boxes needed. This reduces overall equipment costs and the cost of operating them, as less power is consumed and less floor space is needed overall. A high-radix switch also benefits network performance by decreasing latency as fewer hops between switch stages are needed when two servers chat.

A market leader in 100-Gb switch silicon is Broadcom. The fabless chip company’s flagship Ethernet switch integrated circuit (IC) is the StrataXGS Tomahawk with a switch capacity of 3.2 Tb: 32 ports at 100 Gb or 128 ports at 25 Gb [8]. While the bandwidth is higher than the top-of-rack switch shown in Fig. 7.5, the number of servers and other end devices supported is the same.

Ethernet switch equipment vendors have copackaged multiples of these switch chips to increase overall port count. Internally, these switches have a multitier CLOS, named after its inventor Charles Clos, a leaf-and-spine architecture, and the overall platform is typically used as a spine switch. But they are costly and power-hungry. Fig. 7.6 shows various vendors’ 100-Gb high-port-count switches.

7.4.1 An Electronic-Optic Switch Architecture for Large Data Centers

Keren Bergman’s career has always been at the intersection of computing systems and photonics, first with high-performance computing and now data centers. A professor of electrical engineering at Columbia University, Bergman’s current focus is on how photonics can improve overall computational performance of the data center while minimizing energy consumption.

A key challenge Professor Bergman and her research team are tackling is how photonics can best link 100,000 or more servers. Instead of developing a large high-radix—320 or even 1000-port—optical switch that would connect to the top-of-rack or leaf switches to offer an optical complement to the existing electrical switching architecture [9], Bergman and her team are taking a different approach: embedding photonics inside the electronic switches. Such an approach will be enabled by silicon photonics, says Bergman: “That is key.”

Bringing silicon photonics onto the switch chip board will create a new entity, a combination of electrical routing and optical switching. This results in the best of both worlds: all the benefits of an electronics switch chip such as programmability and packet buffering, with much denser optical input–output enabled by silicon photonics. In turn, by adding silicon photonics switching, traffic will be sent either through the electrical or the optical switching.

Embedded photonic switching offers a fundamentally different way of moving data across the data center, says Bergman. Photonics lends itself to data being broadcast to multiple locations—a networking approach commonly used for data packets known as multicasting—while individual transmitted wavelengths carrying data can be picked off at locations as required. Multicasting is much harder to do electrically and is costly in terms of latency and energy, especially over data center spans.

These optical-enabled data-transfer techniques offer a powerful way to deal with the huge growth in data traffic transiting the data center. Applications such as data analytics, used to extract valuable information from large data sets, is generating vast amounts of traffic between servers. Bergman sees a direct link between data analytics, which has great commercial value to companies, and embedded optics: “This is very recent, only in the last few months have we converged on this in our research.”

One goal of the team’s work is that a data center manager will not even know that the networking systems include embedded optics; the system will look and be controlled as current systems are today except it will run faster and consume less energy. This is extremely attractive to data center managers, whereas the adding of a large external optical switch requires them to change the data center controller software, she says.

But once silicon photonics becomes embedded within such systems, things will change. The resulting bandwidth of the networking architecture will become much higher and more uniform. Then, the need for hierarchical switching architectures will start to diminish. Professor Bergman predicts that we will see differently architected data centers in the next 3 to 5 years.

Silicon photonics start-up Rockley Photonics is also developing a switch architecture for the data center based on optics and a custom ASIC. As of this writing, Rockley is still in stealth mode. But it has said that its switch concept will scale with Moore’s law, meaning it will support a doubling of capacity every 2 years [10].

The idea of embedding optics in systems is not new. Intel announced a disaggregated server architecture enabled using silicon photonics interconnects incorporated within the system. And start-up Compass Electro Optical Systems (Compass EOS) went a step further and copackaged optics with electronics to create a complex ASIC with optical input–output. While both designs faltered commercially, they provide valuable case studies in highlighting the potential of bringing optics inside systems. Both systems are now described.

7.4.2 Intel’s Disaggregated Server Rack Scale Architecture

Intel introduced its Rack Scale Architecture in 2013, a disaggregated server design enabled by silicon photonics. A disaggregated design physically separates the various parts of a server—processors, memory, storage, and switching—linking them using high-speed, low-cost interconnects instead [11].

Using such an approach, the server’s performance can be tailored to a given task. If more memory is needed, it can be added without upgrading the entire server. And as the elements making up the server have different upgrade cycles, when a more powerful processor is released, it can be slotted in without having to upgrade all the other elements. This can extend the life of the platform to as long as 20 years [12]. There are also environmental cost benefits. If the processors generate more heat than flash memory, they can be dealt with as a microenvironment and cooled appropriately (see Fig. 7.7).

However, the disaggregated server design only works if the various elements can be connected with sufficiently low latency, and that requires high-speed, cheap interconnect. Intel chose to use silicon photonics. The silicon photonics transceiver operated at 1310 nm and was coupled to multimode fiber. Intel worked with a leading fiber manufacturer Corning to develop the multimode fiber and demonstrated it transmitting at 28 Gb/s over nearly a 1-km distance [13], although distances of a few 100 m are sufficient for these applications.

Intel also worked with US Conec to develop a custom connector that had high bandwidth and was robust. The resulting connector used a fiber ribbon, 16 strands wide and 4 rows high, yielding a cable, called the MXC, with 64 lanes, each at 28 Gb/s, for a total duplex bandwidth of 1.6 Tb/s [14].

Intel developed an expanded beam connector to ensure the MXC’s ease of use. This enabled robust coupling between the light source and the fiber strands ensuring a tolerance to dust and dirt, the cause of most optical link failures.

By choosing multimode fiber and developing a low-loss, terabit-plus connector, Intel hoped to offer a novel server architecture for the data center with silicon photonics at its core. However, in early 2015, Intel announced a delay in its silicon photonics modules and said they would ship by the year’s end [15]. In August 2016, Intel did indeed announce silicon photonics modules, but not for the Rack Scale Server. Rather, these are pluggable 100-Gb PSM4 and CWDM4 QSFP28 transceivers for switch equipment in the data center [16].

So where does that leave Intel’s Rack Scale Architecture?

According to an Intel spokesperson, silicon photonics is not being used in Intel’s Rack Scale design, primarily because optical connectivity is not required at the data rates and distances used inside a rack today. Copper continues to push back the technical boundaries and keeps an all-optical interconnection the subject of “future” plans. But the spokesperson added that as server resources disaggregate, and throughput and distance requirements increase beyond the capability of current interconnect technologies such as copper and vertical-cavity surface-emitting lasers (VCSELs), then silicon photonics is expected to provide the high-speed optical connectivity between a new generation of pooled resources.

So despite Intel’s technological heft and the design’s development effort and ambition, the company concluded that a silicon photonics design is premature for this application. Moreover, copper already enables such disaggregated designs. Equipment makers such as Cisco Systems and Dell have server disaggregation products that use copper-based high-speed connections. Ericsson has also announced a disaggregated hardware system based on Intel’s Rack Scale Architecture where the connectivity is optical but it is not based on silicon photonics [17].

7.4.3 Compass-EOS: Copackaging Optics and Silicon

Compass-EOS was arguably the first company to offer optics copackaged with a complex chip. The company, which later became Compass Networks, provides a valuable case study from a technology and commercial perspective.

The ambitious Israeli start-up developed an IP core router to compete with the likes of Cisco Systems, Juniper Networks, Alcatel-Lucent (now Nokia), and Chinese giant Huawei.

It developed a way to integrate optics with its complex traffic manager chip design, resulting in a simpler and lower-power optically enabled core router platform. However, despite the novel chip, the company ultimately failed commercially, largely because its software team of 60 engineers could not compete with its much larger IP core router rivals.

Simply put, a router is a networking platform at the heart of the Internet. It takes IP traffic in the form of packets on its input ports and forwards them to their destination via its output ports. To do this, two chip types are used: a network processor and a traffic manager.

The network processor does all the packet processing—it takes the packet’s header and uses a lookup routing table to determine its destination and inserts a new updated header frame in the packet.

The traffic manager oversees billions of packets. The chip implements the queueing protocols and, based on a set of rules, determines which packets have priority on what ports. In a conventional IP router, there are also switch fabric chips that send the packets between the cards to the right router output ports.

Compass-EOS designed its router between 2007 and 2008 and used a merchant chip for the network processor. But it designed its own complex traffic manager ASIC and added a twist by figuring out a way to add optics to the chip. As a result, no switch fabric was needed. Instead of the traffic manager going via the router’s electrical backplane to a traffic manager on another card, each optically enabled traffic manager had sufficient bandwidth to connect to all the other traffic managers. Eight traffic manager chips in total were used on four line cards—all linked in a fully connected optical mesh.

As lead engineer Kobi Hasharoni put it, there was no backplane, which is why the Compass-EOS routers were so much more compact than those of its competitors.

At the time of the design, Compass-EOS did not consider using silicon photonics, which was deemed too immature. Instead, Compass-EOS used its ingenuity to figure out how to couple multiple VCSELs and photodetectors onto the chip.

VCSELs in 2007 were at 10 Gb/s, and Compass-EOS chose to operate them at a more relaxed rate of 8 Gb/s. In total, 168 VCSELs and 168 photodetectors were used on the traffic manager chip, enabling 1.344 Tb of traffic for transmit and the same for receive. The resulting chip with optical input–output consumed one-fifth of the power of a chip using electrical-only connections and achieved a 12-fold bandwidth-density improvement compared with electrical for the same chip area. Compass-EOS was developing a second-generation chip with a 16-Tb input–output before the project was canceled.

Hasharoni says that were he and his hardware colleagues to tackle a similar design today, they would use silicon photonics instead of VCSELs. The design would better support single-mode fiber, and the packaging would be easier with both the ASIC and optics being silicon.

What Compass-EOS demonstrated in 2010—arguably at least a decade ahead of its time—is how optics can be integrated alongside a complex chip to benefit the system architecture. In this case, it resulted in a more compact, lower-power IP router that was less costly to operate [18]. But despite all the router’s hardware ingenuity, the venture ultimately failed.

The start-up’s experiences were different from Intel’s with its Rack Scale Architecture. Compass-EOS’ hardware was a more ambitious copackaged chip, and the company did sell its IP router to several leading telcos. But the technical benefits of integrated optics within a system did not guarantee commercial success.

Before discussing the particular component technologies that will enable wide adoption of optics, first on the printed circuit board, then copackaged with silicon, the issues associated with pluggable optical modules are the subject of the next section.

7.5.1 Pluggable Optical Transceivers Evolution: Photonic Integration for 100-Gb

The 100-Gb transceivers being adopted to connect leaf and spine switches, and between the spine switches, use multiple data lanes, each lane having a transmitter and receiver. The main two approaches are to use wavelength-division multiplexing or parallel optics, as described in Appendix 1, Optical Communications Primer. For the former, four lasers are used to generate 4 wavelengths, each at 25 Gb/s, which are multiplexed onto a single-mode fiber, while with parallel optics one laser can be shared to generate the four physical transmit and four receive lanes, each at 25 Gb/s.

The cost of a laser is important. A single-mode transceiver is dominated by the laser cost, which can account for over 30% of the total bill of materials.

The parallel optics 100-Gb PSM4 optical module specification has a cost advantage because it can share a single laser across all four lanes, each lane being independently modulated. But link cost increases with distance because a fiber ribbon cable, eight lanes wide, is used. In contrast, a single fiber pair is all that is needed for wavelength-division multiplexing using the 100-Gb CWDM4 or CLR4 modules. Photonic integration is the only technological approach that provides optical module vendors with the opportunity to meet the demanding price goals.

For these 100-Gb modules, there is ostensibly no difference in optical performance between a silicon photonics-based transceiver design and an indium phosphide one. But like what happened with 10-Gb devices, which went from an initial parallel design to a serial one, developers are plotting to build a 100-Gb serial design that uses one transmitter and one receiver only. Such a design requires increasing the electrical lanes to 50-GBd signaling coupled with the multilevel signaling scheme of four-level pulse-amplitude modulation (PAM4) that allows 2 bits to be transmitted per symbol [19]. PAM4 is becoming an important modulation scheme to address evolving data center requirements.

Moreover, 100-Gb is the latest but not the last speed stop in the data center. Internet content providers are already eyeing 200- and 400-Gb links and even 1-Tb ones. All these designs require photonic integration and high-speed electronics that will benefit from a close coupling between the optics and electronics, whether the optics are embedded on the printed circuit board near the chip or are copackaged with the chip.

Using an advanced CMOS packaging process for silicon photonics provides the most promising path for high performance and low cost.

7.5.2 The Move to Single-mode Fiber

Both single-mode and multimode optical transceivers are used in data centers. Traditionally, links from top-of-rack switches to spine switches use multimode fiber while links between spine switches use single-mode fiber. Microsoft is one Internet content provider that plans to use only single-mode fiber for all its future data centers. Many other Internet content providers are following Microsoft, but enterprise players, e.g., continue to use multimode.

Multimode fiber links are the cheapest because they use inexpensive lasers and hence inexpensive transceivers based on VCSELs. However, with each increase in data rate, the transmission distance standardized over a multimode fiber diminishes. This, in part, explains Microsoft’s decision to adopt single-mode fiber, to support long spans within its data centers, to simplify its networking decisions and future-proof its fiber investment as data rates inevitably rise. Using only single-mode fiber also reduces the module types that a data center manager must keep as inventory and hence costs.

Data center managers’ adoption of single-mode fiber is good news for photonic integration and for silicon photonics because it increases the overall volume for single-mode transceivers, essential for reducing costs.

7.5.3 Pluggable Transceivers for the Server–to–Top-of-Row Switch Optical Opportunity

Current links between servers and top-of-row switches are at 10 Gb/s because that is the link speed needed by the servers used by the largest Internet content providers. Copper cabling is commonly used to connect the two given the link distance is typically is under 3 m.

Optical cables have started replacing copper for such links as costs have come down. And optics beats copper when it comes to power consumption. Comparing optical to electrical switches in, e.g., a 10-Gb port, the power consumption of copper is 4–6 W, whereas for optics it is 0.5 W [20].

Another trend benefiting the switchover to optics is that as transmission speeds increase, the distance supported by a copper link decreases. As network interface cards for servers move to 25 Gb/s, copper can support up to 3-m link length. But moving to 50-Gb and then 100-Gb links, copper’s reach may become too short for server-to-leaf switch connections.

Optics will play an increasingly important role connecting servers to switches. Here, silicon photonics is battling multimode VCSEL-based links. For silicon photonics to replace VCSELs, it must match VCSELs’ optical performance and low power consumption and be lower cost—a challenging set of requirements. Silicon photonics has shown it can be competitive with VCSELs when used in active optical cables.

7.5.4 Transition One: From Pluggables to On-Board Optics

Given how switch capacity will continue to grow, a key issue that needs to be addressed is increasing the capacity that can be supported on the faceplate of platforms. The platform itself will not get any larger so what is required is to pack more, higher-capacity pluggable modules on a switch’s front panel.

The market currently uses 100-Gb pluggable modules, and the smallest optical transceiver package available for 100-Gb is the QSFP28 form factor. Thirty-two QSFP28s are typically used on the front panel matching the 3.2 Tb of capacity of current state-of-the-art switch chips, and this can be stretched to as many as 36 QSFP28 modules.

Mellanox is one equipment company that is confident it can get to 200 Gb/s in a QSFP28 module using its silicon photonics technology. The module is expected in 2017 and will enable a faceplate density of 7.2 Tb/s using 36 200-Gb modules [21].

According to Mellanox’s Mehdi Asghari, the company’s 25-Gb silicon photonics modulator and photodetector can already operate at 50 Gb/s. This means it can use the simpler nonreturn-to-zero modulation scheme rather than PAM4.

The challenge using nonreturn-to-zero signaling is getting the associated electronics to work at 50 Gb. This is a complex radio frequency design challenge that requires skilled analog design engineers. The design challenges include getting the drive electronics, the assembly, the wirebonds used to connect the chip to the packaging, and the printed circuit board tracks to all work with good signal integrity at 50 Gb. The benefit of going from 25 to 50 Gb/s using nonreturn-to-zero is that there is less signal loss than using PAM4 [22]. Using PAM4 also introduces extra latency. Nonreturn-to-zero is always the best option if you can do it, says Asghari.

The next-generation data rate is 400 Gb/s, and the proposed pluggable optical transceiver packages to house the optics are the quad small form-factor pluggable double density (QSFP-DD) and the µQSFP. Fig. 7.9 highlights these new pluggable form factor options being developed for next-generation switches as summarized by the industry organization, the Ethernet Alliance [23].

The QSFP-DD uses eight lanes that can operate at 25 Gb using nonreturn-to-zero modulation and 50 Gb using PAM4 to support 200- or 400-Gb ports. Thirty-six modules can be packed on a one-rack-unit front panel, for an aggregate bandwidth of 14.4 Tb.

The µQSFP is smaller—approximately, the same size as the small form-factor pluggable (SFP) pluggable module. As in the QSFP, there are four electrical channels, each lane using 25-Gb nonreturn-to-zero signaling. The multisource agreement is intended to support 50-Gb using PAM4 as well. While operating at 25 Gb, an aggregate one-rack-unit faceplate density of 7.2 Tb is achieved using 72 ports.

Optical transceiver vendors are well on the way to defining and preparing the next high-bandwidth transceivers with a small enough size to support next-generation 12.8-Tb switches. Internet content providers are already clamoring for these switches and associated optical interconnects.

One way to tackle the issue of limited faceplate density is to adopt on-board optics. Embedded, mid-board or on-board optics moves the optical transceiver off the front plate and onto the printed circuit board, closer to the ASIC.

Several advantages result. First, the high-speed electrical signal path from the ASIC to the optics is shorter, simplifying the printed circuit board design. And using on-board optics only, a fiber connector is fed to the front of the equipment. This means the number of transceivers depends on how many embedded optics modules can be fitted on the board, not how many pluggable modules can be fitted on the equipment’s front panel. Freeing the front panel of pluggable modules also allows for more ventilation holes, improving the air flow used to cool the equipment.

However, on-board optics has its own issues. One is that all the switch slots are populated from the start with optics, something that can be avoided with pluggable modules. This may be acceptable for the Internet content providers, but many enterprises prefer a pay-as-you-grow model.

Another is that if an optical link fails, the entire board has to be replaced—an expensive proposition. Solutions lie in optical redundancy, in understanding the failure mechanism, and in design and planning, such as making the optical modules field-replaceable.

There is also no industry standard for embedded optics. COBO, the Consortium for On-Board Optics, is an industry initiative backed by Microsoft that is working to develop the first standard solutions for embedded optics [24]. COBO is working on 400-Gb/s designs and a way of combining two side-by-side to deliver 800-Gb/s interfaces. The embedded optics will support multimode and single-mode fiber over different reaches. COBO is considering enabling mid-board optics to support distances of 100 km using coherent technology, as discussed in Section 5.5.1.

Luxtera is providing 2-by-100-Gb PSM4 embedded optics modules in volume to several equipment makers. Luxtera is also a member of COBO, as is Mellanox. Yet while both companies welcome the advent of a standard, they admit there is an industry caution among companies about embracing on-board optical technology.

Luxtera cites the issue of field serviceability as one concern customers have. “Front faceplate pluggable modules are much easier to replace but come with a higher implementation cost,” says Luxtera’s Peter De Dobbelaere. He claims that the greater reliability of silicon photonics when used for embedded optics also helps allay customers concerns.

Mellanox says that the key issue is the input–output bottleneck on the ASIC and embedded optics, while increasing the input–output of equipment, does nothing to address the chip’s pinch point, as is explained in the next section. Asghari also warns not to dismiss pluggable optics which he expects to continue to evolve for at least two more generations.

The use of silicon photonics suits mid-board optics design due to the small size, low power, and low cost needed. The potential for high volumes is also attractive. However, embedded modules can use any optical technology as long as it provides adequate performance. And while silicon photonics may be well suited, it is up against traditional VCSEL technology for short-link lengths and indium phosphide–based offerings for longer reach ones.

The advent of on-board optics is an important milestone in the development of photonics in the data center. Brad Booth, the chair of COBO, says that the deployment of technology in systems will help the industry learn what embedded optics brings and what some of the challenges are. There is no revolutionary change that happens with technology, he says, it all has to be evolutionary. In other words, embedded optics represents an important stepping-stone for the next development—and the most significant development for silicon photonics—that of copackaged optics.

7.5.5 Transition Two: From On-Board Optics to Copackaged Optics

The introduction of higher-capacity switch chips is driving pluggable module developments and a need for optics inside systems.

Current generation switch chips support 3.2 Tb of capacity, and the 6.4 Tb Tomahawk II from Broadcom, at the time of writing, is already sampling. The QSFP28 pluggable module developments will be able to support these next-generation chips, as discussed in the previous section. The next switch chip advance—to 12.8-Tb, a capacity already sought by the Internet content providers—is expected in 2018. Here too, next-generation pluggables such as the QSFP56, QSFP-DD, and the µQSFP will deliver sufficient input–output capacity at the system’s front panel to support such switch silicon.

But faceplate density—having sufficient input–output capacity to support the faster switch silicon—is only one of the system design issues. To go from 3.2 to 6.4 Tb, the input–output of the switch chip must double. Broadcom’s 6.4 Tb switch chip crams 256 25 Gb/s signals on-chip by using a more advanced CMOS process node.

Each of these 25-Gb input–output signals is generated using a serdes (serializer/deserializer) circuit. The role of the serdes is to serialize the switch data and drive it across electrical lines on the printed circuit board to the pluggable optical transceiver. Clearly, going from 3.2 to a 6.4-Tb chip doubles the number of 25-Gb lines driving signals across the printed circuit board to the 100-Gb pluggable modules. Inside the optical module, there is a retimer circuit that cleans up the received electrical signal before being passed to the optics for transmission.

According to Luxtera’s De Dobbelaere, switch vendors have to sometimes put an additional retimer chip between the switch ASIC and module to address signal integrity issues associated with longer printed circuit board traces. Retimer circuitry adds to the board’s design complexity and dissipates power, and the problem only worsens as channel count and data rate go up.

And of course that is just what will happen with the advent of 12.8-Tb switch silicon. To achieve 12.8 Tb, two approaches are possible: either go to an even more advanced—and expensive—CMOS process node such as 7 nm to get the serdes to 50 Gb/s or continue to use 25-GBd/s serdes but add PAM4 to double the switch capacity. According to De Dobbelaere, the industry has settled on 25 GBd and PAM4 for now. Note, whichever approach had been chosen, it would only exacerbate the chip’s power consumption—the serdes are expected to consume over half the chip’s total power—and the issue of retimers.

This is where using on-board optics close to the switch chip helps. The shorter traces between the chip and the on-board optics consume less power. But embedded optics does nothing to solve the fundamental problem associated with switch ASICs: the finite number of input–output lines a chip can support.

Simply put, the chip’s high-speed input–output signaling must be placed on the chip’s edge due to printed circuit board design considerations. The chip’s perimeter is finite, and the input–output signals must be spaced a certain distance apart. The serdes circuitry connects to the printed circuit board through little balls on the chip packaging’s base, an arrangement known as a ball grid array. There are only so many balls that can be supported on a chip, and only so much of the ASIC’s area that can be used for input–output without exceeding the maximum chip size.

The next switch chip after 12.8 Tb will be 25.6 Tb, scheduled for 2020. For the chip to support such a capacity, 100-Gb signals will be required using 50-GBd and PAM4. Such a solution will burn an even greater proportion of the chip’s power, complicate the packaging and printed circuit board designs, and diminish the transmission distances possible over the board’s traces. On-board optics simplifies the signaling between an ASIC and the on-board optics and lowers the power consumed, but it cannot address the chip’s input–output bottleneck.

What is required is a way to expand the number of input–output signals off the chip and simplify the drive requirements on the individual serdes. This is where interposer technology—developed by the chip industry—can help. By using interposer technology to copackage the ASIC with silicon photonics, both these goals are achieved.

The semiconductor industry is adopting interposer technology as part of its work developing 2.5D- and 3D-packaging technology. The chip industry has long used 2D-packaging techniques whereby more than one chip die—the bare silicon chip before it is packaged—is bonded onto a common substrate. The substrate material includes laminates (a form of printed circuit board with fine copper lines), ceramic, or silicon. 2D packaging is known by a variety of names such as system in package and multichip module. The advantage of 2D packaging is the ability to combine different chip technologies in one package. This benefits yield and allows device evolution without having to redesign a single more complex chip [25].

The 2.5D chip extends the concept by adding a silicon interposer between the different dice and the substrate, as shown in Fig. 7.10. The interposer is a slice of silicon which has metal traces on both its surfaces. The chips are bonded to the upper surface while the lower surface is connected to the common substrate using standard flip-chip bumps. Through silicon vias (TSV) technology is used to connect the two metal layers to allow the dice to communicate with the printed circuit board. For completeness, 3D packaging extends the concept by having dice stacked on top of each other and which are connected to the interposer.

What Fig. 7.10 shows is that the chip dice are connected to the interposer’s upper surface using micro-bumps. These are a tenth the size of the flip-chip bumps and the bumps used by the ball grid array. And this is the key benefit: the interconnect on the interposer allows for much greater interconnect bandwidth between the dice, enabling a tremendous increase in capacity and device performance.

For silicon photonics, the technology holds huge potential. By adding a silicon photonics die on the interposer, an additional input–output route using fiber off the 2.5D chip becomes available.

Using 2.5D technology, silicon photonics vendors can thus benefit from yet another technology that has been developed by the chip industry. Using such a technology, designers can develop designs similar to the ASIC with optical input–output pioneered by Compass-EOS.

For the data center, 2.5D packaging offers a solution that overcomes the input–output bottleneck of the switch chips, something that optical pluggable modules and on-board optics cannot address. It is also a technology that silicon photonics is inherently able to exploit. This is not the case with the established optical technologies.

And it doesn’t stop there: high-performance computing processors and future field-programable gate arrays (FPGAs) can benefit from being copackaged with silicon photonics. An FPGA is an important chip, with the more advanced versions containing huge numbers of logic gates as well as 25-Gb/s serdes that can be customized to implement datacom and telecom chip designs [26].

Professor Bergman points out that her team’s research work includes ways to embed photonics within electronic switches to improve networking performance while reducing the energy required, as discussed in Section 7.4.1. This will be enabled using 2.5D integration technology, and silicon photonics allows the embedding of optical input–output right next to the electronics switch chip.

Another part of her data center work is looking at computing nodes. Such nodes comprise general multicore devices, specialist graphic processing units, and high bandwidth memory. Chip-to-memory communications are becoming a key pinch point in these systems and are ripe for the use of interposer technology.

Bergman points out that only so many dice can fit on an interposer but that the bandwidth that silicon photonics brings means that 2.5D packages can be linked. “The really beautiful thing with photonics is that you add more interposers because the bandwidth off that 2.5D chip is just as high as on the chip,” she says.

Silicon photonics luminary Professor Lionel Kimerling, at MIT’s Department of Materials Science and Engineering, agrees that board-level integration is an important area of focus and the question is what solution will scale for the next 20 years. The chip industry will not make a big investment in something that will not scale for more than a couple of generations, he says.

In his view, the chip is becoming almost irrelevant given it will no longer scale with the demise of Moore’s law. “So it is all about how can I scale the number of chips in a package,” he says. This is promoting system-in-package technology. Once this starts, there is no doubt that there will need to be optical interconnect coming out of such a package, he says: “And no one really doubts that before long, you will need optical interconnect between the chips inside the package.”

This is the potential interposer technology offers. But combining photonics and electronics chips using an interposer is not without challenges. The semiconductor industry itself has only recently started to embrace the technology and adding photonics presents its own issues. Thermal management and where to place the laser being two major ones designers must resolve.

Will companies pass on on-board optics and adopt a 2.5D design when they transition their systems away from pluggable optical transceivers? It is possible but a more likely path is that the industry will embrace on-board optics first. New technologies take time, and there is much to be learned from embedded optics before copackaged optics will be ready and the industry will be willing to make the jump.

Meanwhile, startups are now becoming active in this area.

Ayar Labs is a US start-up that is developing a 3.2-Tb optical interconnect chip designed to sit alongside switch silicon. The chip aims to replace 32 100-Gb pluggable modules or, in future, eight more complex 400-Gb modules, and it is expected to emerge starting in 2018.

Another start-up developing a silicon photonics device to sit alongside chips in the data center is Sicoya from Germany. It is targeting servers first but says the technology can be used for other equipment in the data center such as switches and routers. It says its silicon photonics device is designed for chip-to-chip communications and could be placed very close to the processor or even copackaged in a system-in-package design [27].