Nowadays, data center is no longer as simple as one server room, but a huge data center cluster composed of multiple server rooms. At present, new data centers built by major Internet companies already commonly use 100G interconnection technology, and will be commercially available on a large scale in the next 2 to 3 years with 400G interconnection technology. In this article, we will introduce 400G optical modules and their application to data centers.
What is 400G optical transceiver?
In terms of optical wavelength, 400G transceivers can be divided into multi-mode(MM) and single-mode(SM); In terms of signal modulation mode, 400G transceivers can be divided into NRZ and PAM4 modulation (mainly PAM4 at present); In terms of transmission distance, 400G transceivers can be divided into SR, DR, FR and LR; In terms of packaging form, 400G transceivers can be divided into CDFP, CFP8, OSFP, QSFP-DD, etc. The following mainly introduces the 400G optical modules in OSFP and QSFP-DD form factors because they are more widely used.
- OSFP (Octal Small Formfactor Pluggable)
The OSFP optical module has an integrated heat-sink that greatly improves thermal performance. Its size is much smaller than that of CFP8 module, slightly larger than that of QSFP-DD module, and its power consumption is relatively low of only 15W. The OSFP optical modules are mainly used in the telecommunications market. The OSFP is able to meet the projected thermal requirements for 800Gbps optics when those systems and optics become available in the future.
- QSFP-DD (Quad Small Form-factor Pluggable–Double Density)
The electrical interface of a QSFP-DD connector has 8 electrical lanes, running at 50Gb/s each, for a total bandwidth of 400Gb/s. The QSFP-DD form factor is similar to the QSFP form factor, except a second row of electrical contacts has been added to increase the number of high-speed electrical lanes from 4 (in a QSFP) to 8 (in a QSFP-DD). The QSFP-DD is forward and backward compatible with the QSFP port and is also compatible with the existing QSFP28 optical modules and AOC/DAC.
QSFP-DD has two modulation modes: when using NRZ modulation, the transmission rate is 25Gb/s*8(channel), which can be used for 200G optical communication; when using PAM4 modulation, the transmission rate is 50Gb/s*8(channel), which can be used for 400G optical communication. Compared with OSFP optical modules, QSFP-DD is mainly used in high-speed data centers of large Internet enterprises. So, what are the main types of 400G QSFP-DD optical modules?
- 400G SR8 QSFP-DD
The transmission distance of 400G SR8 module through multimode fiber can be up to 70m on OM3 fiber or 100m on OM4 fiber. The transmission rate is up to 425Gbps and the central wavelength is 850nm. 400G QSFP-DD SR8 optical module is more suitable for short-distance interconnection or multi-channel data communication.
- 400G DR4 QSFP-DD
400G DR4 module achieves the transmission over SMF(single-mode fiber) with an MPO-12 connector. It supports a max transmission distance of 500m on single-mode fiber. In addition, 400G DR4 is also the base form of 400G silicon optical modules. In the 400G era of the data center, 400G DR4 can not only realize the four-in-one Breakout networking, which can be transmitted to 100G DR1/FR1, but also replace the short-range multimode 400G optical module interconnection on the access side, with end-to-end cost competitiveness.
- 400G FR4 QSFP-DD
400G FR4 module supports link lengths of up to 2km SMF with duplex LC connector. This module uses CWDM (wavelength division multiplexing) technology, using 8 channels of 53Gbps PAM4 signals on the electrical side, and 4 channels of 106Gbps PAM4 signals on the optical side, which is twice the rate of the electrical side. The transmitter unit of the module uses a 4-channel laser driver chip to drive 4 groups of TOSA to realize the electric/optical conversion, and the 4 optical signals with the center wavelengths of 1271, 1291, 1311 and 1331nm are outputted after the conversion, and then a MUX couples these four groups of signals into one SMF.
Moreover, there are other types of 400G QSFP-DD optical modules, such as 400G SR4 (100m transmission), FR8 (2km transmission), LR8 (10km transmission), ER8 (40km transmission), which are not widely used, so they will not be introduced here.
What are the application scenarios of 400G QSFP-DD transceivers in the data center?
There are three main types of connections between optical modules and switches in data centers: intra-cabinet ToR switch and server interconnection, inter-cabinet switch interconnection, and data center interconnection (DCI).
- Intra-cabinet ToR switch and server interconnection
There are two main schemes to realize the interconnection between switches and servers: direct attach cable (DAC) and active optical cable (AOC).
DAC has the advantage of low price, but its disadvantage is also very obvious, that is, short distance. In addition, the cable is bulky and difficult to manage. However, AOC cable vs DAC, has the advantages of light weight, long distance, anti electromagnetic interference (EMI), easy line management and so on. Although the theoretical maximum transmission distance of AOC is 150m, because there are modules on both sides of AOC, it is not suitable for cross cabinet layout. Therefore, AOC is generally used in scenes with a distance of less than 30m.
- Inter-cabinet switch interconnection
Data center Spine-Leaf switch interconnection or Leaf-ToR switch interconnection can be achieved using three types of 400G optical modules: 400G SR8, 400G SR4.2, 400G DR4.
Firstly, from the perspective of the cost of optical modules, the 400G SR8 solution uses a mature 25G baud rate VCSEL laser chip with PAM4 pulse amplitude modulation. The 25G baud rate VCSEL laser chip has been very mature in the market, so the cost of 400G SR8 optical module is the lowest. For the 400G SR4.2 optical module, it adopts a dual wavelength 25G baud rate VCSEL laser chip, and needs a 2:1 Mux combiner and a 1:2 Demux splitter, which increases the cost of the optical module. In addition, there is only one optical chip supplier for 400G SR4.2 optical module, so the cost of this type is significantly higher than that of 400G SR8 optical module. Then, the 400G DR4 optical module using more expensive DML laser or SiPh silicon optical technology, which can greatly reduce the power consumption, size and packaging cost of the optical module. However, silicon optical technology is in the early stage of industrial development, with low shipments, so it can not form scale effect.
Secondly, from the perspective of wiring, 400G DR4 uses 8-core single-mode fiber for parallel transmission and MPO/MTP-12 or MPO/MTP-8 for fiber connectors. 400G SR4.2 uses 8-core multimode fiber for parallel transmission and MPO/MTP-12 or MPO/MTP-8 for fiber connectors, while 400G SR8 uses 16-core multimode fiber for parallel transmission and MPO/MTP-16 for fiber connectors to reduce return loss. For new data centers, in order to support 400G SR8 optical modules, it is recommended to use MPO/MTP-16 fiber cabling system, which can directly support 400G SR8 Ethernet. For data centers that have already deployed MPO/MTP-12 or MPO/MTP-24 cabling, a smooth upgrade to 400G SR8 can be achieved by replacing the 4×3 MPO/MTP converter box.
Finally, from an application perspective, the 400G SR8 can support the most types of rate combinations, including 4x100G, 8x50G, and 2x200G. The overall cost of a data center network includes the cost of optical modules and cabling costs. Although the 400G SR8 requires a large number of optical fiber cores, the data center generally adopts modular design, and the interconnection distance of switches is generally about 50m. Therefore, the difference in wiring cost is much lower than that of optical modules. Overall, 400G SR8 is the most cost-effective solution for 400G switch interconnection.
- Data center interconnection (DCI)
- Data center interconnection within 2km-10km distance: The 400G FR4 module is an economical solution within 2km distance. Because 400G FR4 adopts 20nm wavelength interval, the larger the wavelength interval, the looser the requirements for Mux/Demux combiner. And the laser does not need cooling, so the lower the cost of optical module. If it exceeds 2km, the 400G LR4 optical module with slightly higher cost can be selected.
- Data center interconnects within 40km: the 400G ER8 or 400G ER4 optical modules can be used, but the latter is more expensive.
- Data center interconnects within 80km: 80km can be achieved using the 400G ZR4 optical modules. 400G ZR uses a more complex polarization multiplexing 16-stage quadrature amplitude modulation (DP-16QAM) method, which is approximately twice the price of 400G ER4.
Finally, to summarize how to choose the right 400G optical module for your data center.
- For ToR switch-to-server interconnect scenarios not exceeding 2.5 meters, 400G DACs are recommended.
- 400G AOC is recommended for ToR switch-to-server interconnect scenarios up to 30 meters.
- For Spine-Leaf switch interconnection scenarios up to 100 meters, 400GBASE-SR8 QSFP-DD optical modules are recommended, and MPO/MTP-16 OM4 multimode fiber is used for cabling.
- For Spine-Leaf switch interconnection scenarios up to 150 meters, 400GBASE-SR4.2 QSFP-DD optical modules are recommended, and MPO/MTP-12 OM5 multimode fiber is used for cabling.
- For Spine-Leaf switch interconnect scenarios up to 500m, 400G DR4 QSFP-DD optical modules are recommended, and MPO/MTP-12 OS1a single-mode fiber is used for cabling.
- For data center interconnection scenarios not exceeding 2Km, 400G FR8 QSFP-DD optical modules are recommended.
- For data center interconnection scenarios not exceeding 10Km, 400G-LR8 OSFP optical modules are recommended.
- For Data center interconnection scenarios not exceeding 40km, 400G-ER8 OSFP optical modules are recommended.
- For data center interconnection scenarios up to 120km, 400G-ZR CSFP2 coherent optical module is recommended.