Azure Networking CVP Yousef Khalidi posted a blog on March 15 2017 here, which states “Microsoft owns and runs one of the largest WAN backbones in the world.”
To serve more than 1 billion customers across more than 140 countries and regions, Microsoft has built huge datacenters that have a combined total of more than 1 million servers. These datacenters are strategically placed at different geographic locations and are connected by high-performance fiber-optic networks. They provide continuous supports to more than 200 cloud services, such as Microsoft Bing, Office 365, OneDrive, Xbox Live, and Azure platform.
Managing enormous resource pools is not an easy task. Microsoft has invested tremendous resources to build reliable, secure, and sustainable datacenters. The team that manages and runs Azure infrastructure is called Microsoft Cloud Infrastructure and Operations (MCIO), formerly known as Global Foundation Service (GFS). This objective goes behind the scenes and reveals how these datacenters are designed, built, and maintained.
Azure traffic between our datacenters stays on our network and does not flow over the Internet.
Over the last three years, we’ve grown our long-haul WAN capacity by 700 percent.
Within a given region, Microsoft can support up to 1.6 Pbps of inter-datacenter bandwidth.
“Our latest investment is the MAREA cable, a 6,600 km submarine cable between Virginia Beach, Virginia, USA, and Bilbao, Spain, which we jointly developed with Facebook.”
“MAREA will be the highest-capacity subsea cable to cross the Atlantic, featuring eight fiber pairs and an initial estimated design capacity of 160 Tbps. “
“To provide the reliability our cloud needs, we have many physical networking paths with automatic routing around failures for optimal reliability.”
“we developed a range of SDN technologies to optimally manage routing and centralize control to meet network-wide goals. “
“We use standard switches and routers, and then we manage them with our own software, which is built to handle the enormous volume of traffic on the Microsoft network.”
“We use an SDN-based architecture called SWAN to manage our WAN, “
https://aks.ms/microsofts-network
————————————————————————————————————————–
Software-Defined Networking innovations
“Hardware takes time to rack, stack, and configure, but we wanted to let customers scale their services up and down with a click.”
“we built a scalable and flexible datacenter network. It uses a virtualized layer 3 overlay network that is independent of the physical network topology”
“It uses a virtualized layer 3 overlay network that is independent of the physical network topology.”
The SDN paper is here.
James R. Hamilton
https://www.wired.com/2013/02/james-hamilton-amazon/
“For our Azure datacenters, we use scalable software load balancing developed by Microsoft Research which pushes networking intelligence into software. We eliminated hardware load balancers and replaced them with Azure Load Balancer running on standard compute servers.”
The SLB paper
“Azure handles the most demanding networking workloads by providing each virtual machine with up to 25 Gbps bandwidth with very low latency within each region.”
“Servers running in our datacenters have special network cards (NICs) that offload network processing to the hardware.”
“We’ve also developed novel network acceleration techniques using Field Programmable Gate Array (FPGA) technology incorporated into our SmartNIC project introduced at SIGCOMM 2015. These network optimizations free up the server CPU to handle application workloads”
Microsoft Showcases Software Defined Networking Innovation at SIGCOMM
https://azure.microsoft.com/en-us/blog/microsoft-showcases-software-defined-networking-innovation-at-sigcomm-v2/
“When our world-wide deployment completes in April, we’ll update our VM Sizes table so you can see the expected networking throughput performance numbers for our virtual machines.”
Note： this is why the link didn’t show expected network performance details.
“Worldwide, Microsoft has regions comprised of multiple campuses and each campus may have multiple datacenters”
“We built a backbone network that spans the globe, even laying undersea cables to Europe and Asia. All our datacenters connect to this global network that supports Azure, Bing, Dynamics 365, Office 365, OneDrive, Skype, Xbox, and soon LinkedIn. It’s one of the largest backbone networks in the world.”
Microsoft Invests in Subsea Cables to Connect Datacenters Globally
https://azure.microsoft.com/en-us/blog/microsoft-invests-in-subsea-cables-to-connect-datacenters-globally/
“We have 37 ExpressRoute sites with one near each Azure datacenter, as well as other strategic locations. Every time we announce a new Azure region, like we recently did in Korea, you can expect that ExpressRoute will also be there.”
“To meet growing cloud demand for Azure, we have invested over $15 billion in building our global cloud infrastructure since we opened our first datacenter in 1989.”
“In particular, over the last ten years, we have revised the physical network design every 6 months, constantly improving scale and reliability. we ship new network virtualization capabilities weekly,”
“To make VNets work at higher levels of reliability, we needed to develop two technologies: (a) scalable controllers capable of managing 500,000 servers per regional datacenter, and (b) fast packet processing technologies on every host that light up the functionality through controller APIs.”
“we leverage cloud technologies to build the Azure SDN. In particular, Azure Service Fabric provides the micro-service platform on which we have built our Virtual Networking SDN controller. Service Fabric takes care of scale-up, scale-down, load balancing, fault management, leader election, key-value stores, and more, so that our controllers can focus on the key virtual functions needed to light up networking features on demand and huge scale”
“Virtual Filtering Platform (VFP). In VFP we have developed the extensions for SDN that run on every host in the data center. VFP provides a simple set of networking APIs and abstractions that allow Microsoft to introduce new networking features in an agile and efficient way”
“we introduced new super low latency RDMA technologies, and a new protocol that we run in Azure’s NICs, DCQCN”
“the question of how we can leverage hardware to offload the VFP technologies, and get even better performance as networking continues on its journey to every greater speed, feature set, and support for larger numbers of VMs and containers. Azure SmartNIC meets these challenges. Our SmartNIC incorporates Field Programmable Gate Arrays (FPGAs), which enable reconfigurable network hardware. Through FPGA’s we can create new hardware capabilities at the speed of software. “
Pingmesh:
http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p139.pdf
Networking innovations that drive the cloud disruption
https://azure.microsoft.com/en-us/blog/networking-innovations-that-drive-the-cloud-disruption/
Packet-Level Telemetry in Large Datacenter Networks (Everflow)
http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p479.pdf
Microsoft Showcases Software Defined Networking Innovation at SIGCOMM
https://azure.microsoft.com/en-us/blog/microsoft-showcases-software-defined-networking-innovation-at-sigcomm-v2/
EverFlow
https://github.com/Azure/SONiC/wiki/Everflow-High-Level-Design
Network Function Virtualization (NFV)
Microsoft Invests in Subsea Cables to Connect Datacenters Globally
https://azure.microsoft.com/en-us/blog/microsoft-invests-in-subsea-cables-to-connect-datacenters-globally/
SONiC: Global support and updates
https://azure.microsoft.com/en-us/blog/sonic-global-support-and-updates/
Heran on Azure 