Network Design

Network isolation

Public, Private and Out-of-Band management networks are isolated from each other on the hardware level, which significantly increases security for our customers. Each server has five network connections. Two redundant connections to two different switches in a private network. Two redundant connections to two different switches in a public network. And, a connection of server's IP KVM (iDRAC) to a separate switch. These networks are fully independent. And here is a short list of the reasons why it is important.

Security

No configuration error will ever lead to data leaking outside private network to the Internet. Private and public networks are built on top of different hardware so that no logical error will lead to your sensitive data in the wrong place.

Economy

Private network traffic hits the ports of different switches, which are all 100% unmetered. You pay only for data transfer across the Internet.

Stability

Imagine a huge spike of traffic in the direction your server. This spike can be even large enough to make your server unavailable via the Internet. Although this is a very unlikely event, if it happens, you want to access your server and stabilize it promptly. Good news: the private network is isolated from public one - you can just connect via the private interface - and fix the server fast. However, if the whole networking stack of the server becomes irresponsive - which is even more unlikely - you're still not being cut off the server. You can just connect directly to its console via out-of-band network by opening a VNC console in the Self-Service portal.

L3 fabric

Both private and public networks of Servers.com are L3 Fabrics. So, why L3, why fabric, and what does this all mean? When we say L3, we mean IP. We have chosen L3 fabric (or IP fabric, that's the same), as it is easy to deploy, does not depend on the vendor, and allows us to implement any network features we want. Most data centers that offer private networks do it by providing a private VLAN per customer - which means, they create an L2 domain for each customer's private network. This approach has several flaws, which yield to a very limited scalability. These are just several reasons why L3 fabric suits better modern datacenter environment than L2-based solutions.

True redundancy

IP fabric is the only way to achieve true redundancy. Traditional L2 networks built using the spanning-tree protocol which utilizes only single “best path” chosen from all available paths, this means there is an active/standby redundancy. The problem is at the moment something goes wrong you can’t be sure that standby path is reliable. Unlike that, L3 fabric is active/active utilizes all available paths at the same time while still remaining stable and avoiding loops within the network.

Commoditization

With IP fabric and Leaf/Spine topology there is no reason to invest into BIG data center switches upfront. We can easily grow on demand by just adding more, less dense, switches. Better port utilization ratio really saves your bill! And finally, IP fabric is vendor agnostic.

Failure Domain

IP fabric is protected from switching loops, means it’s impossible to make the whole network unusable with a configuration error. It’s also protected from unknown unicast flood, when a DoS attack towards single customer can paralyze the whole network being multiplied by network equipment.

Global Private Network

IP fabric does not have to bear the burden of communication between each and every server even when they are absolutely idle. That makes scaling IP fabric very easy - and that is why all our customers can have their Global Private Networks by default, containing all their servers regardless of the location.

So, L3 Fabric is error-proof, scalable and provides better network uptime than the alternatives.

Now, about the fabric. Fabric in its common meaning is structured fiber. The L3 fabric is very similar to that. Switches in the L3 fabric are interconnected in a manner that allows 3-hop connection between any two servers. Each server is connected to four 'leaf' switches (that is the name for the top-of-rack switch in L3 fabric), two for the public network, and two for the private network. Each leaf switch is connected to two spine switches. Not only that provides the redundancy; it guarantees low latency inside the POD: each server can reach any other server within no more than three hops.

What if I L3 does not work for me, and I still need a private VLAN

Despite all L3 fabric benefits, some customers might require L2 connectivity between their servers. There are several reasons for that - the major one is using some HA tooling, requiring floating IPs shared across several servers - it might be pacemaker/corosync, carp, or some virtualization software - like Xen Server or VMWare.

Using a VXLAN technology, Servers.com has built a service for those who need to have a VLAN for their servers. VXLAN stands for Virtual Extensible LAN. While only 4096 VLANs can coexist in the same infrastructure; VXLAN uplifts this limit significantly, allowing over 16 millions of L2 domains. In Servers.com environment, VXLAN runs over L3 fabric. That allows us to preserve the protection L3 fabric provides against actions of one tenant affecting the network of another one and virtually infinite scalability.

The best part about L2 domains in Servers.com - they are easily configurable. You just create them in the Self-Service Portal, add or remove servers from a VLAN - all with several clicks. No need to write tickets and wait, no need to talk to a support engineer, no risk of human error.

Redundancy

Each layer - leaf, spine, and core, is redundant in Servers.com, both in private, and public networks. Each server is equipped with at least two dual-port NICs. One port on each NIC is used for private network connection, and one is for public. Each port is connected to a different switch - which means, each server has a connection to two public and two private network switches. Every leaf switch is then connected to two spine switches, each of them is connected to one of two routers. On every public network router, several Tier1 carriers are present. Each private network router has at least two connections to the private network. Thanks to this redundancy on each networking layer, Servers.com customer is protected from:

Failure of any carrier;
Failure of any piece of network equipment;
And even failure of a network interface card on a server.

Servers.com fully redundant network infrastructure is cost-efficient, and there are several reasons for that. One of them is our smart cabling - a solution that brought port utilization on switches to 100%, and thus helped reduce the cost. Another one is Servers.com Smart Traffic Dispatcher - allowing us to make use of multiple carrier connections, increasing the quality and preserving the costs reasonable at the same time.