10 Gigabit Networking for the Homelab: Beginner's Guide

Intro

When I first started to get into building a homelab, I didn't realize 10 gigabit was something that was really reasonable to have in a home network. In my mind, a 10 Gigabit connection was something for large datacenters or cloud providers. After looking into it, I found out that 10GbE has been widely used for well over a decade, meaning that there is indeed equipment that is well-supported and decently affordable that can help make this a reality for hobbyists.

If you're a beginner or experienced homelabber, this guide is intended to help you understand the 10 Gigabit landscape, what equipment is out there, and what to look out for in order to get the most performance for your money.

Mikrotik networking equipment in my homelab

This is my 10 Gigabit setup in the homelab. I use a Mikrotik CRS328, which has 4 10Gigabit SFP+ ports and a Mikrotik CRS317, which has 16 10bGigabit SFP+ ports. Both of these switches can be bought for well under $500 (I found them used for under $400 each).  Stay tuned and we'll dive a bit more into these switches, as well as the rest of the equipment necessary to set up a 10 gig network.

Why do I need 10 Gigabit networking in my homelab?

Well, you probably don't. But if you are doing some research on 10 gigabit networking, you probably want it. Here are some reasons why you'd want to have 10 Gigabit speeds in your networking:

• You frequently transfer large files between machines and would like to speed up the process
• You run distributed computing jobs and need to be able to quickly transfer data between cluster machines
• You want to learn about working with different types of networking equipment for your job or simply to learn more

Even if you don't need it, which I admittedly don't, it is still really cool to have that type of speed and power in your home network. And as we're beginning to see, it isn't all that expensive nowadays (relative to all the other stuff you've got in your homelab).

Equipment

Now, let's have a look at the equipment necessary for a 10gigabit setup. We'll have a look at 3 types of items:

• Network switches
• Cables and transceiver modules
• NICs (Network Interface Cards) AKA LAN adapters, network adapters

10 Gigabit Switches

I have two 10 gigabit capable switches in my homelab, the Mikrotik CRS328 and the Mikrotik CRS317:

Inside the CRS328 Switch

As you'll notice the majority of the CRS328's ports are Gigabit Ethernet ports, and it only has 4x 10 gigabit SFP+ ports:

CRS328 internal components

The vast majority of the devices in my network are gigabit (or less) ethernet devices, and so they are connected to the 24 regular old ethernet ports on the switch. Using one or more of the SFP+ ports on the CRS328, a trunk connection can is made with the CRS317, which has 16 10 gigabit SFP+ ports.

Mikrotik CRS317 Switch

It might seem unnecessary to have 10 Gigabit capability on the CRS328 if it is really only being used to connect gigabit ethernet devices. Well, by having a 10 gigabit trunk port connecting the switch to the CRS317, it means that we avoid the CRS328 becoming a low-throughput bottleneck.

Say there are 10 different nodes of a Kubernetes cluster, each connected with a 1 gigabit Ethernet cable to the CRS328. There is also a 10Gigabit-capable server connected to the CRS317 via an SFP+ port. With this setup, you'll be able to get a full gigabit connection to all 10 of the cluster nodes at the same time.

If your ethernet switch was not 10 gigabit capable, and didn't have a trunk port with 10 gigabit throughput, the switch would be a bottleneck, and 1 gigabit of throughput would need to be shared among the 10 cluster nodes communicating with the server on the CRS317.

Ethernet Cables, Modules, and Network Speed

Here is a comparison of the different types of CAT Ethernet cables:

CAT Ethernet Cable Types and Speed

You'll notice that not all CAT Ethernet cables are capable of transmitting 10 gigabits of data, or even 1 gigabit. It's worth noting that Ethernet cables are not the only way to connect machines to your network switch. Here is a table giving a comparison of a few popular choices:

Types of SFP+ Modules

A Direct Attach Cable (DAC) is a type of twinax cable that can be used to make 10 gigabit network connections. In terms of both cost and power efficiency. I have found DAC to be the best choice.

Not only do 10 gigabit RJ-45 connections usually consume more power and generate more heat, but they usually cost significantly more. NICs (which we'll cover in a bit) usually cost significantly more than their SFP+ equivalents, and you'll usually need a transceiver module, like this one:

RJ-45 SFP+ Transceiver

When I first started setting up 10 gigabit networking in my homelab I was insistent on using RJ-45 modules and ethernet cables because I thought it would look nicer and keep my setup simpler.

Eventually I gave up that resistance and started to use purely SFP+ NICs with DAC cables and things went a lot smoother for me. While the RJ45 SFP+ transceivers do work, they get VERY hot. If you are trying to decide which to use for your homelab, I would strongly recommend just going for SFP+/DAC.

Types of 10 Gigabit Ethernet Cards

Now that we know about the different types of connection that can transmit a 10 gigabit data signal, lets look at some NICs or network cards.

First up, the RJ45 10 gigabit ethernet connectors. Here are two cards which I've tried:

ASUS XG-C100C 10G

The ASUS XG-C100C is a decent one that has good out of the box functionality, as is this TP-Link RJ45 card:

TP-Link 10GB PCIe Network Card (TX401)

Both of them sell for around $100 new, which is not too crazy, but is a bit expensive for a network card and carries the same disadvantages mentioned in the previous section. Unless you really need to use an RJ45 connection, I would recommend against this type of card for 10 gigabit networking.

This brings us to SFP+ NICs. This class has what I find to be much more cost effective and efficient cards. One example is this TRENDnet 10 Gigabit SFP+ NIC, which I found worked well out of the box with a couple different linux installations:

TRENDnet 10 Gigabit PCIe SFP+ Network Adapter

By far my first choice, now that I've tried a bunch of 10 gigabit NICs, is the Mellanox ConnectX. When I first found one of these for sale on eBay for $40 shipped, I wasn't sure if it was too good to be true, but the simple fact is there are so many of these cards available from decommissioned servers, they are a great deal:

Mellanox ConnectX-3 10 Gigabit SFP+

The same model can be found on Amazon for about $65, so I would recommend searching for a deal on eBay and buying for anywhere in the range of $35-45 from a seller with good reviews. I haven't had any issues with this card yet.

Now that we've gone through the landscape of 10 gigabit equipment on the market, we ask ourselves this tough question:

Is 10 Gigabit networking worth it for a home network?

For me, it absolutely was. Even though I don't really need it, I don't really need most of the things I have set up in my home network. It was a great way to learn about different types of networking equipment, and I like the idea that I have the throughput available in my homelab when I finally do get around to some distributed computing or data transfer that could benefit from it.

I hope this guide helps you on your homelab and networking journey. As always, please feel free to get in touch with me on twitter @pubudeux if you have any questions or suggestions.