The problem with speedtests

Imagine this scenario. Outside your house, the most awesome super highway has been built.  It has a speed limit of 120 Mile Per Hour.  You calculate at those speeds you can get to and from work 20 minutes earlier. Life is good.  Monday morning comes, you hop in your Nissan GT-R, put on some new leather driving gloves, and crank up some good driving music.  Your pull onto the dedicated on-ramp from your house and are quickly cruising at 120 Miles an hour. You make it into work before most anyone else. Life is good.  

Near the end of the week, you notice more and more of your neighbours and co-workers using this new highway.  Things are still fast, but you can’t get up to speed to work like you could earlier in the week.  As you ponder why you notice you are coming up on the off-ramp to your work.  Traffic is backed up. Everyone is trying to get to the same place.  As you are waiting in the line to get off the super highway, you notice folks passing you by going on down the road at high rates of speed.  You surmise your off-ramp must be congested because it is getting used more now.

Speedtest servers work the same way. A speedtest server is a destination on the information super-highway. Man, there is an oldie term.  To understand how speedtest servers work we need a quick understanding of how the Internet works.   The internet is basically a bunch of virtual cities connected together.  Your local ISP delivers a signal to you via Wireless, Fiber, or some sort of media. When it leaves your house it travels to the ISP’s equipment and is aggregated with your neighbours and sent over faster lines to larger cities. It’s just like a road system. You may get access via a gravel road, which turns into a 2 lane blacktop, which then may turn into a 4 lane highway, and finally a super-highway.  The roads you take depend on where you are going. Your ISP may not have much control over how the traffic flows once it leaves their network.

Bottlenecks can happen anywhere. Anything from fiber optic cuts, oversold capacity, routing issues, and plain old unexpected usage. Why are these important? All of these can affect your speedtest results and can be totally out of control of your ISP and you.  They can also be totally your ISP’s fault. They can also be your fault, just like your car can be.  An underpowered router can be struggling to keep up with your connection. Much like a moped on the above super-highway can’t keep up with a 650 horsepower car to fully utilize the road, your router might not be able to keep up either.  Other things can cause issues such as computer viruses, and low performing components.

Just about any network can become a node or a node with some of the other speedtest sites.  These networks have to meet minimum requirements, but there is no indicator of how utilized these speedtest servers are.  A network could put up one and it’s 100 percent utilized when you go running a speedtest. This doesn’t mean your ISP is slow, just the off-ramp to that speedtest server is slow.

The final thing we want to talk about is the utilization of your internet pipe from your ISP.  This is something most don’t take into consideration.  Let’s go back to our on-ramp analogy.  Your ISP is selling you a connection to the information super-highway.   Say they are selling you a 10 megabyte download connection.  If you have a device in your house streaming an HD Netflix stream, which is typically 5 megs or so, that means you only have 5 megs available for a speedtest while that HD stream is happening. Speedtest only test your current available capacity.  Many folks think a speedtest somehow stops all the traffic on your network, runs the test, and starts the traffic. It doesn’t work that way. A speedtest tests the available capacity at that point in time.  The same is true for any point between you and the speedtest server.  Remember our earlier analogy about slowing down when you got to work because there were so many people trying to get there.  They exceeded the capacity of that destination.  However, that does not mean your connection is necessarily slow because people were zooming past you on their way to less congested destinations.

This is why speedtest results should be taken with a grain of salt. They are a useful tool, but not an absolute. A speedtest server is just a destination.  That destination can have bottlenecks, but others don’t.  Even after this long article, there are many other factors which can affect Internet speed. Things we didn’t touch on like Peering, the technology used, speed limits, and other things can also affect your internet speed to destinations.

Some Random Visio diagram

Below, We have some visio diagrams we have done for customers.

This first design is a customer mesh into a couple of different data centers. We are referring to this as a switch-centric design. This has been talked about in the forums and switch-centric seems like as good as any.

This next design is a netonix switch and a Baicells deployment.

Design for a customer

Use tarpit vs drop for scripts blocking attackers

There are many scripts out there, especially on Mikrotik, which list drop as the action for denying bad guy traffic.  While this isn’t wrong, you could put the tarpit action to better use for actions which are dropping attacking type of traffic.

So what is Tarpit?
Tarpit is fairly simple. When connections come in and are “tarpitted” they don’t go back out. The connection is accepted, but when data transfer begins to happen, the TCP window size is set to zero.  This means no data can be transferred during the session.  The session is held open, and requests from the sender (aka attacker) to close the session are ignored. They must wait for the connection to timeout.

So what’s the downside?
TCP is not really designed to hold onto a connection.  It can be additional overhead on a taxed system.  Most modern firewalls can handle tarpitting without an issue. However, if you get thousands of connections it can overwhelm a system or a particular protocol.

How can I use it?
If you have scripts, such as the SSH drop off the Mikrotik wiki, simply change the action to “tarpit” instead of “drop”.

The state of Data Centers and Co-Location in Indianapolis

We like to refer to Indianapolis, Indiana as an “NFL  City” when explaining the connectivity and peering landscape.  It is not a large network presence like Chicago or Ashburn but has enough networks to make it a place for great interconnects.

At the heart of Indianapolis is the Indy Telcom complex. (currently down as of this writing).  This is also referred to as the “Henry Street” complex because West Henry Street runs past several of the buildings.   This is a large complex with many buildings on it.

One of the things many of our clients ask about is getting connectivity from building to building on the Indy Telcom campus. Lifeline Data Centers ( ) operates a carrier hotel at 733 Henry. With at least 30 on-net carriers and access to many more 733 is the place to go for cross-connect connectivity in Indianapolis.   We have been told by Indy Telcom the conduits between the buildings on the campus are 100% full. This makes connectivity challenging at best when going between buildings. The campus has lots of space, but the buildings are on islands if you wish to establish dark fiber cross-connects between buildings. Many carriers have lit services, but due to the ways many carriers provision things getting a strand, or even a wave is not possible.  We do have some options from companies like Zayo or Lightedge for getting connectivity between buildings, but it is not like Chicago or other big Date centers.  However, there is a solution for those looking for to establish interconnections.   Lifeline also operates a facility at 401 North Shadeland, which is referred to as the EastGate facility. This facility is built on 41 acres, is FEDRAMP certified, and has a bunch of features.  There is a dark fiber ring going between 733 and 401.  This is ideal for folks looking for both co-location and connectivity.  Servers and other infrastructure can be housed at Eastgate and connectivity can be pulled from 733.  This solves the 100% full conduit issue with Indy Telcom. MidWest Internet Exchange ( ) is also on-net at both 401 and 733.

Another location where MidWest-IX is at is  365 Data Centers ( ) at 701 West Henry.  365 has a national footprint and thus draws some different clients than some of the other facilities.  365 operates Data centers in Tennessee, Michigan, New York, and others. MidWest has dark fiber over to 365 in order to bring them on their Indy fabric.

Another large presence at Henry Street is Lightbound ( ).  They have a couple of large facilities. According to PeeringDB, only three carriers are in their 731 facility.   However, their web-site lists 18+ carriers in their facilities. The web-site does not list these carriers.

I am a big fan of peeringdb for knowing who is at what facilities, where peering points are, and other geeky information.  Many of the facilities in Indianapolis are not listed on peering DB.  Some other Data Centers which we know about:

Zayo (
LightTower ( )
Indiana Fiber Network (IFN) (
Online Tech ( )

On the north side of Indianapolis, you have Expedient ( ) in Carmel. Expedient says they have “dozens of on net carriers among all markets”.  There are some other data centers in the Indianapolis Metro area. Data Cave in Columbus is within decent driving distance.

Where does Trill and VXLAN fit in your strategy?

As networking trends yo-yo between layer-3 and layer-2,  different protocols have emerged to address issues with large layer-2 networks. Protocols such as Transparent Interconnection of Lots of Links (TRILL), Shortest Path Bridging (SPB), and Virtual Extensible LAN (VXLAN) have emerged to address the need for scalability at Layer2.   Cloud scalability, spanning tree bridging issues, and big broadcast networks start to become a problem in a large data center or cloud environment.

To figure out if things like TRILL is a solution for you, you must understand the problem that is being addressed by TRILL. The same goes for the rest of the mentioned protocols. When it boils down to it the reason for looking at such protocols is you want high switching capacity, low latency, and redundancy.  The current de facto standard of Spanning Tree Protocol (STP) simply is unable to meet the needs of modern layer2 networks.  TRILL addresses the problem of STP’s ability to only allow one network path between switches or ports.  STP prevents loops by managing active layer -2 paths.   TRILL applies Intermediate System-to-Intermediate System protocol (IS-IS), which is a layer3 routing protocol translated to Layer 2 devices.

For those who say TRILL is not the answer things like SPB also known as 802.1aq, and VXLAN are the alternatives. A presentation at NANOG 50 in 2010 addressed some of the SPB vs TRILL debate. This presentation goes into great detail on the differences between the two.

The problem, which is one most folks overlook, is that you can only make a layer 2 network so flat.  The trend for a while, especially in data centers, is to flatten out the network. Is TRILL better? Is SPB better? The problem isn’t what is the better solution to use.  What needs to be addressed is the design philosophy behind why you need to use such things.   Having large Layer2 networks is generally a bad idea. Scaling issues can almost always be solved by Layer-3.

So, and this is where the philosophy starts, is TRILL, SPB, or even VXLAN for you? Yes, but with a very big asterisk. TRILL is one of those stop-gap measures or one of those targeted things to use in specific instances. TRILL reduces complexity and makes layer-2 more robust when compared to MLAG. Where would you use such things? One common decision of whether to use TRILL or not comes in a virtualized environment such as VSPHERE.

Many vendors such as Juniper, have developed their own solutions to such things.  Juniper and their Virtual Chassis solution do away with spanning tree issues, which is what TRILL addresses.   Cisco has FabricPath, which is Cisco’s proprietary TRILL-based solution. Keep in mind, this is still TRILL.   If you want to learn some more about Fabric Path this article by Joel Knight gets to the heart of Fabric path.

Many networks see VXLAN as their upgrade path.  VXLAN allows layer 2 to be stretched across layer 3 boundaries. If you are a “Microsoft person” you probably hear an awful lot about Network Virtualization using Generic Routing Encapsulation (NVGRE) which can encapsulate a layer two frame into IP.

The last thing to consider in this entire debate is how does Software Defined Networking (SDN) play into this. Many folks think controllers will make ECMP and MLAG easy to create and maintain. If centralized controllers have a complete view of the network there is no longer a need to run protocols such as TRILL.   The individual switch no longer makes the decision, the controller does.

Should you use Trill, VXLAN, or any of the others mentioned? If you have a large Layer-2 virtualized environment it might be something to consider.  Are you an ISP, there is a very small case for running TRILL in anything other than your data center. Things such as Carrier Ethernet and MPLS are the way to go.

Netflix, IPv6, and queing

While trying to get my Playstation to download the latest “No Man’s Sky” download quicker I figured I would share a little torch action.  This is showing my wife’s Ipad talking to Netflix while she is watching a streaming TV show. Keep in mind this is just an Ipad, not some 4k TV.

Some things to note as you watch this (no sound).

1.Uncapped the connection bursts to 50-60+ megs.
2.The slower your que the connection the more time it spends downloading data.  At slower ques the bursts last longer.
3.If you are handing out IPv6 to customers you should be queing them as well.

Just something to quick and dirty to keep in mind.

ethernet MTU and overhead

One of the most common questions is how much overhead do I need to account for on my transport network? I have put together a quick list to help when you are calculating your overhead.

-GRE (IP Protocol 47) (RFC 2784): 24 bytes (20 byte IPv4 header, 4 byte GRE header)
-6in4 encapsulation (IP Protocol 41, RFC 4213): 20 bytes
-4in6 encapsulation (e.g. DS-Lite RFC 6333): 40 bytes
Addition IPv4 header:20 bytes
-IPsec encryption:
73 bytes for ESP-AES-256 and ESP-SHA-HMAC overhead (overhead depends on transport or tunnel mode and the encryption/authentication algorithm and HMAC)
-MPLS: 4 bytes for each label in the stack
-802.1Q tag: 4 bytes
Q-in-Q: 8 bytes
-VXLAN: 50 bytes
-OTV: 42 bytes

Some rules of thumb when setting MTUs. You won’t get fragmentation if your L2 MTU is higher than your L3 MTU. This is just not the setting, but the actual overhead in use. Just setting it to a number doesn’t necessarily make it right. The above list will help you calculate the minimum MTU you may need. I try to get gear that supports a 1548 MTU and set everything to that. Makes it simple. I still want to know how much MTU I am utilizing because it helps me validate my designs.
The most important rule of thumb is you won’t get fragmentation if your l3 MTU is less than your L2 MTU.

BGP local Pref and you

One of the bgp topics that comes up from time to time is what does “bgp local-pref” do for me? The short answer is it allows you to prefer which direction a traffic will flow to a given destination. How can this help you? Well before we start, remember the high number wins in local-pref.
Let’s assume you are an ISP. You have the following connections:
-You supply a BGP connection to a downstream client.
-You have a private peer setup with the local college
-You are hooked into a local internet exchange
-You have transport to another internet exchange in the next state over
-and you have some transit connections where you buy internet.

So how do we use BGP preference to help us out? We might apply the following rules to routes received from our various peers
Our downstream client we might set their local pref to 150
The college we may set them to 140
Preferred internet exchange peering: 130
Next state IX: 120
Transit ISPs: 100

Now these don’t make much sense by themselves, but they do when you take into account how BGP would make a decision if it has to choose between multiple paths. If it only has one path to a certain route then local-pref is not relevant.

Let’s say you have a customer on your ISP that is sending traffic to a server at a local college. Maybe they are a professor who is remoting into a server at the college to run experiments. There are probably multiple ways for that traffic to go. If the college is on the local Internet exchange you are a member of, that is one route, the next route would be your transit ISPs, and obviously your private peer with the college. So, in our example above the college, with a local pref of 140 wins out over the local exchange, wins out over the next state IX, and wins out over the Transit ISPs. We want it to go direct over the direct peer with the college. Mission accomplished.

local-pref is just one way to engineer your traffic to go out certain links. Keep in mind two things:
1.Higher number wins
2.local-pref only matters if there are multiple paths to the same destination.
3.Local-pref has to do with outbound path selection