The problem with peering from a logistics standpoint

Many ISPs run into this problem as part of their growing pains.  This scenario usually starts happening with their third or 4th peer.

Scenario.  ISP grows beyond the single connection they have.  This can be 10 meg, 100 meg, gig or whatever.  They start out looking for redundancy. The ISP brings in a second provider, usually at around the same bandwidth level.  This way the network has two pretty equal paths to go out.

A unique problem usually develops as the network grows to the point of peaking the capacity of both of these connections.  The ISP has to make a decision. Do they increase the capacity to just one provider? Most don’t have the budget to increase capacities to both providers. Now, if you increase one you are favouring one provider over another until the budget allows you to increase capacity on both. You are essentially in a state where you have to favor one provider in order to keep up capacity.  If you fail over to the smaller pipe things could be just as bad as being down.

This is where many ISPs learn the hard way that BGP is not load balancing. But what about padding, communities, local-pref, and all that jazz? We will get to that.  In the meantime, our ISP may have the opportunity to get to an Internet Exchange (IX) and offload things like streaming traffic.  Traffic returns to a little more balance because you essentially have a 3rd provider with the IX connection. But, they growing pains don’t stop there.

As ISP’s, especially WISPs, have more and more resources to deal with cutting down latency they start seeking out better-peered networks.  The next growing pain that becomes apparent is the networks with lots of high-end peers tend to charge more money.  In order for the ISP to buy bandwidth they usually have to do it in smaller quantities from these types of providers. This introduces the probably of a mismatched pipe size again with a twist. The twist is the more, and better peers a network has the more traffic is going to want to travel to that peer. So, the more expensive peer, which you are probably buying less of, now wants to handle more of your traffic.

So, the network geeks will bring up things like padding, communities, local-pref, and all the tricks BGP has.  But, at the end of the day, BGP is not load balancing.  You can *influence* traffic, but BGP does not allow you to say “I want 100 megs of traffic here, and 500 megs here.”  Keep in mind BGP deals with traffic to and from IP blocks, not the traffic itself.

So, how does the ISP solve this? Knowing about your upstream peers is the first thing.  BGP looking glasses, peer reports such as those from Hurricane Electric, and general news help keep you on top of things.  Things such as new peering points, acquisitions, and new data centers can influence an ISPs traffic.  If your equipment supports things such as netflow, sflow, and other tools you can begin to build a picture of your traffic and what ASNs it is going to. This is your first major step. Get tools to know what ASNs the traffic is going to   You can then take this data, and look at how your own peers are connected with these ASNs.  You will start to see things like provider A is poorly peered with ASN 2906.

Once you know who your peers are and have a good feel on their peering then you can influence your traffic.  If you know you don’t want to send traffic destined for ASN 2906 in or out provider A you can then start to implement AS padding and all the tricks we mentioned before.  But, you need the greater picture before you can do that.

One last note. Peering is dynamic.  You have to keep on top of the ecosystem as a whole.

Transit, peer, downstream..what do they all mean?

As a service provider you have a mountain of terms to deal with. As you dive into the realm of BGP, you will hear many terms in regards to peers.  Knowing their names AND your definition of them will serve you well.  I emphasized the and in the last sentence because many people have different definitions of what these terms means. This can be due to how long they have been dealing with networks, what they do with them, and other such things.  For example, many content providers use the term transit differently than an ISP.  So, let’s get on to it.

Transit or upstream
This is what you will hear most often.  A transit peer is someone who you go “through” in order to reach the internet.  You transit their network to reach other networks.  Many folks use the term “upstream provider” when talking about someone they buy their internet from.

Downstream
Someone who is “downstream” is someone  you are providing Internet to.  They are “transiting” your network to reach the Internet.  This is typically someone you are selling Internet to.

Peer
This is the term which probably needs the most clarification when communicating with others about how your BGP is setup.  A peer is most often used as a generic term, much like Soda (or pop depending on where you are from). For example someone could say:
“I have a peer setup with my upstream provider who is Cogent.” This is perfectly acceptable when used with the addition of “my upstream provider”.  Peers are often referred to as “neighbors” or “BGP neighbors”.

Local or Private Peer
So what is a local peer? A local peer is a network you are “peering” with and you are only exchanging routes which are their own or their downstream networks.  A local peer usually happens most often at an Internet Exchange (IX) but can happen in common points where networks meet. The most important thing that defines a local peer is you are not using them to reach IP space which is not being advertised form their ASN.   Your peering relationship is just between the two of you. This gets a little muddy when you are peering on an IX, but thats being picky.

I have trained myself to qualify what I mean by a peer when talking about them. I will often say a “transit peer” or a “local peer”. This helps to add a little bit of clarity to what you mean.

Why is this all important? For one, it helps with keeping everyone on the same page when talking about peering.  I had a case a few weeks ago where a Content provider and I wasted configuration time because our definition of transit was different.  Secondly, you want to be able to classify your peers so you can apply different filter rules to them. For example, with a downstream peer you only want to accept the IP space they have shown you which is their own.  That way you are not sending your own transit traffic over their network. This would be bad.  However, if you are accepting full routes from your transit provider, you want your filters to accept much more IP than a downstream provider. So if you have a team being able to be on the same page about peers will help when it comes to writing filters, and how your routers “treat” the peer in terms of access lists, route filters, etc.

Internet Routing Registries

Routing Registries are a mysterious underpinning of the peering and BGP world. To many they are arcane and complicated. If you have found this article you are at least investigating the use of a registry. Either that or you have ran out of fluffy kittens to watch on YouTube. Either way one of the first questions is “Why use a routing registry”.

As many of us know BGP is a very fragile ecosystem. Many providers edit access lists in order to only announce prefixes they have manually verified someone has the authority to advertise. This is a manual process for many opportunities for error. Any time a config file is edited errors can occur. Either typos, misconfiguration, or software bugs.

Routing registries attempt to solve two major issues. The first is automating the process of knowing who has authority to advertise what. The second is allowing a central repository of this data.

So what is a routing Registry?
From Wikipedia: An Internet Routing Registry (IRR) is a database of Internet route objects for determining, and sharing route and related information used for configuring routers, with a view to avoiding problematic issues between Internet service providers.

The Internet routing registry works by providing an interlinked hierarchy of objects designed to facilitate the organization of IP routing between organizations, and also to provide data in an appropriate format for automatic programming of routers. Network engineers from participating organizations are authorized to modify the Routing Policy Specification Language (RPSL) objects, in the registry, for their own networks. Then, any network engineer, or member of the public, is able to query the route registry for particular information of interest.

What are the downsides of a RR?
Not everyone uses routing registries. So if you only allowed routes from RR’s you would get a very incomplete view of the Internet and not be able to reach a good amount of it.

Okay, so if everyone doesn’t use it why should i go to the trouble?
If you are at a formal Internet Exchange (IX) you are most likely required to use one. Some large upstream providers highly encourage you to use one to automate their process.

What are these objects and attributes?
In order to partipate you have to define objects. The first one you create is the maintainer object. This is what the rest of the objects are referenced to and based from. Think of this as setting up your details in the registry.

From this point you setup “object types”. Object types include:
as-set
aut-num
inet6num
inetnum
inet-rtr
key-cert
mntner
route
route6
route-set
If you want to learn more about each of these as well as templates visit this ARIN site.

So what do I need to do to get started?
The first thing you need to do is setup your mntner object in the registry. I will use ARIN as our example. You can read all about it here:https://www.arin.net/resources/routing/.

You will need a couple of things before setting this up
1.Your ARIN ORGID
2.Your ADMIN POC for that ORGID
3.Your TECH POC for that ORGID

Once you have these you can fill out a basic template and submit to ARIN.

mntner: MNT-YOURORGID
descr: Example, Inc.
admin-c: EXAMPLE123-ARIN
tech-c: EXAMPLE456-ARIN
upd-to: hostmaster@example.net
mnt-nfy: hostmaster@example.net
auth: MD5-PW $1$ucVwrzQH$zyamFnmJ3XsWEnrKn2eQS/
mnt-by: MNT-YOURORGID
referral-by: MNT-YOURORGID
changed: hostmaster@example.net 20150202
source: ARIN

The templates is very specific on what to fill out. The mnt-by and referral-by are key to following instructions. MD5 is another sticking point. The process is documented just in a couple of places. In order to generate your MD5-PW follow these instructions.

1.Go to https://apps.db.ripe.net/crypt/ Enter in a password. Make sure you keep this cleartext password as you will need it when sending future requests to ARIN’s Routing Registry.
2.Submit the password to get the md5 crypt password. Keep this password for your records, as you may need it when interacting with ARIN’s IRR in the future.
3.Add the following line to your mntner object template in the text editor.
auth: MD5-PW
Our example above has a MD5 password already generated.
Once this is done and created you can add objects. The most commonly added objects are your ASN and IP space.

Create your ASN object using the as-num template

aut-num: AS65534
as-name: EXAMPLE-AS
descr: Example, Inc.
descr: 114 Pine Circle
descr: ANYWHERE, IN 12345
descr: US
import: from AS65535 accept ANY
import: from AS65533 accept AS65534
export: to AS65533 announce ANY
export: to AS65535 announce AS2 AS65533
admin-c: EXAMPLE456-ARIN
tech-c: EXAMPLE123-ARIN
mnt-by: MNT-YOURORGID
changed: user@example.com 20150202
source: ARIN
password:

The things to know about the above template are the import and export attributes.

Now on to adding IP space
Suppose you have IP space of 192.0.2.0/24 Your template would look like:

inetnum: 192.0.2.0 – 192.0.2.255
netname: EXAMPLE-NET
descr: Example, Inc.
descr: 115 Oak Circle
descr: ANYWHERE, IN 12345
country: US
admin-c: EXAMPLE123-ARIN
tech-c: EXAMPLE456-ARIN
notify: user@example.com
mnt-by: MNT-YOURORGID
changed: user@example.com 20150202
source: ARIN
password:

The password attribute is the cleartext password for your MD5 key.

Further Reading:
Using RPSL in practice

NANOG IRR

MTIN WISPA Announcement

MTIN would like to announce some exciting new services for ISPs and network operators

The first is Midwest Internet Exchange ( www.midwest-ix.com )

MidWest-IX has created a peering fabric we are expanding to data centers focused on the needs of WISPs and network operators such as yourself. Peering can be a valuable, and cost-effective solution for your ISP. MidWest-IX has created a solution based around those needs.

We are tailoring and providing these services to the WISP community as a way of making everyone stronger. WISP operators need advantages. MidWest-IX can provide lower latency to content providers such as Netflix. MidWest-IX can cut down on transit costs through peering. We are also creating an ever-growing marketplace for members to provide redundancy, market goods and services to each other, and create a WISP peering cloud. We have many more benefits of an exchange listed at: http://www.midwest-ix.com/benefits.html

We have exchange services available at:

350 East Cermak Chicago Illinois
733 West Henry Street Indianapolis Indiana
401 North Shadeland, Indianapolis Indiana
900 Walnut St. Louis Missouri
535 Scherers Court Columbus, Ohio

If you are a WISP in Indiana, Ohio, Illinois, Kentucky, Michigan, or Missouri contact us on how we can leverage the exchange to help your business. Other locations planned for 2015.

Our next announcement includes services in several Data Centers.

MTIN in cooperation with Midwest Internet Exchange (M-IX) offers co-Location, bandwidth, peering, transport, and managed services.

Do you have a need for circuit termination, server/router space, or peering in any of the above locations? Let us put together a managed solution for you. MTIN can handle the ins and outs of cross-connects, facilitating ports to the exchange fabric, and other data center needs. A data center can be an intimidating thing. Let us take the guesswork out of it for you.

Services Include
-Bandwidth (let our experts provide unique and out-of-the-box solutions)
-Transport
-Cross connects and cable landings
-Off-site backup and DR
-Co-location (TierIV and basic Co-location)
-Connections to 3rd Party networks such as Internet2
-BGP Peering

MTIN provides xISP consulting and backend solutions. BGP, OSPF, routing, DNS, network engineering, and other services. Talk to us how we can put together a complete solution to optimize your network. Our Engineers can design a cost-efficte solution that fits you and your needs.

Contact us:

Web: www.mtin.net
Web: www.midwest-ix.com
e-mail: support@mtin.net
Tel: 317.644.2224

MTIN announces IPv6 peering in Indianapolis, Indiana

MTIN is happy to announce free ipV6 peering in Indianapolis.  If you are a provider with a presence at 733 Henry Street Indianapolis, Indiana we will peer on the IPv6 backbone with you for free.  All you have to do is make it to us via a cross connect.  If you want to peer on IPv4, and are not already, we might be able to assume the cost of that cross-connect.

Contact MTIN for details.