Networking foundations

In an article earlier today, I wrote about certifications and the ISP.  The biggest area I see folks go wrong when it comes to networking is having a good understanding of design. One of the analogies I like to use is building a house.   You have several key roles when you build a house.  These can be directly applied to the networking world.

The first is the Architect. Everything starts with this person or team’s vision. The Architect lays out the design and how the network will function. This person needs a wide variety of skills.  They know the product lines they are supporting, they know how these products fit into the overall vision of the network, and they know the limitations of what they are working with, to name just a few of their many skills. These are your CCIE and higher level CCNP folks in certification terms.  They have enough breadth of knowledge to see the entire picture.  Not just what the device in front of them can do. A network architect would know that a certain wireless CPE does not fully support a VPLS tunnel, and would either recommend not using that equipment or come up with a workaround to implement it into the network, if in fact, they were using VPLS.  Many large companies have Architects who implement and validate network designs. These are then pushed to the next group of folks to implement.

The “tradesmen” or “tradeswomen” are the ones who actually implement the designs around the Architects blueprints for the network.  Just like a house who has carpenters, brick layers, and roofers, so do you have folks who know wireless, routing, security, and other disciplines.  These are the folks who can make the machines do what they want them to do. They work off the blueprints to actually make the network talk and function according to the Architects design. these are your CCNA and CCNP level folks. These folks know the equipment configuration in and out.  They are the most responsible for making things work, and knowing how to make it work. The more experienced of these folks typically collaborate with the Architects to provide expert opinions on the latest features of the equipment they are implementing or any limitations of the equipment.

Many folks working in the ISP world wear multiple hats at the same time.   There is nothing whatsoever wrong with this.  You just have to know the limitations of yourself and the things you have to work with. I see multiple illustrations of this on a daily basis. Clients take a router and make it do BGP, OSPF, PPPoE termination, firewall rules, and other things.  Sometimes this is a budgetary thing, maybe just a lack of understanding, or it can even be a sales hype thing. however, not having an understanding of the design and architecture of a network can be costly.

Anyone can build a house. Go to the lumberyard and get some materials, some tools, and watch a few YouTube videos. Bam you are set.  That will probably work until the first time it rains, or it gets cold. Then you are wondering where your design went wrong.  Using plastic on your roof sounded like a good idea until the wind ripped it.  Same can be said in networks.  Start and always have design considerations in mind.  Not just design, but how individual components are best used in that design. Then rely on the tradesmen  to implement them.  You might be one and the same, but don’t wing it as you go.

Learning, certifications and the xISP

One of the most asked questions which comes up in the xISP world is “How do I learn this stuff?”.   Depending on who you ask this could be a lengthy answer or a simple one sentence answer.  Before we answer the question, let’s dive into why the answer is complicated.

In many enterprise environments, there is usually pretty standard deployment of networking hardware.  Typically this is from a certain vendor.  There are many factors involved. in why this is.  The first is total Cost of Ownership (TCO).  It almost always costs less to support one product than to support multiples.  Things like staff training are usually a big factor.  If you are running Cisco it’s cheaper to train and keep updated on just Cisco rather than Cisco and another vendor.

Another factor involved is economies of scale.  Buying all your gear from a certain vendor allows you to leverage buying power. Quantity discounts in other words.  You can commit to buying product over time or all at once.

So, to answer this question in simple terms.  If your network runs Mikrotik, go to a Mikrotik training course.  If you run Ubiquiti go to a Ubiquiti training class.

Now that the simple question has been answered, let’s move on to the complicated, and typically the real world answer and scenario.  Many of our xISP clients have gear from several vendors deployed.  They may have several different kinds of Wireless systems, a switch solution, a router solution, and different pieces in-between.  So where does a person start?

We recommend the following path. You can tweak this a little based on your learning style, skill level, and the gear you want to learn.

1.Start with the Cisco Certified Network Associate (CCNA) certification in Routing and Switching (R&S).  There are a ton of ways to study for this certification.   There are Bootcamps (not a huge fan of these for learning), iPhone and Android Apps (again these are more focused on getting the cert), online, books, and even youtube videos. Through the process of studying for this certification, you will learn many things which will carry over to any vendor.  Things like subnetting, differences between broadcast and collision domains, and even some IPV6 in the newest tracks.  During the course of studying you will learn, and then reinforce that through practice tests and such.  Don’t necessarily focus on the goal of passing the test, focus on the content of the material.  I used to work with a guy who went into every test with the goal of passing at 100%.  This meant he had to know the material. CompTIA is a side path to the Cisco CCNA.  For reasons explained later, COMPTIA Network+ doesn’t necessarily work into my plan, especially when it comes to #3. I would recommend COMPTIA if you have never taken a certification test before.

2.Once you have the CCNA under your belt, take a course in a vendor you will be working the most with.  At the end of this article, I am going to add links to some of the popular vendor certifications and then 3rd party folks who teach classes. One of the advantages of a 3rd party teacher is they are able to apply this to your real world needs. If you are running Mikrotik, take a class in that. Let the certification be a by-product of that class.

3.Once you have completed #1 and #2 under your belt go back to Cisco for their Cisco Certifed Design Associate (CCDA). This is a very crucial step those on a learning path overlook.  Think of your networking knowledge as your end goal is to be able to build a house.  Steps one and two have given you general knowledge, you can now use tools, do some basic configuration.  But you can’t build a house without knowing what is involved in designing foundations,  what materials you need to use, how to compact the soil, etc.  Network design is no different. These are not things you can read in a manual on how to use the tool.  They also are not tool specific.   Some of the things in the Cisco CCDA will be specific to Cisco, but overall it is a general learning track.  Just follow my philosophy in relationship to #1. Focus on the material.

Once you have all of this under your belt look into pulling in pieces of other knowledge. Understanding what is going on is a key to your success.  If you understand what goes on with an IP packet, learning tools like Wireshark will be easier.  As you progress let things grow organically from this point.  Adding equipment in from a Vendor? Update your knowledge or press the new vendor for training options.  Branch out into some other areas ,such as security, to add to your overall understanding.

Never stop learning! Visit our online store for links to recommend books and products.

WISP Based Traning Folks.
These companies and individuals provide WISP based training. Some of it is vendor focused. Some are not.  My advice is to ask questions. See if they are a fit for what your goals are.
-Connectivity Engineer
Butch Evans
Dennis Burgess
Rickey Frey
Steve Discher
Baltic Networks

Vendor Certification Pages

If you provide training let me know and I will add you to this list.

ePMP elevate webinar

Some highlights on the ePMP Elevate platform.

-Allows ePMP Elevate software to run on on non-Cambium 802.11n subscriber modules
-ePMP Elevate subscribers function as ePMP subscribers
-Solution for WISPs who have hardware deployed. Offers a migration plan.
-ePMP AP must be licensed for ePMP Elevate
-5GHZ Only
-XW and XM based Ubiquiti hardware
-17 supported models
-2.4 GHZ is in on the roadmap. No official comment on timeframe.
-cnMaestro support is available for XW hardware
-Replaces the UBNT software with a cambium interface. Looks pretty close to any other ePMP interface.
-If DFS is FCC certified on the hardware it will work on the hardware when it gets the elevate system.


1 Subscriber license $35
10 Subscriber License $315
You would buy multiples 10 subscriber licenses for more than 10 subs.  Locked to wired mac Address of the AP.





Dirty Cow is Coming – Update your *nix boxes

Dirty COW (CVE-2016-5195) is a privilege escalation vulnerability in the Linux Kernel.

This is an old vulnerability but appears to be something being exploited regularly.  In otherwords, keep your stuff up-to-date.

Check to see if your systems are vulnerable:

Apple has abandoned us..

I have been a long time Apple fan. I don’t think of myself as a Fan Boy, but a fan.  My first Mac was a Performa 6200.  I have been anxiously awaiting the refresh of the MacBook pro.  I am using a 15” Macbook pro from 2008.  It has been upgraded with an SSD and more ram, but is starting to see some age.

From a stockholder perspective, I see several fails from Apple.  The first, and most egregious of these is the removal of the 3.5mm jack from the iPhone.  Two years ago Apple bought Beats by Dre for 3 Billion dollars.  Guess What? Most of the product line uses 3.5mm plugs. Apple just killed a huge upsell to their user base by killing the 3.5mm plug.  Sure, you can get wireless Beats, but those are very high end and not everyone wants wireless.  So why did Apple spend 3 billion dollars on a highly popular brand, which now does not work with their product line? It’s a major let down of shareholders because it is more missed opportunities.

Now, let’s move on to what started this rant.  The new Macbook pros.  My current setup is my previously mentioned 2008 model. It has dual monitors, 2 usb hubs, and a hardwired Ethernet connection routinely plugged in. At the very least I need dongles. The new USB-C ports on the new MacBook Pro models means I can no longer plug my iPhone directly into my laptop for charging or syncing.  Syncing is no big deal as wifi syncing is kind of working.  I am not a big cloud user so I have definite benefits to a cable sync.  However, the charging aspect of it is very handy.  I spend a fair amount of time in the field.  I routinely visit data centers, remote network locations, and backrooms where network equipment is shoe-horned into.  Most of the time power outlets are few and far in-between.  I am lucky to have a power outlet to keep my MacBook going. This means I tend to plug my phone into the laptop to charge. I can no longer do this. #fail

Another major issue is the removal of the escape key.  This is more symbolic of how Apple has abandoned the power user than function.  Anyone who has worked in Linux probably uses the VI text editor. Using the escape key is an essential key to this software.   I also routinely log-in to network routers.  One of the first things you are presented with many of them is “press escape to get started”.  I am already having to use an old terminal program and a USB to serial adaptor in most cases.  I worry about how adding and escape key to the touch bar is going to work on an older setup like this.

At this point in my thinking process I have decided I needed to plan out what all I need to make a new MacBook Pro work with my current setup.  I figure I need to come up with several dongles.  I need one for each of my external monitors, one for Ethernet, and something to replace the connections to my 2 8 port USB hubs.  These hubs have hard drives, a usb headset, usb mic, accept flash drives, and a card reader. I am fully expecting I need a home dock and some dongles to take on the road with me.  I routinely use Ethernet in my day to day job activities.  So off to the Apple store I go.  This is where I run into the next fail.  I pick out the MacBook I wish to purchase and there are no “Suggested accessories” like with previous models.  You used to be able to add on dongles and such right from the same screen. You can no longer do this.  Again, this is a major fail on the part of upselling the customer.  Here I am, wanting to purchase additional dongles and I have to go hunting for them. After many frustrating minutes of digging I found a SB-C to VGA adaptor, but not a DVI adaptor.  The whole experience was frustrating.  Now, I have to treck into a dreaded Apple store and hope one of the folks in there understand what I am talking about. This is the second stockholder fail I see.  Lost revenue from a lack of upselling options.

I realize the trend tends to be leaning toward mobile and phone.   There is still a huge segment of the business population which gets work done on desktop and laptops.  Abandoning them is a sure fire way to drive away further sales.  If the I.T. decision makers at companies are not using your product, that loses direct revenue.  This reaches much further, though.  More and more folks are looking for integrated solutions.  If they see work is buying a certain brand of computers, you tend to buy that brand.  Then you tend to buy that brand for your children and integrate it into your homes.  Brand awareness plays into this as well. Beats are a popular brand and if they have to have dongles to work it creates a shoddy looking product. Why buy a dongle when I can plug it directly into my new google phone? Why would work buy products they have to stock an Apple version of and an everyone else version.

Not only has Apple abandoned the power user, but they are failing the shareholders as well.  Their job is to maximize the value of the company for us shareholders.   Apple needs to innovate, but taking stuff away is not the way to do it.  I remember the days when you had to have a special “Apple microphone” to plug into a mac.  The plug was slightly longer.  This caused issues and much confusion.  I know people who did not buy mac products because of this attitude.  It wasn’t about the microphone. It was the fact everything was proprietary. Apple is an innovator, but forcing people into their way of thinking can backfire.  I can pay off big on the flip side.  In this case I think this will be the day the Macbook died.  Unless Apple changes things the Macbook will have a place in education, like they always have. Once folks get out into the business world, the value of a portless, escape-key-less laptop will hamper them. I hope I am wrong.

Metro Ethernet Terms

As some of you reading this dive into metro ethernet you should know some terminology

• User-Network Interface (UNI): The UNI is a physical Ethernet port on the service provider side of the network along with a predefined set of parameters to provide data, control and management traffic exchange with the end-customer CPE device. The customer CPE device can be a Layer 2 Ethernet switch, Layer 3 routing node or some of LTE nodes.

• Network-to-Network Interface (NNI):  NNI is represented by the physical Ethernet port on the service provider access node that is used to interconnect two Ethernet MANs of two different service providers. We are also using E-NNI as a reference point for the interconnection of Layer 2 MAN service with Layer 3 service nodes—the provider edge router (PE), a broadband network gateway (BNG), vertical handover (VHO), etc—in the provider network.

• Ethernet Virtual Connection (EVC) is the architecture construct that supports the association of UNI reference points for the purpose of delivering an Ethernet flow between subscriber sites across the MAN.

Simple shut-off scripting

I had a client today who is doing some manual things as they are using Quickbooks for billing and such.  One thing they kind of struggle with is turning off people for non-payment and such.  Their current method is adding a que and throttling someone to a low-speed to make them call.  Their network is a routed network utilizing DHCP to the CPE at the customer.  Everything is in router mode and they control the addressing of the units via DHCP reservations.  So how do we make this better without adding radius and all kinds of stuff into the network?

First we set up a web-proxy

/ip proxy
set enabled=yes port=8089

/ip proxy access
add dst-port=80
add dst-host=* dst-port=80
add dst-port=53
add action=deny

What the above code does is says anyone coming into the proxy is only allowed to go to (used our domain as an example), use port 53 (DNS), and anything else gets redirected to We chose port 53 because they are in the process of cleaning up some of the radios and such which are using and other DNS servers.

Next we set up a nat rule

/ip firewall nat
add action=redirect chain=dstnat dst-port=80 protocol=tcp src-address-list=\
SHUTOFF to-ports=8089

This nat rule says anyone making a port 80 request coming from our SHUTOFF address-list gets redirected to port 8089 (our proxy port setup earlier).

Our third step is to setup our address list. this is very straightforward.  Just modify and add users to this list when they are to be turned off.

/ip firewall address-list
add address= list=SHUTOFF

Lastly, we add a filter rule which denies the SHUTOFF folks from using anything except port 53 and port 80.  We do this because we can’t proxy port 443 and other SSL traffic. If folks go to a HTTPS site it simply fails.  This is a drawback of using a web-proxy.

/ip firewall filter
add action=drop chain=forward dst-port=!53,80 protocol=tcp src-address-list=\

If you have an SSL payment gateway you can modify your filter rules to allow traffic to it. This is just one quick and dirty way of letting customers know they have been turned off.

Soft Reconfiguration inbound

Several people have been asking what soft Reconfiguration Inbound is on a BGP peer.

In the dark days of BGP you had to tear down the BGP session and do a full reestablishment in order to bring it up.  What soft reconfiguration does is copies of all routes received (this is why it is called inbound) are stored separately from the regular BGP table.   When a change is made the new change is applied to the stored copy of the BGP routes.

Disadvantage? This takes up memory because you have two files basically.

So how is this different than route refresh described in RFC 2918? This is a standard, with an RFC unlike Soft Reconfiguration inbound, which is a Cisco thing. Route refresh asks the peer to resend all its routes.