My Home Lab/Testing ground

A few days ago, my buddy, Greg Sowell posted his Mobile Home Lab. I figured I would show off the rack in my home office.

This is a mixture of gear that powers the basic network for the network in my home and for testing, blog posts, support, and videos\. Each floor of our 3 story home currently has a Unifi Access point on it powered by a toughswitch POE.  My top level, which is where my office is has a unifi pro that does both 2.4 and 5GHZ.  The other levels just do 2.4ghz.  This will change once I get a POE switch that does 48volt to power the UNIFI pro.  I have stuck with UNIFI because of the bar in our house.  Any self-respecting geek needs a guest wifi network.  WPA keys are too hard to dish out for those late arriving guests after some rounds of crown and coke.  So a Cloudkey makes guest access an easy venture.

As stated before the UNIFIs are powered by a Toughswitch, and the PRO has a 48VOLT POE and is linked into a port on the tough switch.  This switch is then uplinked into one of the gig links of the active 2950 switch.  Various other devices, some not plugged in at the moment due to need to get to a cubby hole for a roof project, are plugged into the 100 meg ports on this 2950.  Things such as the DVR for the security system, network printers, ethernet to my desk for testing, network drives, etc.  The other gig port is uplinked to our internet router.

Our internet is handled by a workhorse Mikrotik 493AH. This has a Comcast cable and a local WISP connection, which is a backup.  From this router, I am initiating several VPN, EOIP, and other tunnels to various clients and remote networks.  If you notice, this router also has a little rubber duck antenna.  Inside is a r52 card that is usually disabled by default.  This is a backup network for testing if I suspect an issue on the internal wireless network. I can log in, enable the card, and associate to the SSID and see if things are okay, at least as okay for 802.11b/g speeds.

 

Most everything else is for Cisco certification testing and keeping up on those certs as well as labbing up scenarios. As you guys will hear on our latest podcast, GNS3 and packet tracer are great, but sometimes you can’t beat actual hardware.

I too have a console server for turning my devices on and off. I do not have fancy remote access turned on, but I can remote to 6 devices at a time without getting up and moving the 4 feet to move a cable.  Welcome to the future!

Run down of some equipment
Cisco 2950 (one production and one lab)
2x Cisco 3750
Cisco 3640
Cisco 3560-X
Cisco 1841
Various Mikrotik routers
Ubiquiti EdgeRouter Pro
Ubiquiti EdgeSwitch 16
(The infinity is going into production soon at a data center)

The Cisco 2541 at the top is a shelf for the monitor for the DVR.  Make a great shelf!  In the future, I hope to add a Juniper router and some more gear.  As always, if you are a manufacturer I would be glad to review some of your gear and even do some configuration videos on it.

On a side notes, you don’t see much wireless gear.  That is a separate spot in my office.

MTIN is now a FLexOptic Reseller

MTIN typically is not a reseller for many product lines, for several reasons.  We like to be vendor agnostic and not chasing sales commissions on products, and we are not in the business of stocking product.

Having said this, we now have a reseller relationship with flexoptic.net.  They have optics you can code for a huge variety of manufacturers.  WISP clients will be intersted to know they support the following vendors:
-Brocade
-Cisco
-Ceragon
-Mikrotik
-Netgear
-Netonix
-Ubiquiti
and a whole bunch more. There are over 150 vendors supported.

The optics are coded with a product called Flexbox. The flexbox has several features to it such as coding, wavelength tuning of DWDM, distance analyzer, power measurement, and diagnostics.

FLEXBOX series - Configure Universal Transceivers | CSFP, SFP, SFP+, XFP, QSFP+, QSFP28, SFP28, CFP, CFP2, CFP4

We are working on some reviews, how-tos and other tutorials for these products. At the very least we are recommending everyone have a few optics of the form factors you use for compatibility troubleshooting.  If you have a device that you wonder if it is recognizing your optics correctly you can pull out this kit, code an optic for your device, and go on with troubleshooting.   Very handy for vendor optic issues.

If this is something you are interested in send us an e-mail for a quote on a starter kit and look for more information coming soon.

Interesting Mikrotik GUI behavior

While bringing up a BGP session for a client I kept trying to add our side of a /126.  It kept reverting to the network address.  The video shows what happens when I tried to add ::12/126 to the IPV6 addresses.

After some second-guessing and then some Facebook chatting I decided to do a terminal /ipv6 address print.  Sure enough the proper IP shows up.  Must be a GUI bug.

Vulnerability in WPA2

https://arstechnica.com/information-technology/2017/10/severe-flaw-in-wpa2-protocol-leaves-wi-fi-traffic-open-to-eavesdropping/

An air of unease set into the security circles on Sunday as they prepared for the disclosure of high-severity vulnerabilities in the Wi-Fi Protected Access II protocol that make it possible for attackers to eavesdrop Wi-Fi traffic passing between computers and access points.

The proof-of-concept exploit is called KRACK, short for Key Reinstallation Attacks. The research has been a closely guarded secret for weeks ahead of a coordinated disclosure that’s scheduled for 8am Monday, East Coast time. An advisory the US CERT recently distributed to about 100 organizations described the research this way:

US-CERT has become aware of several key management vulnerabilities in the 4-way handshake of the Wi-Fi Protected Access II (WPA2) security protocol. The impact of exploiting these vulnerabilities includes decryption, packet replay, TCP connection hijacking, HTTP content injection, and others. Note that as protocol-level issues, most or all correct implementations of the standard will be affected. The CERT/CC and the reporting researcher KU Leuven, will be publicly disclosing these vulnerabilities on 16 October 2017.

 

From Mikrotik:

On October 16. CERT/CC/ICASI released a public announcement about discovered vulnerabilities in WPA2 handshake protocols that affect most WiFi users and all vendors world wide.
RouterOS v6.39.3, v6.40.4, v6.41rc are not affected!
It is important to note that the vulnerability is discovered in the protocol itself, so even a correct implementation is affected.
These organizations did contact us earlier, so we have already released fixed versions that address the outlined issues. Not all of the discovered vulnerabilities directly impact RouterOS users, or even apply to RouterOS, but we did follow all recommendations and improved the key exchange process according to the guidelines we received from the organizations who discovered the issue.
We released fixed versions last week, so if you upgrade your devices routinely, no further action is required.
CWE-323
CVE-2017-13077
CVE-2017-13078
CVE-2017-13079
CVE-2017-13080
CVE-2017-13081
CVE-2017-13082
CVE-2017-13083
CVE-2017-13084
CVE-2017-13085
CVE-2017-13086
CVE-2017-13087

 

Use tarpit vs drop for scripts blocking attackers

There are many scripts out there, especially on Mikrotik, which list drop as the action for denying bad guy traffic.  While this isn’t wrong, you could put the tarpit action to better use for actions which are dropping attacking type of traffic.

So what is Tarpit?
Tarpit is fairly simple. When connections come in and are “tarpitted” they don’t go back out. The connection is accepted, but when data transfer begins to happen, the TCP window size is set to zero.  This means no data can be transferred during the session.  The session is held open, and requests from the sender (aka attacker) to close the session are ignored. They must wait for the connection to timeout.

So what’s the downside?
TCP is not really designed to hold onto a connection.  It can be additional overhead on a taxed system.  Most modern firewalls can handle tarpitting without an issue. However, if you get thousands of connections it can overwhelm a system or a particular protocol.

How can I use it?
If you have scripts, such as the SSH drop off the Mikrotik wiki, simply change the action to “tarpit” instead of “drop”.

Learning, certifications and the xISP

One of the most asked questions which comes up in the xISP world is “How do I learn this stuff?”.   Depending on who you ask this could be a lengthy answer or a simple one sentence answer.  Before we answer the question, let’s dive into why the answer is complicated.

In many enterprise environments, there is usually pretty standard deployment of networking hardware.  Typically this is from a certain vendor.  There are many factors involved. in why this is.  The first is total Cost of Ownership (TCO).  It almost always costs less to support one product than to support multiples.  Things like staff training are usually a big factor.  If you are running Cisco it’s cheaper to train and keep updated on just Cisco rather than Cisco and another vendor.

Another factor involved is economies of scale.  Buying all your gear from a certain vendor allows you to leverage buying power. Quantity discounts in other words.  You can commit to buying product over time or all at once.

So, to answer this question in simple terms.  If your network runs Mikrotik, go to a Mikrotik training course.  If you run Ubiquiti go to a Ubiquiti training class.

Now that the simple question has been answered, let’s move on to the complicated, and typically the real world answer and scenario.  Many of our xISP clients have gear from several vendors deployed.  They may have several different kinds of Wireless systems, a switch solution, a router solution, and different pieces in-between.  So where does a person start?

We recommend the following path. You can tweak this a little based on your learning style, skill level, and the gear you want to learn.

1.Start with the Cisco Certified Network Associate (CCNA) certification in Routing and Switching (R&S).  There are a ton of ways to study for this certification.   There are Bootcamps (not a huge fan of these for learning), iPhone and Android Apps (again these are more focused on getting the cert), online, books, and even youtube videos. Through the process of studying for this certification, you will learn many things which will carry over to any vendor.  Things like subnetting, differences between broadcast and collision domains, and even some IPV6 in the newest tracks.  During the course of studying you will learn, and then reinforce that through practice tests and such.  Don’t necessarily focus on the goal of passing the test, focus on the content of the material.  I used to work with a guy who went into every test with the goal of passing at 100%.  This meant he had to know the material. CompTIA is a side path to the Cisco CCNA.  For reasons explained later, COMPTIA Network+ doesn’t necessarily work into my plan, especially when it comes to #3. I would recommend COMPTIA if you have never taken a certification test before.

2.Once you have the CCNA under your belt, take a course in a vendor you will be working the most with.  At the end of this article, I am going to add links to some of the popular vendor certifications and then 3rd party folks who teach classes. One of the advantages of a 3rd party teacher is they are able to apply this to your real world needs. If you are running Mikrotik, take a class in that. Let the certification be a by-product of that class.

3.Once you have completed #1 and #2 under your belt go back to Cisco for their Cisco Certifed Design Associate (CCDA). This is a very crucial step those on a learning path overlook.  Think of your networking knowledge as your end goal is to be able to build a house.  Steps one and two have given you general knowledge, you can now use tools, do some basic configuration.  But you can’t build a house without knowing what is involved in designing foundations,  what materials you need to use, how to compact the soil, etc.  Network design is no different. These are not things you can read in a manual on how to use the tool.  They also are not tool specific.   Some of the things in the Cisco CCDA will be specific to Cisco, but overall it is a general learning track.  Just follow my philosophy in relationship to #1. Focus on the material.

Once you have all of this under your belt look into pulling in pieces of other knowledge. Understanding what is going on is a key to your success.  If you understand what goes on with an IP packet, learning tools like Wireshark will be easier.  As you progress let things grow organically from this point.  Adding equipment in from a Vendor? Update your knowledge or press the new vendor for training options.  Branch out into some other areas ,such as security, to add to your overall understanding.

Never stop learning! Visit our online store for links to recommend books and products.

WISP Based Traning Folks.
These companies and individuals provide WISP based training. Some of it is vendor focused. Some are not.  My advice is to ask questions. See if they are a fit for what your goals are.
-Connectivity Engineer
Butch Evans
Dennis Burgess
Rickey Frey
Steve Discher
Baltic Networks

Vendor Certification Pages
Ubiquiti
Mikrotik
Cisco
Juniper
CWNA
CompTIA

If you provide training let me know and I will add you to this list.

Simple shut-off scripting

I had a client today who is doing some manual things as they are using Quickbooks for billing and such.  One thing they kind of struggle with is turning off people for non-payment and such.  Their current method is adding a que and throttling someone to a low-speed to make them call.  Their network is a routed network utilizing DHCP to the CPE at the customer.  Everything is in router mode and they control the addressing of the units via DHCP reservations.  So how do we make this better without adding radius and all kinds of stuff into the network?

First we set up a web-proxy

/ip proxy
set enabled=yes port=8089

/ip proxy access
add dst-host=mtin.net dst-port=80
add dst-host=*.mtin.net dst-port=80
add dst-port=53
add action=deny redirect-to=www.mtin.net

What the above code does is says anyone coming into the proxy is only allowed to go to mtin.net (used our domain as an example), use port 53 (DNS), and anything else gets redirected to www.mtin.net. We chose port 53 because they are in the process of cleaning up some of the radios and such which are using 8.8.8.8 and other DNS servers.

Next we set up a nat rule

/ip firewall nat
add action=redirect chain=dstnat dst-port=80 protocol=tcp src-address-list=\
SHUTOFF to-ports=8089

This nat rule says anyone making a port 80 request coming from our SHUTOFF address-list gets redirected to port 8089 (our proxy port setup earlier).

Our third step is to setup our address list. this is very straightforward.  Just modify and add users to this list when they are to be turned off.

/ip firewall address-list
add address=10.20.0.192 list=SHUTOFF

Lastly, we add a filter rule which denies the SHUTOFF folks from using anything except port 53 and port 80.  We do this because we can’t proxy port 443 and other SSL traffic. If folks go to a HTTPS site it simply fails.  This is a drawback of using a web-proxy.

/ip firewall filter
add action=drop chain=forward dst-port=!53,80 protocol=tcp src-address-list=\
SHUTOFF

If you have an SSL payment gateway you can modify your filter rules to allow traffic to it. This is just one quick and dirty way of letting customers know they have been turned off.

Mikrotik Router OS 6.36.2

To upgrade, click “Check for updates” at /system package in your RouterOS configuration interface, or head to our download page: http://www.mikrotik.com/download

v6.36.2 forum topic discussion,
http://forum.mikrotik.com/viewtopic.php?f=21&t=111450

What’s new in 6.36.2 (2016-Aug-22 12:54):

*) arm – show cpu frequency under resources menu;
*) capsman – fixed upgrade policy;
*) ccr/crs – fixed SFP+ interface ddmi info reporting function. Info is now refreshed on regular intervals;
*) conntrack – fixed ipv6 timeout display;
*) conntrack – fixed removing icmpv6 connections;
*) dns – avoid unnecessary dynamic server address saving in storage;
*) dns – allow to set query-server-timeout and query-total-timeout only greater than 0s;
*) dns – fixed lockup when dynamic dns server address 0.0.0.0 was received;
*) export – updated default values in /system routerboard settings menu;
*) partitions – fixed crash on repartition when there is not enough free space;
*) sstp – fixed disconnects on transmit for multicore systems;
*) switch – fixed configuration reload on CRS switches;
*) winbox – make queue tree default queue type default-small;

Mikrotik RouterOS 3.36

Lots of things fixed in this release.

What’s new in 6.36 (2016-Jul-20 14:09):

*) arm – added Dude server support;
*) dude – (changes discussed here: http://forum.mikrotik.com/viewtopic.php?f=8&t=110428);
*) dude – server package is now made smaller. client side content upgrade is now removed from it and is downloaded straight from our cloud. So workstations on which client is used will require access to wan. Alternatively upgrade must be done by reinstalling the client on each new release;
*) firewall – added “/interface list” menu which allows to create list of interfaces which can be used as in/out-interface-list matcher in firewall and use as a filter in traffic-flow;
*) firewall – added pre-connection tracking filter – “raw” table, that allow to protect connection-tracking from unnecessary traffic;
*) firewall – allow to add domain name to address-lists (dynamic entries for resolved addresses will be added to specified list);
*) wireless – wireless-fp is discontinued, it needs to be uninstalled/disabled before upgrade;
*) address – allow multiple equal ip addresses to be added if neither or only one is enabled;
*) address-list – make “dynamic=yes” as read-only option;
*) arm – fixed kernel failure on low memory;
*) arp – added arp-timeout option per interface;
*) bonding – fixed 802.3ad load balancing mode over tunnels ;
*) bonding – fixed bonding primary slave assignment for ovpn interfaces after startup;
*) bonding – fixed crash on RoMON traffic transmit;
*) bonding – implemented l2mtu value == smallest slave interfaces l2mtu;
*) capsman – fixed crash when running over ovpn;
*) certificate – added automatic scep renewal delay after startup to avoid all requests accessing CA at the same time;
*) certificate – cancel pending renew when certificate becomes valid after date change;
*) certificate – display issuer and subject on check failure;
*) certificate – do not exit after card-verify;
*) certificate – force scep renewal on system clock updates;
*) chr – fixed CHR seeing its own system disk mounted as additional data disk;
*) clock – fixed time keeping for SXT ac, 911L, cAP, mAP lite, wAP;
*) clock – save current time to configuration once per day even if there are no time zone adjustments pending;
*) cloud – fixed export order;
*) console – fixed get false function;
*) console – show message time in echo log messages;
*) defconf – changed channel extension to 20/40/80mhz for all ac boards;
*) dhcp-pd – correct server listing for commands;
*) dhcp-server – fixed radius framed route addition after reboot on client renew;
*) dhcpv6-client – fixed ia lifetime validation when it is set by dhcpv6 client;
*) dhcpv6-relay – set packet link-address only when it is manually configured;
*) dhcpv6-server – fixed binding last-seen update;
*) disk – added support for Plextor PX-G128M6e(A) SSD on CCR1072;
*) email – fixed send from winbox;
*) email – removed subject and body length limit;
*) ethernet – fixed incorrect ether1 link speed after reboot on rb4xx series routers;
*) ethernet – fixed memory leak when setting interface without changing configuration;
*) fastpath – fixed kernel failure when fastpath handles packet with multicast dst-address;
*) fetch – support tls host name extension;
*) firewall – added udplite, dccp, sctp connection tracking helpers;
*) firewall – do not show disabled=no in export;
*) firewall – fixed spelling in built-in firewall commentary;
*) gps – fixed longitude seconds part;
*) health – fixed broken factory voltage calibration data for some hAP ac boards;
*) health – fixed incorrect voltage after reboot on RB2011UAS;
*) icmp – fixed kernel failure when icmp packet could not be processed on high load;
*) ippool6 – fixed crash on acquire when prefix length is equal with pool prefix length;
*) ipsec – add dead ph2 detection exception for windows msgid noncompliance with rfc;
*) ipsec – added dead ph2 reply detection;
*) ipsec – don’t register temporary ph2 on dead list;
*) ipsec – fix initiator modecfg dynamic dns;
*) ipsec – fixed AH with SHA2;
*) ipsec – fixed checks before accessing ph1 nat options;
*) ipsec – fixed mode-config export;
*) ipsec – fixed route cache overflow when using ipsec with route cache disabled;
*) ipsec – fixed windows msgid check on x86 devices;
*) ipsec – show remote peer address in error messages when possible;
*) ipsec – store udp encapsulation type in proposal;
*) kernel – fixed possible kernel deadlock when Sierra USB mode is being used;
*) l2tp – fixed crash when rebooting or disabling l2tp while there are still active connections;
*) lcd – reduced lowest backlight-timeout value from 5m to 30s;
*) license – do not expire demo license right after fresh installation of x86;
*) log – added whole scep certificate chain print;
*) log – increase excessive multicast/broadcast warning threshold every time it is logged;
*) log – make logging process less aggressive on startup;
*) lte – added allow-roaming option for Huawei MU709, ME909s devices;
*) lte – added cinterion pls8 support;
*) lte – added support for Huawei E3531;
*) lte – added support for ZTE ZM8620;
*) lte – added use-peer-dns option (will work only combined with add-default-route);
*) lte – changed driver loading for class 2 usb rndis devices;
*) lte – display message in lte,error log if no response received;
*) lte – display message in lte,error log when PIN is required;
*) lte – fix crash on SXT LTE while resetting card while at high traffic;
*) lte – fixed access technology logging;
*) lte – fixed connection for Huawei without cell info;
*) lte – fixed modem init when pin request present;
*) lte – fixed modem network configuration version checks;
*) lte – fixed network-mode support after downgrade;
*) lte – Huawei MU609 must use latest firmware to work correctly;
*) lte – improved multiple same model modems identification;
*) lte – show uicc for Huawei modems;
*) lte – use only creg result codes as network status indications;
*) mesh – fixed crash when connection references a mesh network but it is not available any more;
*) modem – added support for Alcatel OneTouch X600;
*) modem – added support for Quectel EC21 and EC25;
*) modem – added support for SpeedUP SU-900U modem;
*) nand – improved nand refresh feature to enhance stored data integrity;
*) ovpn – enable perfect forwarding secrecy support by default;
*) ovpn – fixed compatibility with OpenVPN 2.3.11;
*) pppoe – allow to set MTU and MRU higher than 1500 for PPPoE;
*) pppoe – do not allow to send out bigger packets than l2mtu if mrru is provided;
*) proxy – limit max ram usage to 80% for tile and x86 devices;
*) queue – reset queue type on interfaces which default queue type changes to no-queue after upgrade;
*) rb2011 – fixed ether6-ether10 flapping when two ports from both switch chips are in the same bridge;
*) rb3011 – fixed port flapping on ether6-ether10;
*) rb3011 – fixed reset button functionality;
*) rb3011 – fixed usb driver load;
*) rb3011 – fixed usb storage mounting;
*) rb3011 – improved performance on high cpu usage;
*) route – added suppport for more than 8 bits of options;
*) route – fixed ospf by handling ipv6 encoded prefixes with stray bits;
*) sniffer – fixed ipv6 address matching;
*) snmp – fixed get function for snmp>=v2 when oid does not exist;
*) snmp – fixed interface stats branch from MikroTik MIB;
*) snmp – report current access technology and cell id for lte modems;
*) snmp – report ram memory as ram instead of other;
*) ssh – add rsa host key size parameter;
*) ssh-keygen – add rsa key size parameter;
*) ssl – do not exit while there still are active sessions;
*) ssl – fixed memory leak on ssl connect/disconnect (fetch, ovpn, etc.);
*) sstp – fixed dns name support in connect-to field if http-proxy is specified;
*) supout – erase panic data properly on Netinstall;
*) switch – fixed switch compact export;
*) timezone – updated timezone information from tzdata2016e release;
*) traffic-flow – added ipfix support (RFC5101 and RFC5102);
*) tunnel – added option to auto detect tunnel local-address;
*) tunnel – fixed rare crash by specifying minimal header length immediately at tunnel initialization;
*) upnp – fixed nat rule dst-port by making it visible again;
*) usb – I-tec U3GLAN3HUB usb hub/ethernet dongle now shows up correctly as ethernet interface;
*) usb – implement possibility to recognize usb hubs/ethernet-dongles (if usb hubs/ethernet-dongles are not recognized with this version – send supout.rif file);
*) userman – fixed crash on database upload;
*) userman – use ipnpb.paypal.com for payment verification;
*) wap-ac – fixed performance problems with 2.4GHz wireless (additional reboot after upgrade required);
*) webfig – do not allow to press OK or Apply if current configuration values are not loaded yet;
*) webfig – reduced refresh time for wireless registration table to 1 second;
*) winbox – added 2ghz-g/n band for wireless-rep;
*) winbox – added icons to bridge filter actions similar to ip firewall;
*) winbox – added support for ipv6 dhcp relay;
*) winbox – allow to reorder hotspot walled-garden & walled-garden-ip rules;
*) winbox – do not allow to specify vlan-mode=no-tag in capsman datapath config;
*) winbox – do not show filter for combined fields like bgp-vpn4 RD;
*) winbox – do not show mode setting for WDS interfaces;
*) winbox – fixed crash on disconnect in secure mode;
*) winbox – fixed crash when using ctrl+d;
*) winbox – fixed safe mode;
*) winbox – improve filtering on list fields;
*) winbox – report correctly dude users in active users list;
*) winbox – set default sa-learning value to “yes” for CRS Ingress VLAN Translation rules;
*) winbox – show action column as first in bridge firewall;
*) winbox – show error when telnet is not allowed because of permissions;
*) wireless – fixed multiple wireless packages enabled at the same time after upgrade;
*) wireless-rep – added initial API support for snooper;
*) wireless-rep – fixed crash on nv2 reconnect;
*) wireless-rep – fixed scan-list unset;
*) wireless-rep – treat missing SSID element as hidden SSID;