Aligning an 80GHZ link at a mile and other licensed backhauls

Recently we had a teaching moment for a couple of folks who had not had much experience with aligning higher frequency antennas with very tight beamwidths.  This particular day we were aligning 2 foot Siklu 80GHZ antennas.

One of the questions we often get asked is how do you align these? These questions are usually asked by someone who is familiar with aligning 5ghz antennas with a 10 or 20 degree beam which you can eyeball and has tried a microwave shot. They find out it is much harder.  The higher you go in frequency the tighter and smaller the beam is.  Distance also affects how far off you can be.  Think of it as a laser pointer.  If you have ever taken a laser pointer out at night and shone it a long distance you will notice even the slightest movement will cause it to jump inches, even feet.  Keep laser pointer analogy in mind for this next section.

In order to understand alignment, we need to understand lobes on an antenna. An antenna is just a device that focuses radiation in a direction.  In a licensed microwave setup, these antennas focus the radiation in a tighter “beam”.  Let’s go back to our laser pointer analogy.  Some laser pointers project a smaller dot at 10 feet than others.  Same for antennas.   The diagram below shows what is called the main lobe and the side lobe.

The way to get the best signal is to get both dishes locked on to the main lobe. Sounds easy right? With higher frequencies, you are talking about millimeter waves. This means the main lobe may only be 3mm wide, about the size of this text on a laptop screen.  Now imagine trying to keep that 3mm beam in the center of a paper plate at a mile.  On top of that, the difference between the main lobe and locking onto a side lobe could be the difference of 1-2mm. A slight wind can move a dish 2mm.

To give you a real-world example. A 2ft 23 GHz antenna having 3 dB beamwidth of 1.6 degrees. Allowing for a path length of about 2.5 miles (this is licensed 23GHZ) the actual beamwidth at the receiving antenna is around 370 ft and is, therefore, likely to be greater than the height of the tower. If the antenna’s out of horizontal by even a couple of degrees to start, the antennas will miss by around 460 ft and not be able to “see” each other. This can be amplified as frequency and distance increase.

This is all fine and dandy, but what about the practical world? How do I align the thing?
It all starts with the FCC path coordination paperwork you will receive on your licensed link. There is a wealth of information in here.  It tells you all of the following:
-Your mounting height (this is typically already known)
-Your heading (more on this in a bit)
-The antenna angle downtilt or uptilt (very important)
-The expected signal target

Armed with this information you will have all of the information you need to align the link.  From this point, the philosophical side of things kicks in.  Some tower climbers are good with using a compass to get their exact bearings.  Others have high dollar tools to do it all via GPS such as microwave path alignment from Sunsight.

What everyone doing alignment should have in their toolkit are the following:
-A small magnetic bubble Level. We want to make sure we start with a level mount.  We would be fighting an uphill battle if the pipe or standoff we are mounting to is not level.

-An angle Finder is very helpful for determining the antenna down or uptilt per the path calculation.

Obviously, the above tools are just one of many examples.  There are more expensive ones and bare bones ones.  Tools are only as good as the person using them.

-Ratcheting wrenches for the left and right and up and down adjustments.
Having ratcheting wrenches makes fine-tuning a very easy process.  You will see why later.

-A good hands-free communication method.  Depending on the tower FM communications may or may not work.  Cell phones may or may not work. Being able to talk to the crew on the other end is crucial.  And yes, to make this smooth you want a crew on the other end.

Aligning backhauls, especially microwave, is a skilled trade.  With any skilled trade, you will get all kinds of tips and tricks of the trade.  Some you may use, others you may not.  Ask any Carpenter, Drywaller, or Mason and they will tell you little tips and tricks. They probably all are great and will work, but you may only use some of them.  I am going to tell you mine. You may find others you like better.

We always start with a google earth plot of the path. I call this Phase 1.  The goal of phase 1 is to get the radios talking.  We make sure the line is exactly on the two points, not just approximate.  If the backhaul it on the left side of the tower, we draw the line to/from the left side of the tower.  We then pick 2-3 landmarks along the path as we can.  We start with something close to the tower the climber should be able to see.

In our photo above we have picked out two reference points close to the tower the climber can see.  The first is the clump of trees on the climbers left.  The path passes “just to the right” of the edge of the end of the trees.  The second reference is the intersection of the county roads about 2-3 miles out.  Our path should be just to the right of those.  That point of reference is more of a sanity check. More than anything. The climber at the other end has a similar printout.   I have found communication during this process works best if both climbers and someone logged to at least one radio on the ground with a laptop are on a conference bridge.  Many radios have lights, tones, or multimeter outputs to indicate signal.  Some modern radios only have web-interfaces and apps.  Hold a phone while trying to align can be cumbersome.  This is where the guy on the ground can take some load off what the climbers are doing.

Regardless of the mechanics of the radio, the goal of Phase 1 is to establish a radio link, no matter how bad it is. Now, here is where the real meat and potatoes of backhaul alignment come into play.  This is a very deliberate and calculated process.  Your goal at the end of the entire alignment process is to end up with the following diagram

What many folks don’t realize is it is possible to establish a signal on a side lobe. So how do you know if you are on a side lobe? Here is how we start phase 2. This is what I call fine-tuning. Real original huh? Depending on good, or lucky you were during phase 1 you may have a long way to go or a short way to go to meet target.  Remember that in your paperwork we talked about earlier?  One side and one side only starts moving their fine adjustment on their antenna to the left and right and up and down.  This is typically called sweeping.  The key thing to note here is you need to find the very edges of the radio signal, not just the lobe you happen to be on.

Let’s take a real-world example to explain how sweeping affects main and side lobes.  At the start of this article, we mentioned an 80ghz link.  With our phase 1 rough alignment, we were able to get linked at a -86.  The target was a -32.   The first side to start alignment started sweeping to the right, signal started going from a -86 down to a -72 rather quickly. This was using very small turns of the adjustment.  The ratcheting wrench was only clicking 1-2 times for each 2-3 db of signal change. Once it reached a -72 it started climbing back up.   The climber then kept going to the right to find the edge of the signal, not just the lobe we were on.  The signal started getting worse until we were back into the upper 80’s.

Now, the climber brings the alignment back to the left, and stops at the -72 and makes a mental note of where that is in relationship to the overall placement of the dish, etc.  Some mounts have distinct notches, some guys use markers, others just remember.  Now the climber continues on to the left and the -72 gets worse and goes back down to the -86 and continues to get worse.  So the climber, at least for now, has found the sweet spot for the left and right alignment.  The climber also knows this will probably change, but has found it for now.   Climber repeats the same procedure for the up and down. Due to the anglefinder, the climbers have with them they feel pretty confident they are fairly close with the up and down so they do not adjust the up and down travel as much as the procedure goes on.

Next, the other side does the same procedure the first side did. They do the left to right and get the signal down to a -62. Essentially, what the climbers are trying to do is find the center, which will contain the strongest signal, by sweeping past the other signals.  Keep in mind there may be only millimeters separating these other lobes.  Due to physics, and the shape of the signal, the first lobe is actually stronger than the edges of the main beam.

Say what? The first lobe is stronger than the edges of the main beam? Yes, but not stronger than the main beam.  Let’s go back to our installers. They have each had a go around at alignment and are only at a -62.  On a 5ghz backhaul that would be respectable, depending on your noise floor. But we are 30db away from our target of -32. Some climbers, incorrectly I might add, try to do a shortcut by scanning in an x pattern instead of x and y-axis separately. This makes it easier to lock onto a side lobe.

80ghz backhaul

So now our first climber goes back to making the left and right adjustments.   At this point, the installer finds something odd.  He has gotten the signal down to a -55, but that’s the best he can do. Even a small turn jumps the signal up    Then our installer remembers the above statement.  The first lobe is always stronger than the edges of the main beam.  He gets the signal back down to a -55 and turns the alignment over to the other side.

Here is a very important thing to note.  Both of our installers have now “gotten a feel” for the few turns needed to adjust the signal on these dishes.  To them compared to 5ghz dishes, these are very tiny and almost insignificant movements. But they sure make a difference in signal.  Now our installer at tower B has his second alignment session.  As he is making adjustments the signal is not changing.  He is moving his wrench for what seems like forever and the signal is barely moving, Any other time their signal would have been a -90 or dropped.  What has happened here? The main lobe of one side has locked onto the first lobe because it is always stronger.  Since the main lobe is bigger it seems like it takes forever to make any change.  If we had a guy on the laptop he was probably also probably seeing very mismatched data rates.  One side was probably much higher than the other by a large margin.

Then boom, all of a sudden the signal goes from a -55 to a -42.  A 17 db jump!   We can now tell we are on the main lobe.  If the laptop person looks at the data rates now they should be more balanced.

Data Rates on a Mimosa B11 Rates properly aligned but not fine-tuned

At this point, it is just a simple matter of each side making finer and finer adjustments back and forth to get the signal down.  If you think of the above circle/crosshair you are making smaller and smaller adjustments to nudge toward the center of the circle. This is where the ratcheting wrenches help by giving a very measured amount of travel.  This helps with the whole feel of alignment.  Much of it is feel to see how much you can move the adjustment mechanisms to make the numbers move.  Sometimes it may be a single click of the wrench.  Sometimes it may be one or two.  It just depends.  As you get closer and closer to target you are moving the adjustment less and less.

As you get closer and closer to target you need to be thinking about how tightening down the adjustment bolts will affect the alignment.  Even tightening them down snug can affect the signal.  That extra amount movement to tighten them down can move them slightly past their alignment center.  You may need to take into account the amount of travel it takes to tighten down the adjustment bolt into account on smaller dishes.  If it takes a half turn of the bolt to get it tight you may need to stop a half turn and tighten “into” target.  As you tighten it down fully that is where you end up in align.  If you wait until you are in align and then snug it completely down, the force of snugging it down may pull it past and you will end up with a worse signal.

This article sprinkled in some examples from a real-world install, with some theory, with some practical knowledge. Your mileage and experience will vary.  Your experience with 6ghz vs 80ghz will vary as well. Each frequency will have it’s own quirks and tricks.

MTIN is now a FLexOptic Reseller

MTIN typically is not a reseller for many product lines, for several reasons.  We like to be vendor agnostic and not chasing sales commissions on products, and we are not in the business of stocking product.

Having said this, we now have a reseller relationship with flexoptic.net.  They have optics you can code for a huge variety of manufacturers.  WISP clients will be intersted to know they support the following vendors:
-Brocade
-Cisco
-Ceragon
-Mikrotik
-Netgear
-Netonix
-Ubiquiti
and a whole bunch more. There are over 150 vendors supported.

The optics are coded with a product called Flexbox. The flexbox has several features to it such as coding, wavelength tuning of DWDM, distance analyzer, power measurement, and diagnostics.

FLEXBOX series - Configure Universal Transceivers | CSFP, SFP, SFP+, XFP, QSFP+, QSFP28, SFP28, CFP, CFP2, CFP4

We are working on some reviews, how-tos and other tutorials for these products. At the very least we are recommending everyone have a few optics of the form factors you use for compatibility troubleshooting.  If you have a device that you wonder if it is recognizing your optics correctly you can pull out this kit, code an optic for your device, and go on with troubleshooting.   Very handy for vendor optic issues.

If this is something you are interested in send us an e-mail for a quote on a starter kit and look for more information coming soon.

Client subnet in DNS requests

Some Light Reading:
https://tools.ietf.org/html/draft-vandergaast-edns-client-subnet-00

Many Authoritative nameservers today return different replies based
   on the perceived topological location of the user.  These servers use
   the IP address of the incoming query to identify that location.
   Since most queries come from intermediate recursive resolvers, the
   source address is that of the recursive rather than of the query
   originator.

   Traditionally and probably still in the majority of instances,
   recursive resolvers are reasonably close in the topological sense to
   the stub resolvers or forwarders that are the source of queries.  For
   these resolvers, using their own IP address is sufficient for
   authority servers that tailor responses based upon location of the
   querier.

   Increasingly though a class of remote recursive servers has arisen
   that serves query sources without regard to topology.  The motivation
   for a query source to use a remote recursive server varies but is
   usually because of some enhanced experience, such as greater cache
   security or applying policies regarding where users may connect.
   (Although political censorship usually comes to mind here, the same
   actions may be used by a parent when setting controls on where a
   minor may connect.)  When using a remote recursive server, there can
   no longer be any assumption of close proximity between the originator
   and the recursive, leading to less than optimal replies from the
   authority servers.

   A similar situation exists within some ISPs where the recursive
   servers are topologically distant from some edges of the ISP network,
   resulting in less than optimal replies from the authority servers.

   This draft defines an EDNS0 option to convey network information that
   is relevant to the message but not otherwise included in the
   datagram.  This will provide the mechanism to carry sufficient
   network information about the originator for the authority server to
   tailor responses.  It also provides for the authority server to
   indicate the scope of network addresses that the tailored answer is
   intended.  This EDNS0 option is intended for those recursive and
   authority servers that would benefit from the extension and not for
   general purpose deployment.  It is completely optional and can safely
   be ignored by servers that choose not to implement it or enable it.

   This draft also includes guidelines on how to best cache those
   results and provides recommendations on when this protocol extension
   should be used.

For those of you running BIND here is some practical information
https://ftp.isc.org/isc/dnssec-guide/html/dnssec-guide.html#whats-edns0-all-about

Save bandwidth on Apple updates

Like many networks, you have users using Apple devices. iPhones, Ipads, computers, and other Apple devices are constantly updating apps, downloading updates, and other content.  MTIN can install an OSX Caching server on your network. This low powered server caches software updates, allowing faster downloads, especially for new iPhone IOS updates.

Contact MTIN today and learn about our turnkey solutions for making your Apple users happier.

The problem with speedtests

Imagine this scenario. Outside your house, the most awesome super highway has been built.  It has a speed limit of 120 Mile Per Hour.  You calculate at those speeds you can get to and from work 20 minutes earlier. Life is good.  Monday morning comes, you hop in your Nissan GT-R, put on some new leather driving gloves, and crank up some good driving music.  Your pull onto the dedicated on-ramp from your house and are quickly cruising at 120 Miles an hour. You make it into work before most anyone else. Life is good.  

Near the end of the week, you notice more and more of your neighbours and co-workers using this new highway.  Things are still fast, but you can’t get up to speed to work like you could earlier in the week.  As you ponder why you notice you are coming up on the off-ramp to your work.  Traffic is backed up. Everyone is trying to get to the same place.  As you are waiting in the line to get off the super highway, you notice folks passing you by going on down the road at high rates of speed.  You surmise your off-ramp must be congested because it is getting used more now.

Speedtest servers work the same way. A speedtest server is a destination on the information super-highway. Man, there is an oldie term.  To understand how speedtest servers work we need a quick understanding of how the Internet works.   The internet is basically a bunch of virtual cities connected together.  Your local ISP delivers a signal to you via Wireless, Fiber, or some sort of media. When it leaves your house it travels to the ISP’s equipment and is aggregated with your neighbours and sent over faster lines to larger cities. It’s just like a road system. You may get access via a gravel road, which turns into a 2 lane blacktop, which then may turn into a 4 lane highway, and finally a super-highway.  The roads you take depend on where you are going. Your ISP may not have much control over how the traffic flows once it leaves their network.

Bottlenecks can happen anywhere. Anything from fiber optic cuts, oversold capacity, routing issues, and plain old unexpected usage. Why are these important? All of these can affect your speedtest results and can be totally out of control of your ISP and you.  They can also be totally your ISP’s fault. They can also be your fault, just like your car can be.  An underpowered router can be struggling to keep up with your connection. Much like a moped on the above super-highway can’t keep up with a 650 horsepower car to fully utilize the road, your router might not be able to keep up either.  Other things can cause issues such as computer viruses, and low performing components.

Just about any network can become a speedtest.net node or a node with some of the other speedtest sites.  These networks have to meet minimum requirements, but there is no indicator of how utilized these speedtest servers are.  A network could put up one and it’s 100 percent utilized when you go running a speedtest. This doesn’t mean your ISP is slow, just the off-ramp to that speedtest server is slow.

The final thing we want to talk about is the utilization of your internet pipe from your ISP.  This is something most don’t take into consideration.  Let’s go back to our on-ramp analogy.  Your ISP is selling you a connection to the information super-highway.   Say they are selling you a 10 megabyte download connection.  If you have a device in your house streaming an HD Netflix stream, which is typically 5 megs or so, that means you only have 5 megs available for a speedtest while that HD stream is happening. Speedtest only test your current available capacity.  Many folks think a speedtest somehow stops all the traffic on your network, runs the test, and starts the traffic. It doesn’t work that way. A speedtest tests the available capacity at that point in time.  The same is true for any point between you and the speedtest server.  Remember our earlier analogy about slowing down when you got to work because there were so many people trying to get there.  They exceeded the capacity of that destination.  However, that does not mean your connection is necessarily slow because people were zooming past you on their way to less congested destinations.

This is why speedtest results should be taken with a grain of salt. They are a useful tool, but not an absolute. A speedtest server is just a destination.  That destination can have bottlenecks, but others don’t.  Even after this long article, there are many other factors which can affect Internet speed. Things we didn’t touch on like Peering, the technology used, speed limits, and other things can also affect your internet speed to destinations.

Some Random Visio diagram

Below, We have some visio diagrams we have done for customers.

This first design is a customer mesh into a couple of different data centers. We are referring to this as a switch-centric design. This has been talked about in the forums and switch-centric seems like as good as any.

This next design is a netonix switch and a Baicells deployment.

Design for a customer

Use tarpit vs drop for scripts blocking attackers

There are many scripts out there, especially on Mikrotik, which list drop as the action for denying bad guy traffic.  While this isn’t wrong, you could put the tarpit action to better use for actions which are dropping attacking type of traffic.

So what is Tarpit?
Tarpit is fairly simple. When connections come in and are “tarpitted” they don’t go back out. The connection is accepted, but when data transfer begins to happen, the TCP window size is set to zero.  This means no data can be transferred during the session.  The session is held open, and requests from the sender (aka attacker) to close the session are ignored. They must wait for the connection to timeout.

So what’s the downside?
TCP is not really designed to hold onto a connection.  It can be additional overhead on a taxed system.  Most modern firewalls can handle tarpitting without an issue. However, if you get thousands of connections it can overwhelm a system or a particular protocol.

How can I use it?
If you have scripts, such as the SSH drop off the Mikrotik wiki, simply change the action to “tarpit” instead of “drop”.

DMCA Designated Agent Directory updates

The following text is directly from: https://www.copyright.gov/dmca-directory/ 

A relevant F.A.Q. can be found at https://www.copyright.gov/dmca-directory/faq.html

Service Provider Designation of Agent to Receive Notifications of Claimed Infringement

The Digital Millennium Copyright Act (“DMCA”) provides safe harbors from copyright infringement liability for online service providers. In order to qualify for safe harbor protection, certain kinds of service providers—for example, those that allow users to post or store material on their systems, and search engines, directories, and other information location tools— must designate an agent to receive notifications of claimed copyright infringement. To designate an agent, a service provider must do two things: (1) make certain contact information for the agent available to the public on its website; and (2) provide the same information to the Copyright Office, which maintains a centralized online directory of designated agent contact information for public use. The service provider must also ensure that this information is up to date.

In December 2016, the Office introduced an online registration system and electronically generated directory to replace the Office’s old paper-based system and directory. Accordingly, the Office no longer accepts paper designations. To designate an agent, a service provider must register with and use the Office’s online system.

Transition period: Any service provider that has designated an agent with the Office prior to December 1, 2016, in order to maintain an active designation with the Office, must submit a new designation electronically using the online registration system by December 31, 2017. Any designation not made through the online registration system will expire and become invalid after December 31, 2017. Until then, the Copyright Office will maintain two directories of designated agents: the directory consisting of paper designations made pursuant to the Office’s prior interim regulations which were in effect between November 3, 1998 and November 30, 2016 (the “old directory”), and the directory consisting of designations made electronically through the online registration system (the “new directory”). During the transition period, a compliant designation in either the old directory or the new directory will satisfy the service provider’s obligation under section 512(c)(2) to designate an agent with the Copyright Office. During the transition period, to search for a service provider’s most up-to-date designation, begin by using the new directory. The old directory should only be consulted if a service provider has not yet designated an agent in the new directory.

Where does Trill and VXLAN fit in your strategy?

As networking trends yo-yo between layer-3 and layer-2,  different protocols have emerged to address issues with large layer-2 networks. Protocols such as Transparent Interconnection of Lots of Links (TRILL), Shortest Path Bridging (SPB), and Virtual Extensible LAN (VXLAN) have emerged to address the need for scalability at Layer2.   Cloud scalability, spanning tree bridging issues, and big broadcast networks start to become a problem in a large data center or cloud environment.

To figure out if things like TRILL is a solution for you, you must understand the problem that is being addressed by TRILL. The same goes for the rest of the mentioned protocols. When it boils down to it the reason for looking at such protocols is you want high switching capacity, low latency, and redundancy.  The current de facto standard of Spanning Tree Protocol (STP) simply is unable to meet the needs of modern layer2 networks.  TRILL addresses the problem of STP’s ability to only allow one network path between switches or ports.  STP prevents loops by managing active layer -2 paths.   TRILL applies Intermediate System-to-Intermediate System protocol (IS-IS), which is a layer3 routing protocol translated to Layer 2 devices.

For those who say TRILL is not the answer things like SPB also known as 802.1aq, and VXLAN are the alternatives. A presentation at NANOG 50 in 2010 addressed some of the SPB vs TRILL debate. This presentation goes into great detail on the differences between the two.

The problem, which is one most folks overlook, is that you can only make a layer 2 network so flat.  The trend for a while, especially in data centers, is to flatten out the network. Is TRILL better? Is SPB better? The problem isn’t what is the better solution to use.  What needs to be addressed is the design philosophy behind why you need to use such things.   Having large Layer2 networks is generally a bad idea. Scaling issues can almost always be solved by Layer-3.

So, and this is where the philosophy starts, is TRILL, SPB, or even VXLAN for you? Yes, but with a very big asterisk. TRILL is one of those stop-gap measures or one of those targeted things to use in specific instances. TRILL reduces complexity and makes layer-2 more robust when compared to MLAG. Where would you use such things? One common decision of whether to use TRILL or not comes in a virtualized environment such as VSPHERE.

Many vendors such as Juniper, have developed their own solutions to such things.  Juniper and their Virtual Chassis solution do away with spanning tree issues, which is what TRILL addresses.   Cisco has FabricPath, which is Cisco’s proprietary TRILL-based solution. Keep in mind, this is still TRILL.   If you want to learn some more about Fabric Path this article by Joel Knight gets to the heart of Fabric path.

Many networks see VXLAN as their upgrade path.  VXLAN allows layer 2 to be stretched across layer 3 boundaries. If you are a “Microsoft person” you probably hear an awful lot about Network Virtualization using Generic Routing Encapsulation (NVGRE) which can encapsulate a layer two frame into IP.

The last thing to consider in this entire debate is how does Software Defined Networking (SDN) play into this. Many folks think controllers will make ECMP and MLAG easy to create and maintain. If centralized controllers have a complete view of the network there is no longer a need to run protocols such as TRILL.   The individual switch no longer makes the decision, the controller does.

Should you use Trill, VXLAN, or any of the others mentioned? If you have a large Layer-2 virtualized environment it might be something to consider.  Are you an ISP, there is a very small case for running TRILL in anything other than your data center. Things such as Carrier Ethernet and MPLS are the way to go.