From the archives – Evolution of a network guy part 4

In 2006 I was hired on part-time for Purdue University. My days would consist of mornings on Campus at Purdue doing I.T. support for the Agronomy Department and afternoons, evenings, and weekends doing support and build-outs for NDWave.  We were in the mode of dumping everything back into the company to get to a solid sustainable position.  I didn’t really want to go back to desktop support, but the benefits and part-time position helped to pay off bills we had accumulated with MTIN.   Purdue allowed me to meet Donnie Payne.  Donnie is an infectious personality sort of person.  Just being around him motivates you to do things.  Purdue allowed me to work with Mac OSX Xserves, cutting-edge Linux servers, and special projects.  One of the coolest projects I worked on included remote sensor trailers.  We had servers, remote reboots, and had to deal with how to get connectivity out in BFE.  Several sites had cellular or satellite uplinks. I was able to apply my ISP knowledge to this project.

NDWave exposed us to lots of rockstars in the ISP industry.  JohnnyO and his crew helping with removing feedhorns from a tower is something I will never forget. Sitting outside a hotel with him cooking dinner.  Guys like Chuck Hogg, who helped the industry in several ways and is just a plain cool cat.  The work alone has opened up friendships which were well worth any long days. Guys like Jay Panozzo, who are not directly connected with the WISP industry, but have their parts.  Jay owns Midamerica Towers and is a Man among men.  Jay sets the bar for the tower professional.

I continued to dedicate time to Purdue and Ndwave until April 2008.  One month before Omnicity took over management of the NDWave network I became a full time employee. Omnicity started out good, but quickly went downhill after a year for me.  With all the lawsuits in place that is about all I will say about that.

After being separated form Omnicity I continued to keep my head above water with steady consulting from companies I had helped over the past couple of years. Kenny Johnson at Mooreland ISP and Scott Reed at NewWays were two of my best customers. I truly understood what it was like to be the one the buck stops with.  When you have a tower outage at 1AM. the customer does not care except they have no Internet.  Early morning climbs to repair Aps in the dead of winter were not uncommon.  That is part of what it takes.  Being a former owner really helped me connect with what these guys are going through.

Things were slow so I was able to re-group some.  I did not mess with much technology except when I had to.  I took a step back and concentrated on the activities I enjoyed.  I stepped up collecting G.I. Joe figures visited more friends, and generally did non-tech related things.  This in itself expanded those I call friends.  My mind and soul needed that healing.  Then a funny thing happened.  I started writing this blog, becoming active on mailing lists again, and generally became interested in the technology again.  I started gaining more and more consulting clients and working with more and more networks.  Ubiquiti was just starting to come on the scene, Mikrotik was a mature platform, and prices were starting to come down on gear.  I remember sitting around 3 years ago thinking I needed to step up what I am doing.  It took me up until then to realize I had enough experience and stuff running around in my head to be an expert. I have watched so-called experts screw up even the simplest things.

In the past couple of years, things have really blossomed in the Industry and I have had the pleasure of being a part of some of it.  When you look back on this there are certain key points where you take leaps and bounds.    Once I was able to step back and further enjoy things that next leap up was taken. This allowed me to open new pathways of thinking too.  My confidence soared, my shyness subsided, and life got better.  I attribute this to those who have influenced my life, and the experiences we have shared.

Stripping wireless gear off a 120 foot tower in Chicago with Mike Hammett is one of my fondest memories.  The work was hard and long, but having the comrade in arms with you to do it make up for it.   There is a bond that is formed which last a lifetime.  Mike is one of those guys you want to see succeed.  I am fortunate to have been in a position on a couple of occasions to help him when it would have been a paid for him to hire it out.

This brings us to present day…

Continued in part 5 (the conclusion I promise)

A story about a rotten company

Recently I received an e-mail from a company I haven’t done business with since 2003. They had kept my e-mail all this time and decided now was the time to send me a spam e-mail.  Let me give you a little backstory on Advanced Internet Technologies Inc.

Back in 2001 I went out on my own as an ISP.  Previously, I had worked for a dial-up ISP and they had been acquired by a larger company.  I saw the writing on the wall and decided now was the time to go out on my own.  With some borrowed money I made my first purchase, a white box 1U server. Keep in mind this is 2001.  This server with a pair of 80 gig hard drives and dual 1GHZ Xeon Pentiums cost me $1800. I had talked to a sales guy from AIT, and liked everything about the company. So I had my little server directly shipped to them and the loaded on Redhat Linux 4.  I was able to cobble my way through setting up sendmail, apache, and some other services and my ISP was up and running in a few weeks.  Things were clicking along for a few years.  We were doing dial-up and had a wholesale agreement with DialUpUSA for nationwide dial-up and ISDN.

Now, here is where it gets good.  One night in 2003 I notice my server load going crazy.  Server load was in the 30’s. Anyone who knows Linux knows this is something bad going on. So as I trying to track down what is going on I lose all connection to my server. No pings, nothing.  So, I call the AIT noc and see if they can look into my server.  I am told they would look at it.   Several hours go by and I am calling and calling trying to get an update.  I had suspected I had been compromised in one way or another and my server was doing bad things. 10PM turns into 10AM.  I call my sales guy at AIT basically pleading for him to find out what is going on.  He answers my phone call once, but subsequent phone calls go unanswered and no one returns my calls.  Still no answers.  I am calling every 15 minutes trying to get someone, anyone to give me answers.  This goes on for a few days.  By this time my business is suffering, because e-mail is down, including my own. After 4 days of no answers, I am in full-blown panic mode.  Luckily I was using the DIALUPUSA radius and e-mail and the web-site were the things down. In the meantime, I find FDC Servers in Chicago which had space and cheap bandwidth.  I rented a dedicated server from them and was able to get things back up and going after a week of downtime.

Fast forward about a month I receive my server in a poorly packed box with a note saying my server had been compromised and had taken down their entire network because it was sending out junk.  At this time on their web-site they were advertising a capacity of 45 megs to the Internet.  That was big time for 2003. And included in this note was an invoice for $2700 for work they had to do in order to deal with my server causing an outage on their network. The next day my attorney was sending them a very strongly worded letter with phone records on my attempts to contact them and how we would be pursuing legal action for violation of their SLA, which did include turn around time for trouble resolution. Many letters and calls later we never heard from AIT again. We were sending certified letters on a weekly basis.  My father would have been proud the amount of legal paperwork we sent to try and generate a response.

Anyway, So now 14 years later I get a SPAM e-mail from Byron Briggs, Chief Operating Officer of Advanced Internet Technologies Inc. on their dedicated server special. They kept my information after all these years, even after one of our letters told them to purge all of my information from their databases.

 

Dear Byron Briggs,
Your company is one of the lousiest companies I have ever done business with. Your total lack of response almost ruined my company. I was a loyal, and on-time paying customer every month of me being an AIT customer.  I still have the original server in my house as a reminder of how awful a company can be.  I feel sorry for the server for even having to be in a data center ran by such uncaring and callous people.  The poor Linux box suffered enough in its life.  I see on your Linked in you have only been at AIT since 2008. It would be easy to say that was in the past. However, Charles Briggs was there during my time as a customer. I am assuming you are one of his four children he speaks of.  I remember talking to Charles on the phone on several occasions when the company was small.  I referred business and we talked about the future of things. The lack of response after all of that was just the nail in the coffin.

Justin Wilson

If you are considering any type of co-location with ait.com I would recommend sticking your server in a refrigerator or cardboard box with a box fan hooked to the local Starbucks wifi before trusting this company with your business.

Oh, and don’t take my word for it
https://www.bbb.org/myrtle-beach/pages/business-reviews/internet-services/advanced-internet-technologies-in-fayetteville-nc-11001845/reviews-and-complaints?noskin&clean

http://www.vistainter.com/reviews/A/ait.com/

 

Vendors and core business

I had a client learn a lesson they should not have had to this evening.  The client has had several key servers hosted at a small data center for several years now. These were managed servers the data center took care of. Things like new hard drives were the responsibility of the data center so the client rarely paid attention to these machines.  As many of you know a server can spin for years and it is just forgotten about.

Tonight these servers come under a very heavy Denial of Service (DDoS) attack.  Fifteen plus Gigs come to bear at client’s servers for an extended time.  The client is unable to reach the data center NOC, nor do any of his contacts work.   The servers are knocked offline.  4 hours later the client finally receives an e-mail from the data center saying they unplugged the client’s router because it was taking down their (the DC’s) own network.  After asking to have a call from a manager client finds out the DC has restructured and dropped many of their co-location and other hosting services.  Their multiple 10 gig pipes have been reduced to one, and many clients have left.  The manager says they have re-focused their business to focus on things such as OLED screens, and other things totally unrelated to running a data center. The hosting they do have left “pays the bills” so they can have a place to do research.

The client has redundancy so they are not dead in the water.  However, this redundancy was only supposed to be for a short term duration due to costs.  The lesson learned is to keep in contact with your vital members.  Call up your sales person once or twice a year and see how things are going.  Keep in contact with key folks at the company.  If they are on LinkedIn add the company.  If their focus appears to change or they go silent do some leg work to find out what’s going on.