Jay Little - Software Obsessionist
logo
[Quote]
Boston State House is the hub of the Solar System. ¬†You couldn't pry that out of a Boston man if you had the tire of all creation straightened out for a crowbar.
 -- O. W. Holmes

08/05/2020 13:18:34

Over the last few months, I have been having trouble with the stability of my home internet connection. This became evident to me while live streaming to Twitch as my stream would begin dropping a lot of frames in a really short amount of time. For a time, recycling the power on the cable modem would make a positive difference, but after a few months that stopped working reliably. This post is about my subsequent experience trying to write a scheduled cron job that would let me know whenever my internet connection was having issues.

A few years ago I had a similar issue. That took six months to resolve and it ended up requiring that some cable run underneath the sidewalk in front of my house be replaced. So because of the PTSD associated with that hellish experience (which effectively rendered my connection unusable for hours on end at random times, making working remotely from home quite difficult), I took immediate action when I was presented with a similar issue once again.

Once recycling the power on the modem stopped working, I called Charter / Spectrum. They sent a tech out and he replaced the line run from the street, through my yard, into my crawlspace which goes through the floor into my office where the Cable Modem resides. The tech also replaced my Cable Modem (which incidentally was not only an older model, but also "survived" our house getting struck by lightning in 2019).

However that wasn't enough. I wanted to be able to monitor this situation so I wouldn't be caught with my pants down again. I decided to write a simple cron job which would notify me when my internet connection was not working. So how do I did that? Well I decided that pinging some known IP addresses was the way to go. Initially the script was written to simply ping Google's public DNS server @ 8.8.8.8 one hundred times and only produce output in the event that at least one packet was dropped.

Eventually I updated the script so that it would randomly select an IP from a list of IPs, all of which are IPs of publicly accessible DNS servers. Below is the final version of the script in its entirety:

#!/bin/bash
IPS[0]=9.9.9.9
IPS[1]=208.67.222.222
IPS[2]=208.67.220.220
IPS[3]=1.1.1.1
IPS[4]=1.0.0.1
IPS[5]=205.210.42.205
IPS[6]=64.68.200.200
IPS_SIZE=${#IPS[@]}
IPS_INDEX=$(($RANDOM % $IPS_SIZE))
IP=${IPS[$IPS_INDEX]}

COUNT=100

rm /tmp/ping_test_output.txt > /dev/null 2>&1
ping -c $COUNT -D -O $IP > /tmp/ping_test_output.txt 2>&1
if grep -qi " 0% packet loss" /tmp/ping_test_output.txt; then
 exit 0
else
 echo ALERT: Ping Test to $IP lost at least one packet!
 cat /tmp/ping_test_output.txt | tail -n 2 | head -n 1
 echo -----------------------------------------------------
 cat /tmp/ping_test_output.txt
 echo -----------------------------------------------------
 exit 1
fi

This script ran every half an hour via the magic of cron and in the event any packets were dropped produces an email with the pertinent details. So now I had a great way to track when my internet connection was acting up, right?

Wrong.

As I soon came to realize, even though this script seemed to do its job by sending emails when packets were lost, it didn't actually mean my internet connection was having a problem at all. Nevertheless since I didn't realize this I started calling Spectrum each and every time I started getting more than one or two of these emails within a period of a few hours (as I am willing to tolerate maybe one or two a day, because this is the internet after all). At some point it became clear that unlike before, I wasn't seeing other things fail. My streams were not being negatively effected despite receiving these emails and most importantly Spectrum's phone support wasn't able to replicate the packet loss on their end (which they were able to do before).

At this point, I thought maybe that some other piece of network equipment on my side was responsible. My network is largely powered by Ubiquiti equipment and ever since the Coronavirus became an issue, their firmware updates have been less than stellar. I had already rolled back one upgrade on my Wireless Access Point to resolve a separate issue, so I decided to roll back the most recent update on my EdgeRouter X-1 as well.

Initially this seemed to resolve my issues. I went a few weeks and only got an email here and there and all seemed well. Then sometime last week I started getting A LOT of these emails. Every half an hour in fact. Not only that but the packet loss started creeping up from 1% to as high as 3% and 4%. But everything else appeared to be working fine. I was still live streaming without dropping any frames. Internet video services were working fine (exempting of course the occasional glitch from Philo which is just standard operating procedure sadly).

So what the hell was going on? Sadly my entire approach was absolutely idiotic and had never had a real chance of working. However because networking isn't my primary forte and is probably not yours, you are probably scratching your head just like I was wondering what the flaw was. Remember the point of this script is to let me know that my internet connection is having issues, but so far the output produced by this script didn't seem to correlate with any actual issues. So I started to do some research. Eventually I stumbled on this conversation which helped to shed some light on the problem. The subject of the thread really says it all:

DNS Rate Limiting ICMP (8.8.8.8 and 8.8.4.4)

Holy shit. Wow. It never even occurred to me that a public DNS server like 8.8.8.8 would put rate limits on things like ICMP. But when you think about it, it makes perfect sense. These servers exist primarily to service DNS clients, not allow half ass network admins like me to "test" the quality of their internet connection. So of course you want to prioritize servicing DNS clients over anything else.

06/24/2020 19:53:07

In today's world with everything either burning or smoldering, it can be distressing to wake up each day and face the latest developments. Everything that is happening now is happening because, like the Empires of yesteryear, systems rise and they fall. The sad truth is that most people have developed a blind spot when it comes to issues of this sort. That's primarily because getting there requires a bigger picture perspective most people seem to lack.

Regardless, many systems in tech seem to be modeled after the huge gargantuan centralized bureaucracy that is the US Government, which I would argue is clearly in the midst of it's own downward spiral. You can see this trait present in a wide variety of systems, most notably in the proprietary ones pushed by corporations hellbent on acquiring intellectual property and data on their users as their first, second and third orders of business.

06/11/2020 15:47:26

Today marks an especially great day for me as Presentation Engine, which is the name of the software that makes this website possible, has officially hit its 5.1 release. This marks a huge leap forward for the software from a front end and deployment standpoint. Read on for more details.

For starters, the admin portions of the website have been completely rewritten with React.js. There are no longer any dependencies upon Vue.js in Presentation Engine 5.1. Let me be clear, I really do like Vue.js a lot. But as the primary purpose of this application is to give me a platform with which I can keep my skills current, Vue.js is no longer an appropriate choice. The cold harsh reality is that React.js won the front end library development war. When I look at open position descriptions, I'm not seeing Vue.js but I am seeing quite a bit of React.js.

04/17/2020 14:45:37

This last month has been a wild ride. It's been extremely eye opening as well. Needless to say while I'm still processing a lot of this, I have come to the decision that its time to change the nature of this website.

UPDATE 5/3: I have decided to scale back on this after thinking it over for a few weeks. I have restored all Coronavirus content on the website and reactivated my Twitter account. But fuck Facebook and thin-skinned snowflakes (regardless of which side of the aisle they occupy).

03/12/2020 22:37:14

I haven't posted anything here in over a month but rest assured that isn't because I don't have anything to say. It's been a chaotic, busy and unrewarding month. Ultimately I've decided to hunker down and be grateful for the things I do have at least for the foreseeable future.

So what's happening? Well we are going to talk about a recent experience that I had with a former employer, Paylocity. Now I've been careful to not speak overtly on the subject of Paylocity (non-overt posts not withstanding) since I stopped working for them a couple years back. The are three primary reasons for that.

Search:
[Top] [Rss] [Email]