Jay Little - Software Obsessionist
Every man is as God made him, ay, and often worse.
 -- Miguel de Cervantes

08/05/2020 13:18:34

Over the last few months, I have been having trouble with the stability of my home internet connection. This became evident to me while live streaming to Twitch as my stream would begin dropping a lot of frames in a really short amount of time. For a time, recycling the power on the cable modem would make a positive difference, but after a few months that stopped working reliably. This post is about my subsequent experience trying to write a scheduled cron job that would let me know whenever my internet connection was having issues.

A few years ago I had a similar issue. That took six months to resolve and it ended up requiring that some cable run underneath the sidewalk needed to be replaced. So because of the PTSD associated with that hellish experience (which effectively rendered my connection unusable for hours on end at random times, making working remotely from home quite difficult), I took immediate action when I was presented with a similar issue once again.

Once recycling the power on the modem stopped working, I called Charter / Spectrum. They sent a tech out and he replaced the line run from the street, through my yard, into my crawlspace which goes through the floor into my office where the Cable Modem resides. The tech also replaced my Cable Modem (which incidentally was not only an older model, but also "survived" our house getting struck by lightning in 2019).

However that wasn't enough. I wanted to be able to monitor this situation so I wouldn't be caught with my pants down again. I decided to write a simple cron job which would notify me when my internet connection was not working. So how do I did that? Well I decided that pinging some known IP addresses was the way to go. Initially the script was written to simply ping Google's public DNS server @ one hundred times and only produce output in the event that at least one packet was dropped.

Eventually I updated the script so that it would randomly select an IP from a list of IPs, all of which are IPs of publicly accessible DNS servers. Below is the final version of the script in its entirety:



rm /tmp/ping_test_output.txt > /dev/null 2>&1
ping -c $COUNT -D -O $IP > /tmp/ping_test_output.txt 2>&1
if grep -qi " 0% packet loss" /tmp/ping_test_output.txt; then
 exit 0
 echo ALERT: Ping Test to $IP lost at least one packet!
 cat /tmp/ping_test_output.txt | tail -n 2 | head -n 1
 echo -----------------------------------------------------
 cat /tmp/ping_test_output.txt
 echo -----------------------------------------------------
 exit 1

This script ran every half an hour via the magic of cron and in the event any packets were dropped produces an email with the pertinent details. So now I had a great way to track when my internet connection was acting up, right?


As I soon came to realize, even though this script seemed to do its job by sending emails when packets were lost, it didn't actually mean my internet connection was having a problem at all. Nevertheless since I didn't realize this I started calling Spectrum each and every time I started getting more than one or two of these emails within a period of a few hours (as I willing to tolerate maybe one or two a day, because this is the internet after all). At some point it became clear that unlike before, I wasn't seeing other things fail. My streams were not being negatively effected despite receiving these emails and most importantly Spectrum's phone support wasn't able to replicate the packet loss on their end (which they were able to do before).

At this point, I thought maybe that some other piece of network equipment on my side was responsible. My network is largely powered by Ubiquiti equipment and ever since the Coronavirus became an issue, their firmware updates have been less than stellar. I had already rolled back one upgrade on my Wireless Access Point to resolve a separate issue, so I decided to roll back the most recent update on my EdgeRouter X-1 as well.

Initially this seemed to resolve my issues. I went a few weeks and only got an email here and there and all seemed well. Then sometime last week I started getting A LOT of these emails. Every half an hour in fact. Not only that but the packet loss started creeping up from 1% to as high as 3% and 4%. But everything else appeared to be working fine. I was still live streaming without dropping any frames. Internet video services were working fine (exempting of course the occasional glitch from Philo which is just standard operating procedure sadly).

So what the hell was going on? Sadly my entire approach was absolutely idiotic and had never had a real chance of working. However because networking isn't my primary forte and is probably not yours, you are probably scratching your head just like I was wondering what the flaw was. Remember the point of this script is to let me know that my internet connection is having issues, but so far the output produced by this script didn't seem to correlate with any actual issues. So I started to do some research. Eventually I stumbled on this conversation which helped to shed some light on the problem. The subject of the thread really says it all:

DNS Rate Limiting ICMP ( and

Holy shit. Wow. It never even occurred to me that a public DNS server like would put rate limits on things like ICMP. But when you think about it, it makes perfect sense. These servers exist primarily to service DNS clients, not allow half ass network admins like me to "test" the quality of their internet connection. So of course you want to prioritize servicing DNS clients over anything else.

07/30/2020 20:34:43

End This Nightmare

07/16/2020 19:46:22

I am not in the habit of writing disclaimers for my posts here. However I think given the nature of the content that follows, it would be prudent for me to make this very clear: This is an opinion piece. It reflects my thoughts and feelings on the upcoming election and the standing of the Republican party in general as a result of the chaos they have enabled over the last four years. If you find these opinions insulting, that's a shame but ultimately that's just how it's going to be and I won't be offering any apologies.

The title of this post has been written to be as provocative as possible. The reason for this is simple. I want to draw the eye of wandering Republicans. I know quite a few Republicans as I am related to them, work with them, work for them and am friends with them. I live in a red state after all and as such people of a liberal persuasion are few and far between. But if you've been paying attention to the events of 2020, then you know as well as I do that American society has come to a crossroads of sorts. While it has become commonplace for talking heads to refer to every election as a referendum and every victory as a mandate, it's mostly bullshit meant to stir the zeitgeist a bit. However in this particular case for the next four months, it has never been more true. That having been said, I desperately want to reach these people and make my case.

07/05/2020 15:55:55

Never doubt this one simple truth: History repeats. We as humans are doomed to repeat the same vicious cycle over and over again as long as we refuse to learn the lessons imparted by our own history.

When it comes to the subject of whether or not people should wear masks to slow the spread of this pandemic, the answer is clear. We should. Masks made a big difference in 1918, so much so that they were legally mandated, and they can make a big difference today. In recent history, there has never been a more clear and well defined line between stupid and not-stupid than this particular issue. Deny it all you want, but the data is clear: Masks can and will carry the day.

So Fucking Wear One And Stop Whining About It

1918 Spanish Flu Masks

06/24/2020 19:53:07

In today's world with everything either burning or smoldering, it can be distressing to wake up each day and face the latest developments. Everything that is happening now is happening because, like the Empires of yesteryear, systems rise and they fall. The sad truth is that most people have developed a blind spot when it comes to issues of this sort. That's primarily because getting there requires a bigger picture perspective most people seem to lack.

Regardless, many systems in tech seem to be modeled after the huge gargantuan centralized bureaucracy that is the US Government, which I would argue is clearly in the midst of it's own downward spiral. You can see this trait present in a wide variety of systems, most notably in the proprietary ones pushed by corporations hellbent on acquiring intellectual property and data on their users as their first, second and third orders of business.

[Top] [Rss] [Email]