A DIY project to monitor Internet downtime
These days, having access to the Internet is almost as necessary as having access to food and water. Without the Internet, many activities like worldwide communication, journalism, education, business, and financial operations would simply stop working. But, what if your Internet connection is less than perfect? How can Internet quality be measured or defined?
This article describes a project to measure Internet connection quality by counting total network downtime. The project's features are as follows:
- Use the ThingSpeak platform to display continuously measured downtimes.
- When a downtime event occurs, automatically send a Tweet to your Internet provider to alert them regarding the failure in your Internet connection.
- Register a report of all downtimes that exceed a certain specified limit.
This project can also be run using a headless approach if you so desire.
In order to replicate this project, you will need an understanding (at a basic-intermediate level) of the following topics:
- Embedded Linux (command line interface, basic commands, file system structure, and text editors).
- The Python programming/scripting language.
- Basic knowledge of computer networks.
- At a minimum, you should have heard about ThingSpeak and Twitter. If not, then this would be a really good time to start Googling them.
Regarding the hardware, this project was developed on a Raspberry PI, but it can be easily ported to any single-board computer (SBC) that supports Embedded Linux distro, Python programming, and some form of Internet connection. This means that this project will easily work on a BeagleBone Black, a BeagleBone Green, a Linit Smart 7688, and... the list goes on.
Note that the remainder of this article assumes that you already have full access to a single-board computer via a physical keyboard, mouse, and monitor (or by SSH and/or VNC access).
What is Internet connection downtime?
When we talk about Internet connection downtime, we are referring to the total time an Internet connection is not available -- the more the Internet goes down, the greater is the total downtime. Thus, Internet connection quality is inversely proportional to total downtime.
Ok, but what can I do with downtime measurement?
Well, this depends on where you live and your local laws and regulations. In Brazil, for example, there is a regulatory agency for telecommunication issues called ANATEL (this stands for Agência NAcional de TELecomuniações in Portuguese). In resolution number 514/2013 (article 46), ANATEL specifies that every Internet service interruption with a duration greater than 30 minutes will result in in proportional discounts on the Internet provider's bill. Therefore, claiming discounts due to poor Internet service is a very good usage for downtime measurement, and also a good way to drive Internet providers to increase their service quality.
So, how can a downtime be determined?
Based on our earlier discussions, you should now be comfortable talking about downtime and what it represents with regard to Internet connection quality measurement, so now it's time to begin the project itself.
First, we need to define a method with which to measure downtime. The most intuitive and logical method is to periodically ping a reliable server. Actually, I bet almost everyone has done this to check Internet connectivity at least once!
Ping test to check the Internet's availability (Source: Pedro Bertoleti)
This sounds simple and obvious, right? But, not so fast: the time interval between pings matters A LOT! The reason for this is that, if the time interval between two pings is too short, then your IP address may be blocked by the server you're pinging. This blocking policy makes sense, because multiple pings too close together might indicate a very, very simple DDoS attack (even though pings sound silly for a DDoS attack). So, to avoid the risk of being blocked, a good time interval between two ping/ICMP requests is 20 seconds. Of course, this implies that any downtime measurement will be equal to the number of consecutive downtimes (unanswered pings) multiplied by 20 seconds.
In my opinion, one of the most reliable servers for doing this is Google's DNS server (the famous 18.104.22.168). This choice was made based on two facts as follows:
- This DNS server is one of the most used DNS servers worldwide, which leads to the that highly stable and reliable. Due to its heavy usage, we can infer that it has almost no downtime.
- The number of requests each day is huge! To figure out how huge, just imagine almost every computer, smartphone, and tablet in the world requesting Googles DNS's service to browse webpages. It's massive! Considering this, the chance of a project like ours being blocked by continuous access is minimal.
Finally, to determine a downtime, all our project has to do is count how many pings fail (have no response) and multiply this by 20 seconds (the time interval between consecutive pings). Although this technique doesn't result in the exact Internet connection downtime, it does provide a pretty good estimate.
Following a downtime event (when the Internet resumes its response), the duration of the downtime event is sent to ThingSpeak. Also, a Tweet is sent to the Internet provider (using a web app from ThingSpeak called ThingTweet). Therefore, two actions are taken as follows:
- ThingSpeak stores the incoming data and displays it on a chart (in the user's Thingspeak channel).
- By means of ThingTweet functionality (a ThingSpeak App) and the user's Twitter account integration, ThingSpeak will automatically send a tweet to the Internet provider informing them that a downtime occurred and detailing its duration.
To sum things up, the way in which this project works is illustrated below:
How this project works (Click Here to see a larger image. Source: Pedro Bertoleti)
As a plus, all tweeted messages are followed by the hashtag #DownTimeDetected, resulting in a very interesting feature: the more people use this project, the more Internet providers will be monitored regarding their Internet connection quality. So, if you search for this hashtag in Twitter, you'll automatically see a list of Internet providers that failed to serve good Internet connections. To sum up: you'll be able to use faithful old Twitter as a tool to check which Internet providers provide good or bad service!
First things first: setting-up the Cloud
As discussed this projects requires access to a ThingSpeak account and a Twitter account in order to perform its magic. Therefore, these accounts must be established and available before proceeding further with this project.
Twitter Account: There's no mystery here. Just go to the Twitter website at Twitter.com and create an account. It's as simple as it sounds.
ThingSpeak Account: This includes channel creation and ThingSpeak integration with Twitter. In order to do this, please perform the following steps:
- Go to the ThingSpeak website at ThinkSpeak.com
- Click on "Sign Up" at the top of the webpage and fill the form to create an account
- Once your account has been created and you're logged in, click on "My Channel" at the top of the webpage and then click on "My Channels".
- You will be redirect to your channel's administration webpage. Click the "New Channel' button.
- Fill in the form to create your ThingSpeak channel. Be sure to enable just one field to the channels.
- Now it's time to integrate ThingSpeak and Twitter by means of the ThingSpeak App. Before you proceed, open a new tab in your browser, log into your Twitter account, and keep it logged in.
- In the ThingSpeak browser tab, click on "Apps" and then on the "ThingSpeak" icon as shown below.
- In ThingTweet's webpage, observe the "Example API Endpoint" section. Copy the API Key of this section and save it (or keep in a safe place). This API Key will be required in the project's code, which we will discuss shortly.
- Click the "Link Twitter Account" button and follow the instructions to link your Twitter account to ThingTweet.
That's it! Now you're ready to proceed to the project itself.
Talk is cheap, show me the code!
It's code time! Open your favourite text editor (in my case, nano) and enter the following code into the editor (Click Here to access a Notepad/text version of this code):
(Click Here to access a Notepad version of this code)
Save this as a .py file and, to execute it, just type the following into your CLI (command-ine interface):
Note: Some Linux distros don't allow common (non-privileged) users to send pings / ICMP packets. In this case, it will be necessary for you to execute this command as a super-user (using sudo).
Nice! But, what's next?
Well, considering the points we've already discussed, there are a lot of possibilities we can explore! A few things you can do based on this project are as follows:
- Configure your Linux distro to execute this program in boot (auto-start). By doing this, you can dedicate a single-board computer to this task and ensure Internet connection monitoring 24 hours a day, 7 days a week, 365 days a year.
- You can generate some complementary data of your "Internet connection health" and send it to ThingSpeak as well. In fact, I'm sure you'll want to get more data about your internet connection at some point.
- A nice hack for a rainy weekend would be as follows: most modern routers support OpenWRT (a very lightweight Linux distro focused on working in network devices, such as routers, IP cameras, and IoT devices). So, if you were to burn OpenWRT firmware into your router, you'll have a router and a Linux embedded system at your service in the same hardware. This means you could embed this project into your router and let the router automatically monitor its Internet connection.
This article presented a project to monitor Internet downtime. This project uses very simple network concepts and ThingSpeak Cloud platform functionalities. Also, Tweets regarding downtimes are sent, thereby transforming Twitter into a worldwide database of Internet providers' quality.