Automating and sending speedtest.net data to web services

Recently I got frustrated by a series of broadband service failures. I realized they were difficult to diagnose both by me and my service provider (who, by the way, was very helpful) because it was difficult to determine when exactly they occurred and whether the issue was with the broadband connection or my wireless router. This weekend, inspired by this Make: Magazine feature, I hooked up a Raspberry Pi to my broadband router, set it up to periodically query speedtest.net (using speedtest-cli) and log the results.

I’m not a particular fan of IFTTT, which I find too linear and limiting (not to mention a certain arrogance towards third-party content providers) and thus I looked for alternative ways to post my speedtest results to an online place where I could obsessively check them whenever I’m out of the house. I liked this post describing how to use the same speedtest-cli with Loggly instead of IFTTT. But of course I wasn’t satisfied with hashing together a bunch of perl one-liners, so instead I found this script to manipulate speedtest-cli output, and modified it so it could log results to a CSV file, post them to IFTTT, Loggly or to any URL that would accept JSON, such as Zapier:

./speedtest-extras.sh [-d] [-c] [-h] [-i secret-key] [-l]
    -d: debugging-mode (reuses previously logged speedtest result instead of queriying speedtest - faster)
    -c: CSV mode
    -h: Print CSV header (only if used together with the -c flag)
    -i: IFTTT mode. Takes an IFTTT Maker Channel secret key as argument (required)
    -l: Loggly mode. Takes a Loggly Customer Token as argument (required)
    -j: JSON mode. Posts the result as a JSON document to any URL passed as argument (required)

My modified command-line interface to speedtest.net is available on GitHub, where I’ve also posted a few usage examples. Here, I will concentrate on how to use it to post to Zapier.

How to automatically send speedtest results to Zapier

First, take care of dependencies. My script makes use of speedtest-cli, which in turn is written in Python. Assuming you’ve got a working install of Python, you can use your favourite package manager to get hold of speedtest-cli:

$ pip install speedtest-cli

Then download my code, either as a ZIP archive or by using git:

$ git clone https://github.com/timtomch/speedtest-cli-extras.git

Once you have downloaded my repository, navigate to the bin folder1 that’s inside it:

$ cd speedtest-cli-extras
$ cd bin

Then you can try running my script in CSV mode to make sure everything is working properly:

$ ./speedtest-extras.sh -c
2016-03-29 02:33:38 UTC;2016-03-29 02:34:19 UTC;Start Communications;XXX.XXX.XX.XXX;SoftLayer Technologies, Inc. (Toronto, ON);8.53 km;17.794 ms;23.97 Mbit/s;1.95 Mbit/s;http://www.speedtest.net/result/XXXXXXXX.png

Depending on the speed of your Internet connection, it should take about a minute to run the test. If you see output similar to the above, things are working.

It is now time to setup Zapier to receive your data. If you haven’t got an account yet, go ahead and create one (the free plan should work just fine). Then click the bright red “Make a Zap” button to get started.

Using the search box, choose “Webhooks by Zapier” as your trigger, then select the “Catch Hook” option. Leave the next screen (options) empty and click Next until you reach a screen that should look like this:

Screenshot of the Zapier interface showing which URL to send JSON data to.
Setting up a Webhook on Zapier.

Zapier will issue a custom webhook URL to trigger your events. Copy that URL to the clipboard.

Now run

$ ./speedtest-extras.sh -j <PASTE YOUR ZAPIER URL HERE>

and wait again for the prompt to reappear. If nothing else shows up on your Terminal it’s a good sign. Go back to your browser and click the blue “OK, I did this” button. After a short while, Zapier should display a nice green message saying the test was successful. Go ahead and click on the “view your hook” link to check what data was sent to Zapier. You should see something like this:

Screenshot of the Zapier interface, showing data submitted via a JSON Webhook.
Testing the Zapier Webhook to ensure the JSON data was properly received.

Then you can decide what to do with that data. I chose to have each event add a new line to a Google Spreadsheet:

Screenshot of the Zapier interface, showing options to set up a Google Spreadsheets app.
Setting up Zapier to add rows to a Google Spreadsheet.

Go ahead and test your setup, then save your Zap once you are happy with the results. Don’t forget to turn on your Zap.

Now, every time you fire

$ ./speedtest-extras.sh -j <PASTE YOUR ZAPIER URL HERE>

Zapier will execute the operation you specified (add a row to a Google Spreadsheet in my example). Now, if you had to manually run the script to get a measurement, that would defeat the whole purpose, so the last step is to add a cron job so the script is run automatically:

$ crontab -e

This lets you edit your crontab. To run a speed test every hour, add the following line to it:

0 * * * * ./absolute/path/to/speedtest-extras.sh -j <YOUR ZAPIER URL>

Note that you need to specify the whole path to the speedtest-extras.sh script in your crontab for it to work.

Now watch the data slowly pile up, and start drafting that email to your broadband provider.

Next step: full Raspberry Pi tutorial?

A recent conversation with a friend facing the same issue made me think I could also write up a short tutorial on how to replicate my Raspberry Pi speed tester setup from scratch. Anything to avoid working on more useful things, like getting ahead on my MLIS research or freshening up my resume for this position I’m considering applying to…

  1. This directory structure is not entirely necessary but is a leftover from the original speedtest-cli-extras which I forked.