Category Archives: NumerousApp

Traffic Light shows … Numerous is still dead

A while ago I built a simple visual display for the current Austin-area air quality index: http://neilwebber.com/notes/2015/10/24/traffic-light-shows-air-quality-index/

The light was green for good AQI, yellow for medium, and red for bad air quality. For the AQI data I leveraged a “metric” that I had already developed using the Numerous service. Being a hard core nerd I also covered the case where the Numerous server could not be contacted; the display is red and yellow for that.

As you can see here, the error handling code works. Numerous shut down operations permanently on May 1, 2016 and my traffic light display is now simply a reminder of their demise. 🙁

Numerous is down

 

Traffic Light shows Air Quality Index

I recently saw this project: Traffic Light Shows Internet Status at the “WTH” (whiskey tango hotel) website and decided it would be fun to build. So here we go.

I found the same traffic light available on Amazon: http://smile.amazon.com/gp/product/B00IX73BGW

I decided (of course) to make a few changes to the project:

  • I used a BeagleBone Black (BBB), simply because it was something I had sitting in the “random leftover stuff” bin. It’s the same idea as a Raspberry Pi – it’s a small inexpensive box running Linux.
  • I completely redid the internal wiring of my toy traffic light to separate out each LED connection individually (six wires total) whereas WTH implemented his project leaving the original common anode wiring intact (4 wires total coming out of the traffic light). I’d like to say I rewired for a particularly good reason but I didn’t really; I just did it during debugging because I was confused. But, as a result, I am presenting an alternate implementation of the circuitry.
  • I hooked my traffic light up to Numerous (of course) and it is currently displaying my Austin Air Quality Index metric. It displays the red light for bad air quality, yellow for moderate, and green for good.

Modifying the Traffic Light

As WTH did, I opened the base of the light (four screws) and removed the small circuit board inside by cutting its connections.

I then opened the back of the traffic light itself (another four screws). The original connections look like this:

2015-10-21 19.24.29

 

Schematically, the light is wired like this:

 

original-led copy

Notice how they oriented the LEDs to make the common-anode setup easier to wire.

I cut all these connections, took a cat5 cable, separated out three pairs (6 wires) and brought the individual anode/cathode connections from each LED out of the traffic light. Beware software engineers with soldering irons! But it seemed to work out ok.

To help me keep track of what was going on, I was careful to set up this color code:

  • Brown – anode (positive) side of the red LED
  • Brown/white – cathode (negative) side of the red LED
  • Orange – yellow LED anode
  • Orange/white – yellow LED cathode
  • Green – green LED anode
  • Green/white – green LED cathode

Obviously the color choices don’t matter – but if you do rewire your entire light make sure you keep track of which ones are the anodes and cathodes in some methodical manner.

The LEDs need to be wired in series with a resistor (for current-limiting) and connected to the BBB. On the BBB I used GPIO outputs 1-16, 1-17, and 1-18, which are also known in the /sys/class/gpio filesystem as gpio48, gpio49, and gpio50. The BBB has three GPIO chips; chip 0 occupies gpio0..gpio31, so each input/output on GPIO chip 1 is numbered as “n + 32”. Thus GPIO 1-16 is known as “gpio48” in the /sys/class/gpio filesystem.

The pins for gpio48, 49, and 50 are on connector P9 as shown here:

p89

You’ll also need to make a GND connection, which can be common across all three LEDs. I connected it all up according to this schematic:

gpio-led

The BBB GPIO outputs only supply 3.3V whereas the design used by WTH supplies 5V. 3.3V is still enough to light the LEDs but the calculation for the total amount of current (which determines brightness) will be different. The fixed voltage drop across an LED varies by color, but using 1.9V as an approximate average value the amperage calculation for 470Ω looks like this:

(3.3V – 1.9V) / 470Ω = approximately 3ma

This is within the current that the BBB can supply on these particular output pins. The corresponding current flow from the Pi at 5V is:

(5V – 1.9V) / 470Ω = 6.6ma

I decided to try 470Ω and see if it was bright enough for my purposes; it was. If you want to try experimenting you could reduce the 470Ω resistor. Be careful not to overdrive the LED nor overdraw the GPIO outputs.

I stuck with 470Ω as I already mentioned; this has the added advantage of my trivial little circuit board still being ok if I move it to a Pi at 5V.

Controlling the lights

Now to write some software. While I was changing everything else about the WTH approach, I decided to also just use the /sys/class/gpio “special text files” to control the outputs. To do this first you have to “export” the pins you want to control this way. At the shell you do this:

    root# echo 48 > /sys/class/gpio/export
    root# echo 49 > /sys/class/gpio/export
    root# echo 50 > /sys/class/gpio/export

which tells the kernel you’ll be using the filesystem interface for pins 48, 49, and 50. You only have to do this once per boot (assuming no one ever unexports them after you export them).

Then you have to set their direction. You can also give an initial setting (high or low) at the same time:

    root# GP=/sys/class/gpio
    root# echo high > $GP/gpio48/direction
    root# echo high > $GP/gpio49/direction
    root# echo high > $GP/gpio50/direction

At this point you turn an individual GPIO on or off like this:

    root# GP=/sys/class/gpio
    root# echo 1 > $GP/gpio48/value

Writing 1 turns it on; writing 0 turns it off.

I wrote python code to do exactly this. Note that you have to be prepared for the export requests to fail if the pins have already been exported by a previous run. My code for setting up the pins looks like this:

gp = '/sys/class/gpio'
gpdir = gp + '/gpio{}/direction'

pins = {'red':'48', 'yellow':'49', 'green':'50' }

for color in pins:
  try:
    with open(gp + '/export', 'w') as f:
      f.write(pins[color] + '\n')
    with open(gpdir.format(pins[color]), 'w') as dirf:
      dirf.write('high\n')
  except OSError:
    pass

If you want, you could check the OSError and make sure that if something fails it was because the export was already done a previous time. But pragmatically this suffices.

I wrote a python function to turn an individual light on or off:

gpio_v = gp + '/gpio{}/value'

# call with, for example:
#    color == 'red' 
#    onoff == 1
# to turn on the red LED
def light_onoff(color, onoff):
  with open(gpio_v.format(pins[color]), 'w') as f:
    f.write('{}\n'.format(onoff))

# convenience function:
def alllightsoff():
  light_onoff('red', 0)
  light_onoff('yellow', 0)
  light_onoff('green', 0)

Now that we have the basic tools for manipulating the lights, we’re ready for the final code.

Connecting to Numerous

I used my Numerous class library (https://github.com/outofmbufs/Nappy and/or “pip install numerous”) to get the value of a metric I wanted to display; in this case I am reading my Austin Air Quality Index . This is a public metric, anyone who wants to read it is permitted to do so.

The government defines an AQI value of 50 or less as “Good”, a value of 51 to 100 as “moderate”, and values above 100 as various increasing levels of “bad”. I’m displaying those on my light with the obvious mapping to red/yellow/green. The code for doing this looks approximately like this (simplified for presentation here):

import numerous
m = numerous.NumerousMetric('7896524241619416448')

try:
  q = m.read()
except numerous.NumerousError:
  q = -1

alllightsoff()
if q == -1:
  light_onoff('red', 1)
  light_onoff('yellow', 1)
elif q < 51:
  light_onoff('green', 1)
elif q < 101:
  light_onoff('yellow', 1)
else:
  light_onoff('red', 1)

Notice that if there was an error talking to the Numerous server I display both a red and yellow light simultaneously.

The actual code loops around this, reading from Numerous once every three minutes to get any new values.

Bells and Whistles

Right after I get a (possibly) new value from Numerous I run a loop that “rolls” the traffic lights, rapidly looping green, yellow, red, green, yellow, red, etc. This lets you know that the system is working and also turned out to be a cool looking effect. The code for doing this looks like this:

alllightsoff()
dt = 0.03
for i in range(30):
  light_onoff('green', 1)
  time.sleep(dt)
  light_onoff('green', 0)
  time.sleep(dt)
  light_onoff('yellow', 1)
  time.sleep(dt)
  light_onoff('yellow', 0)
  time.sleep(dt)
  light_onoff('red', 1)
  time.sleep(dt)
  light_onoff('red', 0)
  time.sleep(dt)

# ... now update the light based on Numerous

It almost looks visually as if the lights are “spinning” and finally settling on whatever the new value dictates.

In addition to rolling the lights when updating from Numerous, every thirty seconds I blink whichever light happens to be turned on as another way for you to know that the system is still alive and well.

The Actual Code

For presentation purposes here I’ve simplified the code snippets shown in the discussion and edited them to make it fit the screen better. The full source code I am running for my traffic light is here.

Buttoning it all back up

The cat5 wire I used fits very well into the tube/base of the toy traffic light. I did drill a hole in the side of the plastic base so that the light could stand up and the wire could come out through the side of the base. I did this by hand using a pin vise starting with a small bit and basically reaming the hole gradually larger with increasing bit sizes. I’m not 100% sure if that amount of care was necessary, but it worked.

I also wedged some hex nuts into the (now unused/disconnected) battery compartment in the base to weight the traffic light better. I really want to go get some better lead weights for this but for now the hex nuts seem to work reasonably well at keeping the light upright despite the new leverage point from the cable sticking out of the base.

My “circuit” which consists of a whopping three resistors is sitting on a tiny project breadboard that I just double-sticky-taped to the top of the BBB case. The whole thing *almost* fit inside the case; I’m tempted to just hand-solder and hand-construct a small assembly that can in fact be jammed into the case leaving nothing but the cat5 wire coming out of the case and running to the traffic light.

Here’s the result – before I was able to add some cable dressing/management:

done

Air quality was good ( < 50) at the time this picture was taken.

New versions of www.numerousapp.com API libraries

Released new versions of both the Python and Ruby interfaces to the Numerous APIs. This update allows you to call write() with the “only write it if the value changed” feature and just ignore any exception that comes (when the value hasn’t changed). This is probably a more common usage pattern. So now, for example, you can do this in python:

m.write(newvalue, onlyIf="IGNORE")

where before you would have had to write:

try:
    m.write(newvalue, onlyIf=True)
except NumerousMetricConflictError:
    pass

to handle the reported “no change” exception even when you didn’t care about it. Specifying onlyIf as True gives you this older behavior and specifying the string ‘IGNORE’ (must be UPPER CASE) gives the newer behavior.

The ruby library has the same enhancement.

As a reminder of typical installation methods:

% sudo pip install numerous

You may need to do “pip2” or “pip3” instead, depending on your specific environment and which flavor of python you are installing. Also if you are upgrading from an already installed version:

% sudo pip install –upgrade numerous

Similarly you install the ruby gem via “sudo gem install numerousapp” (note slight name difference here).

Github sources:

Documentation for either library can be found on its wiki page (accessible from the github home page links given above).

Arduino and Numerous, with help from IFTTT

Here’s one way to connect an Arduino to a Numerous metric.

The best solution would be to port the Numerous API to the Arduino; however, there is a footprint problem especially with SSL: there is only 2K of RAM on a standard Arduino. I could not find any Arduino ports of SSL (which is required by Numerous).

Maybe Numerous needs to provide a port 80 (non-SSL) interface, though that risks exposing your API key “to the world” in transit. In the meantime, that leaves two alternatives:

  1. Use another computer, local to the Arduino, as a bridge.
  2. Send data directly from the Arduino to Numerous another way.

Option 1 is what I’ve been doing up until now. You write Arduino code to send data out the USB port and write “bridge” code on a small Linux box (e.g., a Raspberry Pi). The Linux code reads the USB data and processes it further (e.g., sends it to NumerousApp). A variation of this can be done with a tiny web server (or other TCP server) on an Arduino instead using USB (I’ve done that too).

With IFTTT integration we can go the option 2 route and eliminate the Linux box, at the cost of giving up some security (more on this later).

You will need:

  • An IFTTT account (free).
  • Connect the Maker channel to your IFTTT account (free).
  • An Ethernet shield (wired or WiFi) on your Arduino.

Go sign up for IFTTT if you haven’t already, and connect the Maker channel.

When you connect the Maker channel you will see a page that looks like this:

mc1

You need the key as shown; this will go into your Arduino code in a moment.

[ No, of course that’s not my real key shown above ]

Next you need to create an event that will be triggered from the Arduino. Go back to your home IFTTT page and create a new recipe, using the Maker channel as the IF part. Give your event a name, in this case I used “arduino_example”

xt1

When you press Create Trigger you’ll be taken to the “THAT” part of the recipe. Find the Numerous channel and select it (you will have to “connect” it if this is your first time), then for this example choose “Update a Number”:

ct

You will next see a screen where you can select one of your Numerous metrics to be updated, and select what value to update it with. You want to use an “ingredient” (click on the beaker/test-tube thing) and you should select one of the Value fields like “value1”. Whatever you select here is what you will code into your Arduino code; as you can see you can return up to three different values to use with one single event.

ing

That’s it … now to write some Arduino code to generate this event.

You need to know the secret key you were given when you connected the Maker channel, the name of the event you created (“arduino_example” in this case), and which of the values you decided to connect to the metric (“Value1” in this case). Then “all” you need to do is generate a simple POST request from your Arduino in the appropriate format.  That format can be found from www.ifttt.com/Maker (you can also recover your key from there). Click on “How to Trigger events” and it will show you the request format you need to generate.

I’ve written some example Arduino code to do this and made it available as a Gist:

https://gist.github.com/outofmbufs/d6ced37b49a484c495f0

As you can see from that code, the format of the URL that you POST to is:

/trigger/{event-name}/with/key/{your-key}

With the server being at “maker.ifttt.com” (port 80 for standard HTTP)

You POST a JSON object that looks like this:

{ "value1" : "123", 
  "value2" : "456", 
  "value3" : "values can be strings too" }

As you can see from the code itself, squeezing this into the primitives available in the Arduino environment isn’t always pretty; you especially need to be very careful about using up too much memory for “programming niceness”. But the code shows you one way to construct the POST, with the minimal set of HTTP headers that will work, and some example values. You can put this into your own Arduino application and modify it as necessary.

A few tricky points worth noting:

  • I don’t know how often you can generate an IFTTT/Maker event without triggering some kind of limit. In this code I limited the updates to once every five minutes.
  • Don’t forget that all-important “L” on your delay() constants if they are “large” (larger then 32000 milliseconds). Arduino is a 16 bit machine and you won’t get the result you want if you say delay(300*1000) with no “L” on the constant.
  • All those string literals use up RAM. You probably already understand this (or will soon) if you are an Arduino programmer. You might want to look up PROGMEM and ways of getting those literals into flash (program) memory instead of RAM.

About security: we’ve bypassed SSL on the path from the Arduino to IFTTT. The path between IFTTT/Maker and our Numerous account is still secure (via SSL) but the POST request that we are transmitting to IFTTT is plain text and potentially viewable “in transit” if you are inclined to worry about such things. This means your IFTTT/Maker key will be visible to anyone who has access to the intermediate routers. If this matters to you, you will need to go back to Option 1 (local bridge/Linux machine and secure connection to the outside world). On the other hand, if all you are doing is exporting the current temperature of your fridge to a Numerous metric, it’s not clear you need to worry so much about this. You should also keep in mind that the IFTTT key doesn’t really let anyone “do” anything other than post bogus events to your trigger. If you are worried that someone would do that to confuse you about your refrigerator temperature, then you need to go back to the “option 1” architecture which uses SSL to communicate to the outside world. But for a lot of applications this will be a handy way to easily connect an Arduino to Numerous (or anything else IFTTT can connect to).

Showing off with shell “nr” (NumerousApp) command

I wrote some scripts demonstrating what you can do with all the bits and pieces I have now for manipulating NumerousApp metrics from python and shell.

In particular, I was interested in finding the maximum value of one of my metrics:3626358291300704487 (this is the actual NumerousApp metric ID)

I have already installed the python numerous client library and the corresponding shell command “nr” that goes with it (see the github page for details). I also already have my NUMEROUSAPIKEY environment variable set and exported. With all that in place I can read the current value of the metric from a shell prompt like this:

% nr 3626358291300704487

I can read the event stream with the “-E” option:

% nr -E 3626358291300704487

Each line of that output is a (somewhat) human-readable display of a value update event on the metric and looks something like this:

4 @ 2015-05-27T20:12:09.375Z by 9310717522265
3 @ 2015-05-27T19:12:09.348Z by 9310717522265 
2 @ 2015-05-27T18:12:09.207Z by 9310717522265 
1 @ 2015-05-27T17:12:09.373Z by 9310717522265 
0 @ 2015-05-27T15:42:11.373Z by 9310717522265 
518 @ 2015-05-27T14:42:09.254Z by 9310717522265 
517 @ 2015-05-27T13:42:09.116Z by 9310717522265 
516 @ 2015-05-27T12:42:09.317Z by 9310717522265 ...

Rather than parse that output I can ask the “nr” command to only print the value field for each of the events:

% nr -E '3626358291300704487[value]'

From there it’s a simple task to write an awk script that will display the maximum found on stdin:

awk 'BEGIN {max=-999999999} {if ($1>max) {max=$1}} END {print max}'

NOTE: In my script the awk command is all on one line. It’s likely showing up in your browser broken into multiple lines; you need to be careful about those line breaks if you are cutting and pasting this example from your browser display.

That script returns the wrong result if all the input values are negative and below the initial value in the BEGIN statement, so you could do it this way if you prefer:

awk 'BEGIN {max="NOTHING"} {if (max=="NOTHING") max = $1; if ($1>max) {max=$1}} END {print max}'

Again be wary of poorly-located line breaks in your browser display if you copy/paste this.

Given those pieces, finding the max value is a simple pipeline:

% nr -E '3626358291300704487[value]' |
awk 'BEGIN {max="NOTHING"} {if (max=="NOTHING") max = $1; if ($1>max) {max=$1}} END {print max}'

Pretty cool, or maybe somewhat demented.

Sometimes you might not have awk (some linux distributions don’t install it by default), so you could also compute the max with a python program. We know python is on the machine or none of this works, so here’s a version using a python program for the max:

MAXPGM="import sys
maxv = None
for x in sys.stdin:
    try:
        xval = int(x)
    except ValueError:
        xval = float(x)
    if maxv is None or xval > maxv:
       maxv = xval

print(maxv)
"

nr -E '3626358291300704487[value]' | python -c "$MAXPGM"

Making this a script with more general options is left as an exercise for the reader.

Of course if we are going  to the trouble of writing a python program we might want to go ahead and write the entire script in python, which might arguably have been simpler. But, this example notwithstanding, it’s still fun to hack around and see how many powerful things you can do with the “nr” command and some shell.

NumerousApp Server API Performance

I added yet another numerous metric, this time tracking the performance of the NumerousApp server itself:  http://n.numerousapp.com/m/1dlt3ykfq5vqs

It’s measured using this python code:

import numerous

nr = numerous.Numerous()
nr.user()
nr.user()

t = nr.statistics['serverResponseTimes']
print (int(t*10000)/10.0)"

 which is using my NumerousApp Python library (“pip install numerous”).

I just make two user() API calls (discarding the returned info), then dig into the statistics array provided in the object to get the most recent server response time (measured as “on-the-wire” time). The reason for doing two calls is connection overhead; the first call will be significantly slower because it has to perform a TCP connect and an initial SSL setup; the second call reuses the connection (HTTP keep-alive) and is much faster. The numbers are typically around 300msec for the first API call and 60-70msec for subsequent calls (also varies substantially depending on the particular API).

The single response time in ‘statistics’ is the time from the most recent (i.e., second) nr.user() call. The int and 10000/10.0 hack is just to convert the fractional seconds in milliseconds down to tenths (truncated, not rounded … sue me).

NumerousApp libraries updated

Newest versions of my NumerousApp API bindings for python and ruby are released. Now includes support for the new fine-grained permissions server APIs.

PYTHON details:

RUBY details:

NumerousApp overview: www.numerousapp.com

API: http://docs.numerous.apiary.io/

Friday the 13th and NumerousApp

I made a new NumerousApp metric that will tell you when the next Friday the 13th is: http://n.numerousapp.com/m/1w4c4tk4uh29j

Nerd info: I wrote this brute-force python program to find the next Friday the 13th:

from datetime import datetime, date

dt = datetime.now()
while dt.day != 13 or dt.weekday() != 4:
    dt = date.fromordinal(dt.toordinal()+1)

It gets today via  datetime.now()and loops until it is a Friday and a 13th. The weekday() method returns 0 .. 6 with Monday as 0; hence Friday is 4. If “dt” isn’t a Friday the 13th, I just bump it to the next day. The toordinal() method converts a date into a number of days since some arbitrary long-ago time. So I do that, add one, and convert back to a date to get the next day.

I didn’t optimize this algorithm because it just doesn’t matter. I could have added code to compute exactly how many days to add to get to the next 13th and then just test if that is a Friday. Even more generally we could write an algorithm that would directly compute when the next Friday the 13th is (has to take month lengths and leap years into account).

But: We know that at most we’ll have to iterate no more than a few hundred times because every year has at least one Friday the 13th (proving this is left as an exercise for the reader). Speed just doesn’t matter here at all and this works and it is simple.

Once a night I run the above code and use my python NumerousApp API class to update the metric. This too could have been optimized; there’s no reason to run the code at all except on a 13th (indeed, except on a Friday the 13th) but it was just easier to stick it into my nightly updates I do for my other metrics.