Writing a UDEV rule for the ENVI-R Serial USB

By default the ENVI-R’s serial USB adapter will get a device name such as /dev/ttyUSB0, but this may wander if you have multiple serial devices connected.  In order to get a fixed name, create an appropriate .rules file in /etc/udev/rules.d, and enter this;

SUBSYSTEMS=="usb", ATTRS{idProduct}=="2303", ATTRS{idVendor}=="067b", NAME="%k", SYMLINK+="envir_serial", MODE="0660"

With this, you’ll wind out with a /dev/envir_serial that you can connect your scripts to.  This can be done with any such device, just determine the vendor ID from running lsusb.

ENVI-R Updates, Other Stuff

Whew, been a while.  I’ve been a bit slack with keeping up with my projects lately – too busy with other things.  However, I have a few updates.

I got a solar panel installation done last week, and as such I got a second transmitter for my ENVI-R power monitor.  I discovered a problem with the moving average code in my ENVI-R scripts, so I’ve corrected that and updated the GoogleCode repository.  It will now handle multiple transmitters without problems.

I’ve also gone and made button extensions for the three pushbuttons on the Arduinoven’s front panel.  Manufacture was simple – I used a piece of aquarium-grade rubber tubing, which pushed over the button’s shaft, and then a piece of aluminium rod which went in the tube and was rounded on the end.  Buttons hold well enough, and they won’t fall out.  Now to drill the holes in the front for them, and then cut out a space for the LCD with a coping saw.

In other news, the Microserver is going quite well now that I’ve rebuilt it onto another USB key.  I’m now using tmpfs for both the MRTG data and /tmp.  Doing so appears to have resolved the minor issues I used to have, where I/O would pause for up to 10 seconds at random intervals.  My thoughts here is that when MRTG was doing its five-minutely update, it was queueing up a lot of I/O to the USB key, which prevented any other I/O while it was in progress, causing the pause.  Now it runs against tmpfs I’m no longer getting that problem.

I’ve got some ideas for a battery-powered temperature sensor powered by an ATtiny, which would send out temperature data via radio once a minute or so.  More on that when it happens.

ENVI-R – Putting the last pieces together

In my last post about the ENVI-R setup, I discussed setting publish-envir.pl in order to parse raw serial data from the ENVI-R and post it out to MQTT channels.  In this post, we’ll discuss how to get THTTPD and MRTG going, in order to actually get some useful graphs out there.


Glue Scripts

MRTG can’t use data direct from MQTT, so we need a fairly simple Perl script to pull the MQTT data out and hand it to MRTG in a useable format.  An MRTG “packet” is of the following format;

A response must have four and exactly four lines.  So, our glue script has to haul in data from MQTT and then output four lines for MRTG to use.

Have a read of mrtg-envir.pl at my GoogleCode repository.  The code there should be fairly self-explanatory.  The only complex bit is CalculateReading, which is a horrible bit of code that tokenizes and parses your input so you can do things like “return the first sensor subtract the second sensor” with “0.1-0.2” and stuff like that.

It’s important to note that the script will wait until MQTT publishes a message.  This means that if your publish-envir.pl script isn’t running, or the ENVI-R isn’t working or something, then MRTG will also be held up.

THTTPD Setup

Setting up THTTPD is very, very easy.  From Ubuntu, just do this;

sudo aptitude install thttpd
service thttpd start

And that’s about it.  By default, it’ll share out stuff in /var/www, and has CGI turned off.  You don’t need CGI in the short term.  By default it’ll also have chroot enabled and will be reasonably secure.  All it can really do is pass out pages, but it has a tiny memory footprint and is very fast.  And for the basic cron-driven MRTG setup, that’s all you need.

In order to test, just create something like /var/www/index.html and put the text “Hello world” into it, and make sure you can fetch it.

MRTG Setup

MRTG is also very straightforward to set up.  Just sudo aptitude install mrtg to install it.  By the default setup, MRTG will not run as a daemon, it’ll run as a cron job executed every 5 minutes.

A warning about MRTG.  MRTG isn’t very scalable – you’ll want to use rrdtool or cacti if you want something big.  But for something simple, easy to set up, and only for a few samples, MRTG does the job quite nicely.

Have a look at my example mrtg.cfg at my GoogleCode repository.  Drop that into /etc/mrtg.cfg and run the following commands and you’re sorted;

sudo su –
mkdir /var/www/mrtg
indexmaker –output=/var/www/mrtg/index.html /etc/mrtg.cfg
env LANG=C /usr/bin/mrtg /etc/mrtg.cfg

exit

If you now look at the contents of /var/www/mrtg, you should see a number of files, images and the like.  Pop open your web browser and browse to http:///mrtg and you should see some graphs!

Final Notes

One thing you’ll notice is that the publish script will accumulate a 5-minute moving average, which happily lines up nicely with the sample interval for MRTG.  But the calculated averages for the weekly and monthly graphs are calculated by MRTG, and will often miss out spikes in usage which you may want to see.  Additionally, the linear vertical scale can miss out low levels of base load or make the hard to see  Consider using logarithmic for the vertical scale.

You’ll also notice in my GoogleCode repository a very simple CGI script for dumping out the instantaneous data from envir-last.  You’ll need to enable CGI on THTTPD for this to work.

All in all, the setup was a good bit of fun and some problem solving, and now I’m collecting some useful electricity data.  Some observations are that careless use of lighting burns a surprising amount of power, and that the aquarium heater for our freshwater turtle was turned up too high and was blowing a lot of power as well.

I’ll need to do some baseload analysis of devices on standby, because I still feel that the base load power use is way too high.

ENVI-R – Data Parser

In my last post about my ENVI-R setup, I talked about how to set up RSMB.  Now we’ll talk about setting up the appropriate scripts so that you can get data parsed from the ENVI-R and have it published to the appropriate channels on the message broker.

Note:  It’s been a long time since I’ve done much Perl coding.  As such, this code is probably horrible.  It works, but yeah, don’t expect a grand example of clean Perl code.

Before you can use this script, there’s a number of things that should have already been set up;

  • The Message Broker should be running.  Read my previous post for setup.
  • You should have your ENVI-R all connected and ready on /dev/ttyUSB0.  Read this post for setup.
  • You’ll need Perl installed (this should be in by default).
  • You’ll need the WebSphere::MQTT::Client CPAN module.  Read this howto for how to install.
  • You will also need the Device::SerialPort , Data::Dumper and Clone modules.  Dumper should already be installed.
  • The Perl script available as publish-envir.pl in my GoogleCode repository.

Right.  After all that, drop that script in /usr/local/bin (or somewhere else fairly sane) and just run it.  It will spit out various bits of output about what’s happening by default.  You can make an init.d script of the same kind of form as the one you made for the broker to run it on boot.  But just run it by hand first.

What you will notice is that the script will publish data received from the ENVI-R to three channels; envir-raw, envir-last, and envir-average.  The purpose of those channels is as follows;

envir-raw:  The last raw message received from the ENVI-R.  The only processing done to this is to make sure that the line received is of the form X, and it drops X out to the channel.

envir-last:  The parsed values of the last received ENVI-R message.  The timestamp that is generated here is the UNIX timestamp on the host at the time the message was received, NOT the time as far as the ENVI-R is concerned.  Sensors are defined as X.Y where X is the transmitter number (0 is the first) and Y is the channel number (1 through 3).

envir-average:  The calculated 5-minute moving average of the messages received.  The timestamp here is the timestamp on the host at the time the average was sent to the channel, not the ENVI-R’s time.  Sensor values are the same and are calculated from the sensors present at the start of the moving average period.  So if you add a sensor, it won’t show up in the average for five minutes.

Due to random variance in the temperature and power readings, it’s strongly advised to use the envir-average channel for normal charting, and to just use envir-last for looking at instantaneous data or making sure everything works.

Once you’ve got it running, swap to another window and run stdoutsub envir-last.   After a few seconds you should see parsed packets coming in from the script in a form you can easily use with MRTG or anything else you want.

If you don’t see anything after a while, check the envir-raw channel, look at what the script is outputting to the console, and finally check the /dev/ttyUSB0 device itself to make sure you’re seeing data.

Assuming it’s all working, we’re nearly there.  All that remains now is THTTPD, MRTG, and the final glue scripts.

ENVI-R – Data Broker Setup

Continuing my previous post on this topic, I realized I needed a message broker in order to permit me to have a daemon polling the ENVI-R for data, processing it, and then making it available in a palatable form for various scripts and for MRTG.  It would have been possible for me to simply have a cron job dump a summary table to disk every few seconds, but since my Linux box uses a CompactFlash card for disk storage, that would quickly kill the flash chips.  I needed something that held the messages in memory only.  That’s where a MQTT message broker steps in.


One of my colleagues at work put me on to MQTT, and in particular he put me onto IBM’s Really Small Message Broker (RSMB).

RSMB is a really tiny implementation of an MQTT compliant message broker.  When they say tiny, they aren’t kidding – the broker is a 78k executable which requires an included 73k library.  (config file is optional but a good idea).

Setup Notes

Given that RSMB is such a tiny thing, setting it up is very easy.  However, I did a few extra steps to make it a little more secure and integrated;

  • Dump broker (the executable) into /usr/local/bin.
  • Do the same with stdoutsub and stdinsub .
  • Copy libmqttv3c.so to /lib as libmqttv3c-1.2.0.so and then softlink libmqttv3c.so to it.  This conserves the normal sanity with libraries that ld.so expects, and it should be a bit more maintainable.  Probably not the best way to do things, but it works.
  • Create a new user with no special rights named broker.  After creation, edit /etc/shadow (there’s probably a better way to do this) and change the second field on the broker line to *.  That ensures nobody can log in using the account.
  • Create a config file in /usr/local/etc/broker.cfg (config text follows).
  • Create a new directory in /var/local/broker .  chown that directory to broker:broker.
  • Create a new /etc/init.d/broker script and add it to startup with update-rc.d .

The config file contents are;

# config file for IBM’s RSMB (Really Small Message Broker)

port 1883
max_inflight_messages 500
max_queued_messages 3600
persistence_location /var/local/broker/

The init.d script is;

#!/bin/bash
### BEGIN INIT INFO
# Provides:          broker
# Required-Start:    $network $local_fs
# Required-Stop:     $network $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Should-Start:      broker
# Should-Stop:       broker
# Short-Description: start really small message broker (broker)
### END INIT INFO

pushd /usr/local/bin
su -c “nohup /usr/local/bin/broker /usr/local/etc/broker.cfg >> /dev/null &” broker
popd

Testing the Install

Broker comes with a couple of other binaries, which can be used to publish messages to a channel, and also to view messages on a channel.  Testing broker is quite easy.

  • Start up broker either manually (good idea the first time) or with the init.d script.
  • In one window, run stdoutsub example to start listening on channel “example”.
  • In another window, run stdinpub example and then start typing in lines.  You should see each “message” you enter appear on the stdoutsub window.
  • If you do, great.  If you don’t, verify that the ports are all OK and opened on your firewall (if any).

Next Steps

Right, getting the broker up provides the underpinning that gets used by the rest of the software that I’m discussing

From CPAN, you’ll need to go and install Websphere::MQTT – that’s the CPAN module for interfacing with IBM Websphere’s MQTT implementation.  But because it’s MQTT compliant, it’ll work just fine with RSMB.  That module will be used fairly heavily in the other scripts.  You’ll also need Device::SerialPort and Clone , for other parts of the scripts.

Followup posts will be outlining how to set up THTTPD and MRTG, and then tying it all together with Perl and Bash glue.

ENVI-R & MRTG – Overview

I recently had an ENVI-R wireless power monitor installed, and I set it up to record data to an always-on Ubuntu Linux box I have sitting around using MRTG.  The setup required a fair bit of scripting in Perl, Bash, and a couple of extra bits of software.

This post is the first in a series outlining just how I set up the monitor and what was required.

The ENVI-R and accessories

I bought the ENVI-R unit in a pack which included the transmitter, a receiver LCD display, power supply and one current clamp.  I also bought an additional two current clamps and the USB connector cable for the receiver.

The USB connector is actually a specially wired Prolific PL2303 USB to RS232 serial port adapter, with an RJ45 connector on the end.  I don’t know the exact pinout, but that’s not required.  Anyway, the adapter “just works” with Ubuntu 10.10.

Each transmitter unit can handle up to three current clamps, and one receiver can handle up to ~10 transmitters.  I didn’t want to buy additional transmitters, and there isn’t a huge amount of space in my switchboard, so I just got an additional two current clamps bringing me to three.  That allows for the monitoring of up to three loads.

Clamp Installation

WARNING – Mains voltage can be lethal.  In addition, tampering with your switchboard without an electrician’s license may be illegal, as well as dangerous.  Get an electrician to install the clamps for you.

The clamps themselves are no-contact types, and go around the active wire of whatever feed you want to monitor.  They should be clamped entirely around the wire so that the ferrite core of the clamp encircles the active wire.  The clamp operates by picking up the EM field around the active wire as current passes.

Be aware that the clamp cannot identify the direction that current is flowing.  This isn’t a problem in the case of a house like mine that has no solar power, but if you have solar power you’ll want to install a clamp onto the feed coming from the solar cells, and then put your main power clamp after the location where the solar power feed connects to your main feed.

In my case, my electric hot water is on a different circuit from the main power, so the three clamps were connected as so;

  • Main power:  Clamp connected after main breaker, between breaker and switchboard.  Registers all power going into the house (except hot water)
  • Lights:  Clamp connected after breaker for the lights (I only have one).
  • Hot water:  Clamp connected after breaker for the hot water.

Be aware as well that the ENVI-R assumes if you have multiple clamps connected to one transmitter that you’re measuring 3-phase power, and it therefore just adds together all the currents.  In the case of mains + hot water that’s OK, but since my “main” clamp is actually registering the sum of lights and everything else, it’ll over-read if the lights are on.  For that reason, if you’re setting up like me, don’t trust the display, use the serial data feed.

ENVI-R Receiver Installation

The receiver is pretty straightforward, and it’s just plugged in, and then the USB serial cable is hooked up.  I’ll run through the software to actually interpret the data in a useable format for MRTG later.

The ENVI-R communicates at 57600 8N1 speed.  If you want to just see the raw output, run this command from the command line;

$ stty -F /dev/ttyUSB0 speed 57600
$ cat /dev/ttyUSB0

You should see, about once every five seconds, output something like this;

CC128-v1.310002222.00033221000000029100000

If you do, great.  The ENVI-R is all hooked up.  Note that it also monitors temperature, and that temperature is the temperature of the receiver.  Note, if you don’t have a transmitter attached, you won’t see any output from the ENVI-R at all.

The Software

For this kind of thing, most people seem to use Cacti.  While it’s definite that Cacti can do more, the box I’m using has very limited memory, so I need to keep the number of running daemons to an absolute minimum.  MRTG can run as a Cron job, and doesn’t require a database daemon to be running as well.  For the small number of graphs I’m talking, Cacti is overkill.

In order to pass data to MRTG, I used a message broker written by IBM (Really Small Message Broker), since it does the job and it’s tiny.  Then I wrote a couple of Perl scripts to handle converting the raw ENVI-R data into moving averages for MRTG to slurp up.  So, the software required is;

  • IBM’s RSMB or other MQTT-compatible message broker
  • MRTG for generating the graphs
  • A web server of some type.  I used THTTPD, since it’s very small and fast.
  • Perl.  It’ll come with Ubuntu, but you will need a number of CPAN libraries.

Up Next

If you get to this point, you should have a connected up ENVI-R with a few clamps, and it should be hurling data out to /dev/ttyUSB0 on your Linux recording box.  Fantastic.

Now for the software….