AussieBroadband CheckMK Plugin

I’ve recently changed my ISP to AussieBroadband.  Since I’m now working under a quota, I want a way to monitor my quota in CheckMK.  Enter a bit of python.  If you want to use this, you’ll need to adjust the hashbang to suit your actual OMD site, and then pass a parameter which is your username:password to get onto your ABB account.

#!/omd/sites/checkmk/bin/python2.7
#
# Parses AussieBroadband Quota Page to generate a CheckMK alert and stats pages
#

import requests
import HTMLParser
import re
import time
import sys

# Create a basic HTML parser for fetching the Total Used up/down and Data Left fields
class UsageParser(HTMLParser.HTMLParser):
    # Always given in MB
    quota_up = None
    quota_down = None
    quota_left = None

    # This flag determines the state of the parser
    usage_state = None

    def handle_data(self, data):
        if data == "Total Used":
            # This flag is followed by an upload value
            self.usage_state = "upload"
        elif data == "Data Left":
            # This flag is followed by the quota left value
            self.usage_state = "remain"
        elif self.usage_state is not None:
            m = re.match("([0-9\.]+) ([KMGT])B",data)
            if m:
                value = float(m.group(1))
                unit = m.group(2)

                # Convert unit to MB
                if unit == "K":
                    value = value/1024.0
                elif unit == "M":
                    value = value
                elif unit == "G":
                    value = value*1024.0
                elif unit == "T":
                    value = value*1024.0*1024.0

                # Put unit in correct pool based on value and reset parser state
                if self.usage_state == "upload":
                    self.quota_up = value
                    # Uploads are followed by downloads
                    self.usage_state = "download"
                elif self.usage_state == "download":
                    self.quota_down = value
                    self.usage_state = None
                elif self.usage_state == "remain":
                    self.quota_left = value
                    self.usage_state = None

#####################################################################

status = 0
statustext = "OK"

try:
    creds = sys.argv[1].split(":")

    payload = {
        'login_username': creds[0],
        'login_password': creds[1],
        'submit': 'Login'
    }

    # Fetch the usage page and parse it
    r = requests.post("https://my.aussiebroadband.com.au/usage.php", data=payload)
    parser = UsageParser()
    parser.feed(r.text)

    # Derive some parameters for the check
    total = parser.quota_left + parser.quota_up + parser.quota_down
    critthresh = 0.10*total
    warnthresh = 0.25*total

    # Determine the status of the check
    if parser.quota_left < critthresh:
        status = 2
        statustext = "CRITICAL"
    elif parser.quota_left < warnthresh:
        status = 1
        statustext = "WARNING"

    # Format the output message
    print "{7} - {1} MB quota remaining|left={1};{2};{3};0;{4}, upload={5}, download={6}".format( \
        status, \
        int(parser.quota_left), \
        int(warnthresh), \
        int(critthresh), \
        int(total), \
        int(parser.quota_up), \
        int(parser.quota_down), \
        statustext)

except:
    print "UNKNOWN - Unable to parse usage page!"
    status = 3
    statustext = "UNKNOWN"

sys.exit(status)

Enjoy.  It’s pretty quick and dirty, but it works.  You put this into your site’s local/lib/nagios/plugins directory, then add it as a classical check.

Darktable on Windows through WSL

EDIT:  No longer required.  Since Darktable 2.4.0, there’s an official native Windows installer for it.  Use that instead.  Easy!

With the Windows Subsystem for Linux (WSL) now being much more stable and useable, it turns out it’s possible to install Darktable on Windows with very little fuss.

This will require you to have Windows 10, and also to have at least the 1709 Fall Creator’s Update (run winver and see your version, it should be 1709 or higher).

Follow the instructions here to install WSL, and then go ahead and install Ubuntu from the Windows store.  Don’t bother starting a prompt yet, we have more to do.

Next, you’ll need an X server of some type, to display graphical UI from Ubuntu apps on the screen.  Unless you have something else, I suggest you install VcXsrv, it’s straightforward to install and run.  When running this, just select all the defaults and go ahead.  This will give you an X server on :0, which we will use in a moment.

Now, start up Ubuntu, then type nano ~/.profile and press enter.  Enter the following text down the bottom,

# set display
if [ "$DISPLAY" == "" ]; then
  export DISPLAY=localhost:0
  #export LIBGL_ALWAYS_INDIRECT=1
fi

Press Ctrl-O and Y in order to save.  Now exit that Ubuntu window and start it up again.  If you type echo $DISPLAY you should see the variable above printed out.  This tells programs in your Ubuntu window how to find your X server.

Next, go to this PPA repository, and install Darktable like this;

sudo add-apt-repository ppa:pmjdebruijn/darktable-release
sudo apt-get update
sudo apt-get install darktable

Wait a bit, and Darktable will be installed.  You can now run it by simply typing darktable into that prompt.

No mess, no fuss.  Enjoy.

 

Darktable for Windows using Vagrant

I have an Olympus TG5 camera, which has RAW support for Darktable, but only in the very latest (currently unreleased!) 2.3.0 version.  Since I have Windows, I’ll have to build Darktable directly from source to be able to manipulate it.  Here’s how you can do that.

First, I assume you have Cygwin/X running.  You’ll also need Vagrant installed, along with VirtualBox.  With all that in place, doing the rest is pretty straightforward.  Create a folder, and throw this Vagrantfile into it;

# -*- mode: ruby -*-
# vi: set ft=ruby :

VMBOX = "bento/ubuntu-16.04"
VMHOSTNAME = "darktable"
VMRAM = "1024"
VMCPU = 2

VAGRANT_COMMAND = ARGV[0]

Vagrant.configure("2") do |config|
  # Configure the hostname for the default machine
  config.vm.hostname = VMHOSTNAME

  # Configure the VirtualBox provider
  config.vm.provider "virtualbox" do |vb, override|
    # The default ubuntu/xenial64 image has issues with vbguest additions
    override.vm.box = VMBOX

    # 1gb RAM, 2 vCPU
    vb.memory = VMRAM
    vb.cpus = VMCPU

    # Configure vbguest auto update options
    override.vbguest.auto_update = false
    override.vbguest.no_install = false
    override.vbguest.no_remote = true
  end

  # Mount this folder as RW in the guest, use this for transferring between host and guest
  config.vm.synced_folder "shared", "/srv/shared", :mount_options => ["rw"]

  # Build the server from a provisioning script (which will build Darktable for us)
  config.vm.provision "shell", inline: <<-SHELL
    # Install essential and optional dependencies
    apt-get update
    apt-get install -y gcc g++ cmake intltool xsltproc libgtk-3-dev libxml2-utils libxml2-dev liblensfun-dev librsvg2-dev libsqlite3-dev libcurl4-gnutls-dev libjpeg-dev libtiff5-dev liblcms2-dev libjson-glib-dev libexiv2-dev libpugixml-dev
    apt-get install -y libgphoto2-dev libsoup2.4-dev libopenexr-dev libwebp-dev libflickcurl-dev libopenjpeg-dev libsecret-1-dev libgraphicsmagick1-dev libcolord-dev libcolord-gtk-dev libcups2-dev libsdl1.2-dev libsdl-image1.2-dev libgl1-mesa-dev libosmgpsmap-1.0-dev

    # Install usermanual and manpage dependencies
    apt-get install -y default-jdk gnome-doc-utils libsaxon-java fop imagemagick docbook-xml docbook-xsl
    apt-get install -y po4a

    # Install this for Cygwin/X to work properly
    apt-get install -y xauth

    # Pull the master repo
    git clone https://github.com/darktable-org/darktable.git
    cd darktable
    git checkout master

    # Pull the submodules
    git submodule init
    git submodule update

    # Build Darktable
    ./build.sh --prefix /opt/darktable

    # Build documentation
    cd build
    make darktable-usermanual
    make darktable-lua-api
    cd ..

    # Install Darktable
    cmake --build "/home/vagrant/darktable/build" --target install -- -j2

    # Copy documentation into shared area
    cp build/doc/usermanual/*.pdf /srv/shared/
  SHELL

  # This piece here is run when we use 'vagrant ssh' to configure the SSH client appropriately
  if VAGRANT_COMMAND == "ssh"
    config.ssh.forward_x11 = true
  end

end

Make a shared folder in that folder, and vagrant up followed by vagrant ssh.

Assuming everything is configured correctly, you can then start Darktable with;

/opt/darktable/bin/darktable

And off you go.  You can add some more mounts into the VM as required to share your picture library or whatever with it so you can manipulate it with Darktable.

Adding an RTC to your Raspberry Pi

I use a RPi 3 as a secondary DNS and DHCP server, and time synchronization is important for that.  Due to some technicalities with how my network is set up, this means that I need a real-time clock on the RPi so that it can have at least some idea of the correct time when it powers up instead of being absolutely dependant on NTP for that.

Enter the DS3231 RTC (available on eBay for a few bucks).  The Pi Hut has an excellent tutorial on setting this up for a RPi, which I’m going to summarize here.

Configure I2C on the RPi

From a root shell (I’m assuming you’re using Raspbian like me);

apt-get install python-smbus
 apt-get install i2c-tools

Then, edit your /boot/config.txt and add the following down the bottom;

dtparam=i2c_arm=on
dtoverlay=i2c-rtc,ds3231

Edit your /etc/modules and add the following line;

i2c-dev

Now reboot.  If you do an i2cdetect -y 1 you should see the DS3231 listed as device 0x68.  If you do, great.

Configure Raspbian to use the RTC

After rebooting, the new device should be up, but you won’t be using it yet.  Remove the fake hardware clock with;

apt-get --purge remove fake-hwclock

Now you should be able do hwclock -r to read the clock, and then hwclock-w to write the current time to it.

And lastly, to make it pull time from the RTC on boot, put the following into /etc/rc.local before the exit 0;

hwclock -s

And you can then add a cronjob in /etc/cron.weekly to run hwclock -w once a week.

Done!

Backing up KVM Virtual Machines with Duplicity + Backblaze

As part of my home DR strategy, I’ve started pushing images of all my virtual machines (as well as my other data) across to Backblaze using Duplicity.  If you want to do the same, here’s how you can do it.

First up, you will need a GnuPG keypair.  We’re going to be writing encrypted backups.  Store copies of those keys somewhere offsite and safe, since you will absolutely need those to do a restore.

Secondly, you’ll need a Backblaze account.  Get one, then generate an API key.  This will be comprised of an account ID and an application key.  You will then need to create a bucket to store your backups in.  Make the bucket private.

Now that’s done, I’m assuming here that you have your /var/lib/libvirt library where your VMs are stored on its own LV.  If this isn’t the case, make it so.  This is so you can take a LV snapshot of the volume (for consistency) and then replicate that to Backblaze.

#!/bin/bash

# Parameters used by the below, customize this
BUCKET="b2://ACCOUNTID:APPLICATIONKEY@BUCKETNAME"
TARGET="$BUCKET/YOURFOLDERNAME"
GPGKEYID="YOURGPGKEYIDHERE"
LVNAME=YOURLV
VGPATH=/dev/YOURVG

# Some other parameters
SUFFIX=`date +%s`
SNAPNAME=libvirtbackup-$SUFFIX
SOURCE=/mnt/$SNAPNAME

# Prep and create the LV snap
umount $SOURCE > /dev/null 2>&1
lvremove -f $VGPATH/$SNAPNAME > /dev/null 2>&1
lvcreate --size 10G --snapshot --name $SNAPNAME $VGPATH/$LVNAME || exit 1

# Prep and mount the snap
mkdir $SOURCE || exit 1
mount -o ro,nouuid $VGPATH/$SNAPNAME $SOURCE || exit 1

# Replicate via Duplicity
duplicity \
 --full-if-older-than 3M \
 --encrypt-key $GPGKEYID \
 --allow-source-mismatch \
 $SOURCE $TARGET

# Unmount and remove the LV snap
umount $SOURCE
lvremove -f $VGPATH/$SNAPNAME
rmdir $SOURCE

# Configure incremental/full counts
duplicity remove-all-but-n-full 4 $TARGET
duplicity remove-all-inc-of-but-n-full 1 $TARGET

Configure the parameters above to suit your environment.  You can use gpg --list-keys to get the 8-digit hexadecimal key ID of the key you’re going to encrypt with.  The folder name in your bucket you use is arbitrary, but you should only use one folder for one Duplicity target.  The 10G LV snap size can be adjusted to suit your environment, but it must be large enough to hold all changes made while the backup is running.  I picked 10Gb, because that seems OK in my environment.

Obviously this means I need to have 10Gb free in the VG that the libvirt LV lives in.

Retention here will run incrementals each time it’s run, do a full every 3 months, ditch any incrementals for any fulls except the latest one, and keep up to 4 fulls.  With a weekly backup, this will amount to a 12 month recovery window, with a 3-monthly resolution after 3 months, and a weekly resolution less than 3 months.  Tune to suit.  Drop that script in /etc/cron.daily or /etc/cron.weekly to run as required.

Important.  Make sure you can do a restore.  Look at the documentation for duplicity restore for help.

Splunkd High CPU after leap second addition?

Had my alerting system yell at me about high CPU load on my Splunk Free VM;

A bit of examination revealed that it was indeed at abnormally high load average (around 10), although there didn’t appear to be anything wrong.  Then a quick look at dmesg dropped the penny;

Jan 1 10:29:59 splunk kernel: Clock: inserting leap second 23:59:60 UTC

Err.  The high CPU load average started at 10:30am, right when the leap second was added.

A restart of all the services resolved the issue.  Load average is back down to its normal levels.

Bash on Windows – X Server!

It turns out that you can use Bash on Windows 10 to run X applications, including through ssh tunnels.  Here’s how.

First, go and install XMing.  I’d strongly suggest not allowing it to get access to your network, so it stays on localhost.  This is so that an attacker can’t draw stuff on your screen through your X server.

Run XMing, put it in your startup if you want.  You now have an X server.  Next up, you’ll need to fire up Bash on Windows, and run sudo apt-get install xauth.  Then edit your ~/.bashrc .  Right down the bottom, add the following;

export DISPLAY=localhost:0
xauth generate $DISPLAY

This causes your session to be configured so that you can use X applications and they’ll be pointed to your X server.  It also provides the correct X authentication tokens to make things like ssh work.

Now, log out and back in again.  Start up Bash for Windows, then you can run stuff.

Bash on Ubuntu on Windows 10 – Teething Issues

Just set up the new Bash on Windows 10 feature that comes with the Anniversary Update.  It’s not bad.  But there’s a few annoying things it does that grind my gears.

The default umask is 0000

Yeah.  That’s what I said.  This means all files you create from the Bash shell are read/write/execute to EVERYBODY.  Not smart.  SSH hates that.

echo "umask 0022" >> /etc/profile

To fix that one.

Sudo doesn’t inherit root’s HOME

This causes many commands (pip for example) to dump files into your user directory as root, resulting in an inability to modify files in your own homedir.  Not great.

Add the following in your /etc/sudoers somewhere;

Defaults always_set_home

More as I come across it.

ELK Stack in Docker with NGINX

I’ve done a bit of work in the past few days modifying a Docker ELK Github repository I came across, to make it more suited to my needs.

You can find my efforts at my Github repository.  This setup, when brought up with docker-compose up, will put together a full ELK stack composed of the latest versions of ElasticSearch, Logstash, Kibana, all fronted by NGINX with login required.

The setup persistently stores all Elasticsearch data into the ./esdata directory, and accepts syslog input on port 42185 along with JSON input on port 5000.

In order to access Elasticsearch, use the Sense plugin in Kibana.  You can get at Kibana on port 5601, with a default login of admin/admin.  You can change that by using htpasswd and creating a new user file at ./nginx/htpasswd.users .

A couple of things about Docker in this setup.  When you link containers, it’s not necessary to expose ports between the containers.  Exposing is only required to make a port accessible from outside Docker.  When containers are linked, they get access to all ports on the linked container.

This means that it’s not required to specifically expose all the internal ports of the stack – you only have to expose the entry/exit points you want on the stack as a unit.  In this case, that’s the entry ports to Logstash and the entry point in nginx.

Also, if you use a version 2 docker-compose specification, Docker Compose will also create an isolated network bridge just for your application, which is great here.  It will also manage dependencies appropriately to make sure the stack comes up in the right order.

Oh yeah.  If you bring up the stack with docker-compose up, press Ctrl+\ to break out of it without taking the stack down.

Magic!

NFS Persistent Volumes with OpenShift

Official documentation here.  Following is a (very!) brief summary of how to get your Registry in Openshift working with an NFS backend.  I haven’t been able yet to get it to deploy cleanly straight from the Ansible installer with NFS, but it is pretty straightforward to change it after initial deployment.

NOTE – A lot of this can probably be done in much, much better ways.  This is just how I managed to do it by bumbling around until I got it working.

Creating the NFS Export

First up, you’ll need to provision an NFS export on your NFS server, using the following options;

/srv/registry -rw,async,root_squash,no_wdelay,mp @openshift

Where ‘@openshift’ is the name of a group in /etc/netgroup for all your OpenShift hosts.  I’m also assuming that it a mountpoint, hence ‘mp’.

We then go to that directory and set it to be owned by root, gid is 5555 (example), and 0770 access.

Creating the Persistent Volume

Now first, we need to add that as a persistent volume to OpenShift.  I assume it’ll be 50Gb in size, and you want the data retained if the claim is released.  Create the following file and save as nfs-pv.yml somewhere you can get at it with the oc command.

---
 apiVersion: v1
 kind: PersistentVolume
 metadata:
   name: registry-volume
 spec:
   capacity:
     storage: 50Gi
   accessModes:
     - ReadWriteMany
   nfs:
     path: /srv/registry
     server: nfs.localdomain
   persistentVolumeReclaimPolicy: Retain
...

Right.  Now we change into the default project (where the Registry is located), and add that as a PV;

oc project default
oc create -f nfs-pv.yml
oc get pv

The last command should now show the new PV that you created.  Great.

Creating the Persistent Volume Claim

Now you have the PV, but it’s unclaimed by a project.  Let’s fix that.  Create a new file, nfs-claim.yml where you can get at it.

---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: registry-storage
 spec:
   accessModes:
     - ReadWriteMany
   resources:
     requests:
       storage: 50Gi
...

Now we can add that claim;

oc project default
oc create -f nfs-claim.yml
oc get pvc

The last command should now show the new PVC that you created.

Changing the Supplemental Group of the Deployment

Right.  Remember we assigned a GID of 5555 to the NFS export?  Now we need to assign that to the Registry deployment.

Unfortunately, I don’t know how to do this with the CLI yet.  So hit the GUI, find the docker-registry deployment, and click Edit YAML under Actions.

In there, scroll down and look for the securityContext tag.  You’ll want to change this as follows;

securityContext:
  supplementalGroups:
  - 5555

This sets the pods deployed with that deployment to have a supplemental group ID of 5555 attached to them.  Now they should get access to the NFS export when we attach it.

Attaching the NFS Storage to the Deployment

Again, I don’t know how to do this in the CLI, sorry.  Click Actions, then Attach Storage, and attach the claim you made.

Once that has finished deploying, you’ll find you have the claim bound to the deployment, but it’s not being used anywhere.  Click Actions, Edit YAML again, and then find the volumes section.  Edit that to;

volumes:
  -
    name: registry-storage
    persistentVolumeClaim:
      claimName: registry-storage

Phew.  Save it, wait for the deployment to be done.  Nearly there!

Testing it out

Now, if you go into Pods, select the pod that’s currently deployed for the Registry, you should be able to click Terminal, and then view the mounts.  You should see your NFS export there, and you should be able to touch files in there and see them on the NFS server.

Good luck!