CH340 Issues on Surface Pro 3?

I picked up a few LoLin NodeMCU v3 boards the other day (you can read a comparison of the various NodeMCU dev boards here).  However, I had a hiccup with it.

I couldn’t get it to work reliably on my Surface Pro 3 (running Windows 10).  Yet, when I plugged it into a colleague’s Lenovo running Ubuntu, it worked fine.

The serial UART (a CH340) was disappearing when the NodeMCU attempted to start up WiFi.  If I held down the reset button so that the ESP never initialized, the UART worked just fine.  Strange.

Then I tried plugging the NodeMCU into a dumb powered USB hub instead of directly into my Surface.  It worked just fine!

My suspicion is that the problem is caused by the power demand reported by the CH340.  It says it needs < 100mA, but I know that the power draw on these things can be upwards of 300mA.  So, I suspect that the Surface is limiting the power draw to what the USB device says is its maximum on startup.  The dumb hub doesn’t care about such things, so it powers the device regardless.

TLDR;  If you have problems with NodeMCU devices vanishing off your USB bus when you do things with them, plug them into a dumb USB hub.

ESP8266 Environment Widget – Software

Last post I mentioned how I got an ESP8266 board.  Well, I’ve put together that with a DHT11 and MQTT to create a widget that senses temperature and humidity, then posts that on a regular basis to your MQTT message broker.  The data comes out in JSON format for ease of use.  I’ll cover the hardware later, once I get the rest of my prototype boards.

Preparing your ESP8266

Following guides mentioned in my previous post, wire up your ESP8266.  Then, connect your DHT such that the output pin is going to GPIO2 on the ESP8266.

Go and install NodeMCU.  Remember to ground GPIO0 if you are going to upload firmware!  You can use either the float or integer version (I used the float version).  The DHT11 only produces integer temperature and humidity stats anyway, so it doesn’t make any difference.

Next, go to my Github repository and grab the two LUA files.  Don’t upload them to the ESP yet!  Just put them somewhere.

If you have installed NodeMCU, you can now stop grounding GPIO0.  Technically you should pull that high with a pullup, but I haven’t bothered yet.  I’ll do that in the final hardware.

Lastly, fire up LuaLoader and configure your wifi.  This is persistent, as long as you don’t reflash the NodeMCU firmware.

The init script

If you put immediately-executing code into init.lua, you run into a problem – you can’t break out into the interactive shell anymore, and therefore you have to reflash in order to get back.  Not optimal.

The way around that is to have init.lua wait a short time and then execute a different script.  You then have time to either manually abort the alarm timer, or get rid of init.lua and reboot.

Here’s a code fragment that does that;

function startup()
	print('executing script')
	dofile('dht_reader.lua')
end

print("in startup, 'tmr.unregister(0)' to abort")
tmr.alarm(0,10000,0,startup)

Pretty simple.  You get 10 seconds to abort before the main script executes.

The reader script

I’m not going to go through the reader script line-by-line, but I will mention some bits about how it works.  First up, NodeMCU is heavily callback driven.  You’re intended to use an event-based paradigm with it, leading to some strange-looking code.  Also, my Lua is pretty limited, I mostly just bumbled my way through it.

Anyway, configuration is with the variables at the top.  dht_gpio = 4 corresponds to GPIO2.  Set your mqtt details appropriately, including the client ID you want this widget to use.  Leave mqtt_ip and mqtt_client nil, they’re used internally.

The actual code execution is event driven by callbacks.  Everything is triggered off by attempting to do DNS resolution for your MQTT server.  NodeMCU includes a nice watchdog timer, and the watchdog is used extensively.  It’s armed at the beginning, and gets reset only when an MQTT message is successfully published.  If anything happens so that an MQTT message won’t be published, something bad happened and the device will eventually reset.

That should cover all possibilities of wifi going away, mqtt broker dying, no IP address etc etc.

It’s notable that NodeMCU on my ESP8266 does not have the sntp module.  This means it has no idea what the real time is, not even close.  As such, I don’t address that.  The ‘raw’ JSON that’s published is intended to be cooked by another script which will add an expiry time.  More on that later.

After unwrapping all the callbacks, our control flow looks like this;

  1. Arm the watchdog timer.  If it goes off, we reboot.
  2. DNS resolve MQTT broker.  If we got an IP…
    1. Connect to the MQTT broker.  If we connected…
      1. Start a repeating alarm every minute.  When that alarm goes off…
        1. Get the DHT sensor details.  If that worked…
          1. Publish that to MQTT and reset the watchdog
  3. Twiddle thumbs

It’s a little more complicated than that (ie, it tries to reconnect to the broker if it disconnects), but not much.

It should be fairly straightforward to wrangle that to support more complex devices such as stuff on an I2C bus.

Enjoy!

ESP8266 First Steps

The ESP8266 is a ridiculously cheap ($2 or so from China), WiFi equipped breakout board which has a 32-bit microcontroller and Flash (1MB) onboard.  It comes in a bunch of different versions (mine is an ESP-01), and all with different Flash capacities.  The pinout of the one I have appears below;

ESP-01 Pinout (courtesy MyElectronicsLab)

The most notable limitation of the ESP8266 is the extremely limited I/O capability.  There’s only two GPIOs in addition to the UART, and even those GPIOs configure the boot mode if they’re held high/low during startup!  There is some info about how to get around this limitation at this article though.

DANGER:  The ESP8266 is a 3.3v level device.  Driving the inputs with 5v signalling without a level converter will kill it!

Anyway.  Using a simple 3.3v USB-to-UART converter and a little wiring will allow you to get something useful out of this in a few ways;

I did the last.  I threw on the floating point firmware for NodeMCU onto my ESP-01, which gives you a nice interactive serial interface, standalone capability, and a built-in WiFi connection library.  You can then upload code you’ve written in a LUA-style language to do stuff on the thing.

Now for the real magic.  You can use the GPIO2 pin to read temperature/humidity data into the ESP8266.  A very simple guide about how to do that appears in the NodeMCU documentation for the dht module.

It also turns out that NodeMCU contains a module for communicating to MQTT message brokers (it’s actually got a whole lot of really handy IOT modules already in place).

Next plan is to assemble some code for the ESP8266 so that it reads temperature/humidity data and then publishes that to a MQTT channel for aggregation by something else.

Should make for some pretty cheap WiFi-enabled temperature/humidity sensors!

ADS-B Reception on RPI3

Well, that was straightforward.  I present you with;

ADS-B Output from dump1090

Output collected using the dump1090 tool.  Setup was pretty straightforward;

apt-get install librtlsdr-dev git cmake libusb-1.0-0-dev build-essential
git clone git://github.com/MalcolmRobb/dump1090.git
cd dump1090
make

Once that’s done, you have a build of dump1090, which can be used with your SDR to decode ADS-B signals from nearby aircraft.  Run this to collect data;

./dump1090 --interactive --net

You should start seeing some dumped output for nearby aircraft.  If you see stuff, great!  Pop open your browser, go to http://YOURRPI:8080/ and then drag the map to near where you are.  Aircraft will appear!

I’m pretty surprised with how well I’m picking up aircraft, given how badly placed my antenna is, and how poor the antenna itself is.  Should work even better once I get a decent antenna.

Software-Defined Radio on RPI3 – First Steps!

Got myself a RTL-SDR Software-Defined Radio (also known as a cheap-as-hell USB DVB-T tuner), and hooked it up to a Raspberry Pi 3 running Raspbian.  My objective here was to just get it working, and eventually I’ll use it for spectrum analysis and ADS-B tracking.

So, I hooked it up, installed GNU Radio (by gods this is a complicated toolkit), and shoved on the default terribad antenna and put it in the shed.

The results?  Well, I got something out of it, but by oath it’s noisy.  I was expecting that, since I have an awful antenna and no ferrite chokes on anything.  But it works!

Example Waterfall plot

The above is a waterfall plot of a small subsection of the regular FM radio band.  It was created using rtl_power (a standard part of the rtl-sdr kit), and a heatmap generator (available here).  The horizontal axis is frequency in MHz, and the vertical axis is time.  Each pixel represents 1kHz of bandwidth and 1s of time.  Brightness indicates received power.

You can clearly see the thick wideband FM transmission at 103.9MHz – that’s a commercial radio station.  There’s a dull band at 103.7MHz (it sounds like noise when tuning into it), and many smaller bands all across the spectrum, which all sound like buzzes when tuning in.  That’s interference.  It’s pretty obvious the antenna is terrible.  But the concept works!

That chart was generated like this;

rtl_power -f 103.5M:104.5M:1k -p 20 -g 35 -i 1s -e 10m sample.csv
python heatmap.py sample.csv sample.jpeg

Now, you can also record arbitrary things.  Here’s a command to record audio to a playable WAV file from the radio station in the above waterfall;

rtl_fm -f 103.9e6 -M wbfm -s 200000 -r 48000 | sox -t raw -e signed -c 1 -b 16 -r 48000 - recording.wav

 

Now to wait for my new antenna bits to arrive…

 

Caller ID Name Substitution with Asterisk

There’s only a pretty short list of numbers that I care about having Caller ID name substitution enabled for on my Asterisk setup, so I elected to use Asterisk’s native database and some adjustments to my extensions to substitute in a name into the CALLERID(name) field.

First up, to insert number-to-name mappings, do this;

database put cidname 5551234 "John Smith"

For each number you want to be able to resolve numbers for.  If you then do

database show cidname

You’ll then see the entries you’ve made.  Now, you need to adjust your dialplan.  Let’s say that your incoming calls go to the Asterisk context “incoming-sip”.  Change that context to “incoming-sip-cidlookup”, and then create a new context like this;

[incoming-sip-cidlookup]
 exten => _X!,1,GotoIf(${DB_EXISTS(cidname/${CALLERID(num)})}?:nocidname)
 exten => _X!,n,Set(CALLERID(name)=${DB(cidname/${CALLERID(num)})})
 exten => _X!,n(nocidname),Goto(incoming-sip,${EXTEN},1)

What this does is pretty easy.  If an entry exists in the cidname database for the incoming call’s callerid, then substitute the name field in that callerid for the name in the database.  Once this is done (or if no entry exists in the first place), then direct the call to the regular incoming-sip context.

For small numbers of mappings, this seems to work pretty well.  I’m looking at having an automated system that looks up incoming unknown caller IDs against a reverse phone lookup system.  More on that when I figure it out…

Simple NAT Traversal with Asterisk

As an experiment, I configured Asterisk for full NAT traversal so that my SIP server could be accessed from the Internet.  This isn’t usually what you want, but here’s how you do it…

DANGER – Doing this will expose your Asterisk SIP server directly to the Internet, and you’ll get all manner of violated by SIP spammers.  Be aware of what you’re doing and what the implications are.

I’m assuming that you have a typical home NAT setup, where you have a dynamic IP, you’re in control of the border firewall (so you can do port forwarding), and your Asterisk install is on an internal network.

Step 1- Port Forwarding

First up, on your firewall, port forward the following ports to your Asterisk box;

5060/udp         # SIP control channel
10000:11000/udp  # the ports we will use for RTP

Step 2 – RTP Port Ranges

Right.  Now, create an rtp.conf in your Asterisk config, containing the following;

[general]
rtpstart=10000
rtpend=11000

This constrains the list of allowed RTP ports for SIP to use for communications.

Step 3 – Lockdown

Now, a word on security.  Put the following into the [general] section of your sip.conf;

context=incoming-public
allowguest=no
alwaysauthreject=yes

Now in your extensions.conf, define that context and put nothing in it.  Anyone who dials in anonymously to your SIP server will be directed to that context and go nowhere.  This is what you want (presumably).  For the love of God, do not put anything outbound in that context!

Step 4 – Configure NAT Traversal

In your sip.conf, in the [general] section (it must be there), add the following;

externhost=PUT-YOUR-DYNAMIC-DNS-HOSTNAME-HERE
localnet=192.168.0.0/255.255.255.0

This causes the Contact headers of outbound SIP packets to be substituted with the IP address of the DNS name you specified there, if the destination is not in the localnet field.  This is important to make NAT traversal work.

Step 5 – Configure SIP peers

For each of your peers, configure them to use NAT traversal as follows (some of these options may not be strictly required, but this is what I did and it worked);

nat=force_rport
qualify=yes
canreinvite=no
directmedia=no

Step 6 – Wait for the bots

Now, if this has all worked, you should now be able to connect to your SIP server from the Internet.

May God have mercy on your server.

Music-On-Hold for Asterisk

Got Music On Hold (MOH) working for Asterisk 1.8.  Turns out that it’s not that hard, but there’s some specific requirements to get it going if you don’t want to use mpg123 or something.

First, you’ll need some source music to use.  If you go to Wikipedia, you can find a whole bunch of OGG music that’s free to use.  I grabbed something by Brahms.  I’d suggest you use stuff from there, that way you aren’t license encumbered and at risk of getting sued by the RIAA hit squads.

Once you have that, you’ll need to convert it to an 8000Hz, single-channel, OGG Vorbis sound file, by using SoX (available for Cygwin, which is what I did).  The command you’ll need is;

sox input.ogg output.ogg channels 1 rate 8000

Asterisk is very fussy about the format, so it must be exactly like that.  Next up, fire up the Asterisk console (asterisk -r) and run the following;

core show file formats

Look through that and verify that ogg_vorbis is listed.  Next, we’ll verify that Asterisk can actually transcode ogg to whatever codec your VOIP is using.  Run the following;

core show translation recalc 10

And verify that for the ‘slin’ row there is a number printed in the columns that correspond to the codecs that you’re using (in my case, that’s alaw, ulaw and g729).

Drop your ogg files into /usr/share/asterisk/sounds/moh/ (in my case).  You will now need to put the following into musiconhold.conf to make it work;

[default]
mode=files
directory=/usr/share/asterisk/sounds/moh
sort=random

Your music-on-hold channel should be set to default anyway, so that should ‘just work’.  If you want, you can create a fake extension in extensions.conf like this to test it;

exten = 444,1,Answer()
same = n,MusicOnHold()

Ringing that extension should result in hearing music.  Consult your Asterisk logs, common issues will be the OGG files not being in the exact format required.

Good luck.

Asterisk in Docker

I’ve decided to bite the bullet, and I’m working on converting my existing Microserver Centos 6 setup across to Centos 7 with Docker containers for all my applications.

Why Docker?  Why not OpenVZ or KVM?  KVM was out straight away because my Microserver simply doesn’t have the spare CPU and RAM to be running full virtual machines.  OpenVZ is an attractive option, but there’s no non-beta release of OpenVZ for Centos 7.  So that leaves Docker amongst the options I wanted to look at.

Asterisk poses some challenges for Docker, namely that the RTP ports are pseudo-dynamic, and there’s a lot of them.  Docker does proxying for each port that’s mapped into a container, and spawns a docker-proxy process for each one.  That’s fine if you have 1-2 ports, but if you may have over 10,000 of them that’s a big problem.  The solution here is to configure the container to use the host’s networking stack, then do some config on the container so that it uses a different IP from the host (to keep the host’s IP space “clean”).  We’ll also be configuring the container as non-persistent so it pulls config (read-only) from elsewhere on the filesystem and stores no state between restarts.  And lastly, we’ll be using CentOS 6 as the Asterisk container OS (since Asterisk is available in the EPEL repository for that version).  It’s not a very new version of Asterisk, but it’s stable.

Let’s get started.  For the impatient, here’s the gist.

Create the Asterisk Container

First, we’ll assemble a Dockerfile.  We’ll base it off CentOS 6, and just install Asterisk.  We use the ENTRYPOINT command so that we can pass additional arguments straight to Asterisk on running the container.

FROM centos:6
MAINTAINER James Young <jyoung@zencoffee.org>

# Set up EPEL
RUN curl -L http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm -o /tmp/epel-release-6-8.noarch.rpm && \
 rpm -ivh /tmp/epel-release-6-8.noarch.rpm && \
 rm -f /tmp/epel-release-6-8.noarch.rpm

# Update and install asterisk
RUN yum update -y && yum install -y asterisk

# Set config as a volume
VOLUME /etc/asterisk

# And when the container is started, run asterisk
ENTRYPOINT [ "/usr/sbin/asterisk", "-f" ]

Pretty simple stuff.  Note that processes should always run non-daemonized in Docker so that it can track the pid properly.

Prepare the Docker Host

Use whatever tool is appropriate (I’m forcing systemd, firewalld and network-manager on myself) in order to configure a second IP for your Docker host’s primary network interface.  Bleh.  Network Manager.

Be aware that when you use the host’s network stack in Docker and don’t explicitly expose the ports you’ll be using, Docker does not configure the firewall.  You’ll need to do that on the host.  We’ll cover that in the Makefile.

Create the Makefile

We’ll use a Makefile to handle all the tasks we’re dealing with in this container.  Here it is;

CONTAINER=asterisk
SHELLCMD=asterisk -rvvvvv

all: build install start

build:
 docker build -t zencoffee/$(CONTAINER):latest .

install:
 cp -f $(CONTAINER)_container.service /etc/systemd/system/$(CONTAINER)_container.service
 systemctl enable /etc/systemd/system/$(CONTAINER)_container.service
 firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 --proto udp -d 192.168.0.242 --dport 5060 -j ACCEPT
 firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 --proto udp -d 192.168.0.242 --dport 10000:20000 -j ACCEPT
 firewall-cmd --direct --add-rule ipv4 filter INPUT 0 --proto udp -d 192.168.0.242 --dport 5060 -j ACCEPT
 firewall-cmd --direct --add-rule ipv4 filter INPUT 0 --proto udp -d 192.168.0.242 --dport 10000:20000 -j ACCEPT

start:
 systemctl start $(CONTAINER)_container.service
 sleep 2
 systemctl status $(CONTAINER)_container.service

shell:
 docker exec -it $(CONTAINER) $(SHELLCMD)

clean:
 systemctl stop $(CONTAINER)_container.service || true
 docker stop -t 2 $(CONTAINER) || true
 docker rm $(CONTAINER) || true
 docker rmi zencoffee/$(CONTAINER) || true
 systemctl disable /etc/systemd/system/$(CONTAINER)_container.service || true
 rm -f /etc/systemd/system/$(CONTAINER)_container.service || true
 firewall-cmd --permanent --direct --remove-rule ipv4 filter INPUT 0 --proto udp -d 192.168.0.242 --dport 5060 -j ACCEPT || true
 firewall-cmd --permanent --direct --remove-rule ipv4 filter INPUT 0 --proto udp -d 192.168.0.242 --dport 10000:20000 -j ACCEPT || true
 firewall-cmd --direct --remove-rule ipv4 filter INPUT 0 --proto udp -d 192.168.0.242 --dport 5060 -j ACCEPT || true
 firewall-cmd --direct --remove-rule ipv4 filter INPUT 0 --proto udp -d 192.168.0.242 --dport 10000:20000 -j ACCEPT || true

Of course, I’m using firewalld here (bleh again) and systemd (double bleh).  You can see that this simply does a build of the container, then puts the systemd service into place and punches all the appropriate RTP and SIP ports on the IP address that Asterisk will be using.

Configure the Systemd Unit

Now we need a unit for systemd, so we can make this run on startup.  Here it is;

[Unit]
Description=Asterisk Container
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStart=/usr/bin/docker run --rm=true --name asterisk -v /docker/asterisk/config:/etc/asterisk:ro -v /docker/asterisk/logs:/var/log/asterisk -v /docker/asterisk/codecs/codec_g723-ast18-gcc4-glibc-x86_64-pentium4.so:/usr/lib64/asterisk/modules/codec_g723-ast18-gcc4-glibc-x86_64-pentium4.so:ro -v /docker/asterisk/codecs/codec_g729-ast18-gcc4-glibc-x86_64-pentium4.so:/usr/lib64/asterisk/modules/codec_g729-ast18-gcc4-glibc-x86_64-pentium4.so:ro --net=host zencoffee/asterisk:latest
ExecStop=/usr/bin/docker stop -t 2 asterisk

[Install]
WantedBy=multi-user.target

The run command there does a number of things;

  • Pulls Asterisk config from /data/asterisk/config (read-only)
  • Writes Asterisk logs to /docker/asterisk/logs
  • Pulls in a g723 codec and a g729 codec for Asterisk to use (read-only)
  • Enables host networking

If you are missing those codecs, remove the two -v’s that talk about them.  Also, you will likely have differently optimized versions anyway (Microserver has a pretty weak CPU, so Pentium4 is the right one to use for that).

Edit the Asterisk Config

You’ll need the default Asterisk config, which you can extract by building the container and running it up with;

docker run -d --name extract -v /docker/asterisk/config:/mnt zencoffee/asterisk:latest
docker exec -it extract /bin/bash
cp /etc/asterisk/* /mnt/
exit
docker stop extract
docker rm extract

From there, you can put in your own customized Asterisk config.  There’s a few bits you need to tweak.  In sip.conf, set udpbindaddr and tcpbindaddr to the secondary IP that you want Asterisk listening on.  in rtp.conf, ensure that rtpstart and rtpend match the ports you set up the firewall for.

Finally, putting it together!

Put your asterisk_container.service, Dockerfile and Makefile into the same directory.  Put your config into /docker/asterisk/config (in this example), your codecs into /docker/asterisk/codecs, and create a blank /docker/asterisk/logs .

You will also need a cdr-csv and cdr-custom directory in the logs dir if you want that functionality (Asterisk doesn’t create it).

Quickstart:  Just make all to do the whole lot and start it 🙂

  1. Run make build to construct the image.
  2. Run make install to configure firewalld rules and put the systemd unit in place
  3. Run make start to start the container
  4. Run make shell to get an Asterisk prompt inside the container

You can also do a docker logs asterisk to see what’s going on, and you can start/stop the container like a normal systemd service.

Good luck!

Cisco 7965 Phone Directory Setup

The Cisco 7965 has a Directory button, which can retrieve an XML document from a website and format that as a phone directory.  Setting it up is pretty simple.

In the SEPxxxx.cnf.xml for your phone, find the directoryURL tag and set that;

<directoryURL>http://intranet.example.com/directory.xml</directoryURL>

Then, create that file on your web server.  An example appears below;

<CiscoIPPhoneMenu>
<Title>Home Directory</Title>
<Prompt>Select a number</Prompt>
<MenuItem>
 <Name>Extension-511</Name>
 <URL>Dial:511</URL>
</MenuItem>
<MenuItem>
 <Name>Pizza Shop</Name>
 <URL>Dial:55511123</URL>
</MenuItem>
</CiscoIPPhoneMenu>

The items appear in the order they are listed.  I’m sure there’s more cleverness you can do with the directory, but that should get you started.