Netflow with ELK Stack and OpenWRT

Now we’re getting into some pretty serious magic.  This post will outline how to put together OpenWRT and ELK Stack to collect network utilization statistics with Netflow.  From there, we can use Kibana to generate visualizations of traffic data and flows and whatever else you want to leverage with the power of Elasticsearch.

I’m using a virtualized router instance running OpenWRT 15.05.1 (Chaos Calmer) on KVM with the Generic x86 build.  Using a hardware router is still doable, but you’ll need to be careful about CPU utilization of the Netflow exporter.  Setting this up will require a number of components, which we’ll go through now.

You will need an OpenWRT box of some description, and an ELK Stack already configured and running.

OpenWRT Setup

You’ll need to install softflowd, which is as easy as;

opkg update
opkg install softflowd

Then edit /etc/config/softflowd and set the destination for flows to go to something like;

option host_port 'netflow.localdomain:9995'

Start up the Softflow exporter with /etc/init.d/softflowd start and it should be working.

Note, default config will be using Netflow version 5.  Let that stand for now.  Also, leave the default interface on br-lan – that way it’ll catch flows for all traffic reaching the router.

Logstash Configuration

If you’re using the ELK Stack Docker project like me, you’ll need to set up the Docker container to also listen on port 9995 UDP.  At any rate, you need to edit your logstash.conf so that you have the following input receiver;

# Netflow receiver
input {
  udp {
    port => 9995
    type => netflow
    codec => netflow
  }
}

This is an extremely simple receiver which takes in Netflow data on port 9995, sets the type to netflow and then processes it with the built-in Netflow codec.

In your output transmitter, you’ll then want something like this example;

output {
        if ( [type] == "netflow" ) {
                elasticsearch {
                        hosts => "elasticsearch:9200"
                        index => "logstash-netflow-%{host}-%{+YYYY.MM.dd}"
                }
        } else {
                elasticsearch {
                        hosts => "elasticsearch:9200"
                        index => "logstash-%{type}-%{+YYYY.MM.dd}"
                }
        }
}

What this does is pretty straightforward.  Everything gets sent to the Elasticsearch engine at elasticsearch:9200.  But, messages with the type of netflow get pushed into an index that has the IP address that the flow was collected from in it (this will probably be your router).

Restart Logstash and you should start getting flows in within a few minutes.

Kibana Setup

From there, just go into Kibana and add a new index pattern for logstash-netflow-*.  You can then visualize / search all your Netflow data to your heart’s content.

Nice!

SSH Configuration on OpenWRT

If you’ve configured Dropbear (the SSH server) for OpenWRT so that it has a secondary listener for your WAN port (you may want to do this if you want the WAN SSH listener on a different port from the default), then you’ve probably noticed that it doesn’t come up on its own after your WAN link drops.

There’s a really easy solution to this.  Configure hotplug.d so that when your WAN interface bounces, dropbear gets restarted!  Put this into /etc/hotplug.d/iface/40-dropbear ;

#!/bin/sh

if [ "$INTERFACE" = "wan" ] && [ "$ACTION" = "ifup" ]
then
 /etc/init.d/dropbear restart
fi

This tip was found at the bottom of the documentation for Dropbear listed above.

Automating OpenVPN Push Routes

Using OpenVPN with OpenWRT is a common solution for pushing routes out various egress points from your network.  However, maintaining a list of routes in your OpenVPN config is a nuisance, plus if the mapping from DNS name to IP address changes, they spontaneously break.  Additionally, you ideally want to use the DNS resolution of a name from the remote end of your VPN tunnel, not from the local end.

Enter the script I’ve put in my GoogleCode Repository. This script, when provided a config file like the following;

# Example config file. Don't really use this.

config router myrouter.local

# vpn definitions
config addvpn myfirstvpn dns 10.0.1.1
config addvpn mysecondvpn dns 10.0.2.1
config defaultvpn myfirstvpn # inline comment

# a dns example route
route dns www.whatismyip.com dnsfailok target mysecondvpn

# an ip example route
route 8.8.8.8

# a network example route
route 10.1.1.0 netmask 255.255.255.0

When run, the script will automatically configure routes in the appropriate sections of your OpenVPN config, insert hosts files entries pinning the DNS names to the remote-end resolutions of those names, and even upload the files and reload the relevant services on your OpenWRT router.  Magic!

You can’t just use this as it is though, you need to prepare some files on your OpenWRT box, and you need to have ssh enabled with certificate auth for the machine you’re running the pushroutes script from.  Read the top of the script to understand what it requires, and make sure you back up the relevant files from your OpenWRT router before running it.

The usual disclaimers apply to this kind of thing.  No warranties express or implied, author takes no responsibility for any damages, use at your own risk etc.

Good luck, I’m sure you can think of some great uses for this kind of automation.

DNSMasq Selective Forwarding

Now, if you’re using an OpenVPN selective routing tunnel like I’ve been discussing to push specific subnets through a tunnel, then you probably also have good reason to want to force specific DNS domains to resolve through a DNS server that is also on the the other end of that tunnel (eg, an internal network).

If your DNS server is on your OpenWRT router and is running dnsmasq, this is really easy.  First on the remote end of the tunnel you’ll need a DNS forwarder set up;

yum install dnsmasq
chkconfig dnsmasq on
service start dnsmasq

Make sure that’s suitably firewalled off so it’s only accessible from the tun interface, otherwise you’ll find yourself being used for DNS-based DDOS.

Next, you’ll need to edit your dnsmasq config in /etc/dnsmasq.conf and add the following (this example redirects anything in the google.com domain to resolve on the other end of the tunnel);

server=/google.com/10.0.1.1

I’m assuming that the peer address of the remote end of the tunnel is 10.0.1.1 .  It will be the .1 in whatever subnet that VPN server is pushing.  Restart dnsmasq after doing this.

Now, anything that is inside google.com (eg, ssl.google.com, www.google.com and just plain google.com) will be resolved using the DNS server that’s responding at 10.0.1.1, which is the remote end of your tunnel.

The reason why you want a forwarder set up on the remote end of the tunnel is so that you have a unique per-tunnel address to use in the dnsmasq config.  If you already have unique per-tunnel DNS addresses to use, nothing’s stopping you just using those and skipping the installation of dnsmasq on the remote end.

OpenVPN Routing from Server to Client

There’s a lot of guides about how to use OpenVPN to push arbitrary routes (usually to defeat geolocking) from an OpenVPN client to a server.  However, my requirements are actually backwards.  I need to be able to push routes from my server to a client (since the ‘server’ is my home router).  This requires a different rule set from normal.

Masquerading

Firstly, the machine that has is going to function as the egress point to the Internet has to be configured to allow IPv4 forwarding and also to allow masquerading (so that packets intended to be forwarded from the internal network to the Internet can be re-tagged with the egress point external IP address).

In /etc/sysctl.conf, set net.ipv4.ip_forward to 1.  Then, you’ll need the following iptables rules (eth0 is the egress interface, tun0 is the internal interface);

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -i eth0 -o tun0 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i tun0 -o eth0 -j ACCEPT
iptables -A FORWARD -j LOG
iptables -A FORWARD -j DROP

The first rule causes traffic outbound on the egress interface to be masqueraded (NAT).  The second rule causes inbound traffic going from the egress interface to the internal interface to be accepted if it’s part of an established or related connection (ie, packets coming back).  The third rule causes packets destined to be forwarded from the internal interface to the egress interface to be accepted.  And the last two rules log anything else and drop them.

OpenVPN Server Configuration

Now, the OpenVPN server needs to be told what routes should be directed into the tun adapter.  As an example, we’ll use whatismyip.com .  In /etc/config/openvpn, add the following;

list route '141.101.120.14 255.255.255.255'
list route '141.101.120.15 255.255.255.255'

When OpenVPN is restarted, it will automatically put the correct entries in your router’s routing table to direct traffic to those IPs out your tun adapter.  However, that’s not all.

OpenVPN Client Configuration (on server)

If OpenVPN receives traffic on the tun adapter for those IPs, it doesn’t know which connected client should receive the packets and so it drops them.  You will also need iroutes for those networks in the client configuration directives for your client;

iroute 141.101.120.14 255.255.255.255
iroute 141.101.120.15 255.255.255.255

Right, that’s it.  Restart OpenVPN and connect to it.

Testing

Try and ping one of the routes you’ve added.  If it works, great!  If not, the first thing to check is that the traffic is actually getting routed.  Examine the router’s routing table with ‘route’ and see if the route is listed.

Assuming it is, on your client end, run the following;

tcpdump -i tun0

When trying to ping, you should see packets land.  If you do, this tells you that packets are hitting your router, being redirected into OpenVPN, OpenVPN is passing them down the tunnel and they’re breaking out at the tun interface on your client.  Check your firewall log on the client and make sure your firewall rules are fine.

If you don’t see the packets landing on the tun interface, check logread on your router.  If there are complaints about packets being dropped, examine /tmp/openvpn.status and make sure that the route is listed in the OpenVPN routing table.

Anyway, good luck.  I’m sure you can come up with some creative ways of having your routing come out a different egress point than usual 🙂

OpenVPN – Unidentified Network issue

Discovered a little wrinkle in Windows 7’s Network Identification feature.  If you’re pushing an OpenVPN tunnel to a machine and not substituting the default gateway (because, for example, you want a split tunnel) with the VPN’s gateway, then Windows just consistently won’t identify the network, which means you’re stuck with the “Public” firewall profile.

Fortunately the solution’s pretty easy.  In the client config directives for your client, you define a new default route with a very high metric pointing to the peer address for the client, eg;

ifconfig-push 10.0.2.101 10.0.2.102
push "route 0.0.0.0 0.0.0.0 10.0.2.102 500"

So now, when that client connects, it will have the IP address 10.0.2.101 and a peer address of 10.0.2.102 .  We define a new default route going to 10.0.2.102 with a very high metric.  This ensures that this route doesn’t get used unless the real default route is broken.

Connect up, and voila!  Windows identifies the network, and you can give it a name and change its type from ‘Public’ to ‘Home’.

Building a VPN with OpenVPN and OpenWRT

I wound out replacing my existing router (which had a buggy NAT issue) with a TP-Link TL-WR1043ND running OpenWRT.  It was pretty damned easy to get it all running and set it up as an in-place upgrade.  However, I wanted more out of it.

What I want to do is to establish a VPN tunnel such that my VPS has some (highly restricted) access to my local network, and my local network has (nearly) unrestricted access to the VPS.  I also want to have other devices (my phone) able to connect to my local network using VPN and have unrestricted access.  And lastly, I want to do this with certificates (and not shared secrets).  To do this, I used OpenVPN.

Desired Topology

  • 10.0.0.0/24 – Internal LAN
  • 10.0.2.0/24 – Trusted VPN
  • 10.0.3.0/24 – Untrusted (DMZ) VPN

OpenWRT Configuration

Package Installation and TUN Configuration

First, run the following to install the required packages;

opkg update
opkg install openvpn-openssl
opkg install openvpn-easy-rsa

Once that’s done, edit /etc/config/network and add a declaration of a new TUN interface;

config interface 'vpn'
    option proto 'none'
    option ifname 'tun0'

Reboot your router, and you’ll find a new interface tun0 waiting.  Now you need to set up your PKI infrastructure and generate some certs.

Configure PKI

Follow the installation instructions for easy-rsa.  Once that’s done, you will have a functional self-signed CA.  Go and generate some certificates like this;

build-key-server server
build-key trustedclient
build-key dmzclient

Take server.crt and server.key and copy them into the OpenVPN configuration;

cp /etc/easy-rsa/server.* /etc/openvpn/
mkdir /etc/openvpn/clients/

You’ve now got a basic PKI setup, and two client certificates ready to go, along with the server certificate for OpenVPN.

Configure OpenVPN Server

Edit /etc/config/openvpn like this;

package openvpn

config openvpn trusted_vpn
        option enabled          1
        option port             1194
        option proto            udp
        option dev              tun
        option ca               /etc/easy-rsa/keys/ca.crt
        option cert             /etc/openvpn/server.crt
        option key              /etc/openvpn/server.key
        option dh               /etc/easy-rsa/keys/dh2048.pem
        option keepalive        '10 120'
        option comp_lzo         yes
        option persist_key      1
        option persist_tun      1
        option verb             3
        option status           /tmp/openvpn.status
        option log              /tmp/openvpn.log
        option ccd_exclusive    1
        option client_config_dir        /etc/openvpn/clients
        option server           '10.0.2.0 255.255.255.0'
        option route            '10.0.3.0 255.255.255.0'
        list push               "route 10.0.0.0 255.255.255.0"
        list push               "dhcp-option DOMAIN localdomain.local"
        list push               "dhcp-option DNS 10.0.0.254"

You should disable the log once you’ve got everything working.  What this does is the following;

  • Clients must have a matching client config entry in /etc/openvpn/clients to be able to connect
  • The server by default uses 10.0.2.0/24 for connecting clients, but also has a route for 10.0.3.0/24 down the TUN interface
  • The server pushes a route to the client for 10.0.0.0/24, along with a couple of settings that are used by Windows clients (setting local domain and DNS servers to use)

For your trusted client, create a file /etc/openvpn/clients/trustedclient which has this line in it;

ifconfig-push 10.0.2.101 10.0.2.102

This causes the trusted client to always get the IP address 10.0.2.101.  Note, as per the OpenVPN documentation, in order for the ifconfig-push’ed addresses to work with WIndows clients properly, they must come from the set;

[  1,  2] [  5,  6] [  9, 10] [ 13, 14] [ 17, 18]
[ 21, 22] [ 25, 26] [ 29, 30] [ 33, 34] [ 37, 38]
[ 41, 42] [ 45, 46] [ 49, 50] [ 53, 54] [ 57, 58]
[ 61, 62] [ 65, 66] [ 69, 70] [ 73, 74] [ 77, 78]
[ 81, 82] [ 85, 86] [ 89, 90] [ 93, 94] [ 97, 98]
[101,102] [105,106] [109,110] [113,114] [117,118]
[121,122] [125,126] [129,130] [133,134] [137,138]
[141,142] [145,146] [149,150] [153,154] [157,158]
[161,162] [165,166] [169,170] [173,174] [177,178]
[181,182] [185,186] [189,190] [193,194] [197,198]
[201,202] [205,206] [209,210] [213,214] [217,218]
[221,222] [225,226] [229,230] [233,234] [237,238]
[241,242] [245,246] [249,250] [253,254]

Next, we repeat this with the DMZ client, but with a small change – we’ll also push the route for the trusted network.  This is required because otherwise the router itself cannot contact anything in the DMZ due to the router’s interface IP being in the 10.0.2.0/24 range.

Create /etc/openvpn/clients/dmzclient as follows;

ifconfig-push 10.0.3.101 10.0.3.102
push route "10.0.2.0 255.255.255.0"

That concludes the OpenVPN server config.  Now for the firewall.

Firewall Configuration

OpenVPN Access Port

Edit /etc/config/firewall .  Up fairly high, amongst any other port forwards, add the following;

config rule
    option name 'OpenVPN-Access'
    option src wan
    option proto udp
    option dest_port 1194
    option family ipv4
    option target ACCEPT

This allows your VPN to be accessible from the ‘net.

Full-Access VPN Zone

Next, we’ll create a new zone for the full-access VPN, but we’ll specify that it only applies to a specific subnet;

config zone
        option name             vpn
        option input            ACCEPT
        option forward          REJECT
        option output           ACCEPT
        list network            vpn
        list subnet             '10.0.2.0/24'
        option masq             0
        option mtu_fix          1

config forwarding
        option dest             lan
        option src              vpn

config forwarding
        option dest             vpn
        option src              lan

DMZ VPN Zone

And lastly we’ll define a new zone for the DMZ VPN along with allowed traffic.  Note that this does not specify a subnet, that way any traffic that comes in on the tun network that does not match the trusted VPN falls through to the DMZ zone.

config zone
        option name             dmz
        option input            REJECT
        option forward          REJECT
        option output           ACCEPT
        list network            vpn
        option masq             0
        option mtu_fix          1

config rule
        option name             'http-Linkage'
        option src              dmz
        option dest             lan
        option proto            tcp
        option src_ip           10.0.3.101
        option dest_port        80
        option dest_ip          10.0.0.15
        option family           ipv4
        option target           ACCEPT

config rule
        option name             Allow-DMZ-Ping
        option src              dmz
        option dest             lan
        option proto            icmp
        option icmp_type        echo-request
        option family           ipv4
        option target           ACCEPT

config forwarding
        option dest             dmz
        option src              lan

Notably, there is only a forward here from lan to dmz, that way we have to specifically allow what traffic we want to be passed from dmz to lan.  Here, I allow the DMZ machine to ping into my local network, and to connect to 10.0.0.15 on port tcp/80 only.

OpenVPN Server Summary

Essentially, what we’ve done is create two network zones which are pushed out by OpenVPN.  The firewall controls access between them.  Which user certificate lands in which zone is determined by the IP they’re assigned by OpenVPN on connect, which is defined by the configuration file in /etc/openvpn/clients.  All clients must have a config file waiting for them in this setup.

IMPORTANT – Don’t forget to add /etc/openvpn and /etc/easy-rsa to your /etc/sysupgrade.conf, otherwise you’ll lose all this on upgrading your router.  That would be unfortunate.

OpenVPN Client Setup (DMZ Client)

You will require the following components to configure your client;

  • /etc/easy-rsa/keys/ca.crt
  • /etc/easy-rsa/keys/dmzclient.crt
  • /etc/easy-rsa/keys/dmzclient.key

Download those off your router and store them somewhere.  Then, you’ll need to create a client.ovpn file in the same folder as follows;

client
dev tun
remote yourvpnserverhostname 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ns-cert-type server
ca /etc/openvpn/ca.crt
cert /etc/openvpn/dmzclient.crt
key /etc/openvpn/dmzclient.key
comp-lzo
route-nopull
route 10.0.0.0 255.255.255.0
route 10.0.2.0 255.255.255.0

In my case, the files are in /etc/openvpn, as the config shows.  Notably, this config also does not pull route data from the OpenVPN server but instead sets it by itself.  Then, start your VPN with

openvpn --config client.ovpn

And it should all work!  If not, consult the logs, and good luck!