Kea DHCP Server 1.6 on Ubuntu 18.04 with Cisco IOS

Background

The Internet Services Consortium is a 501 non-profit (according to Wikipedia) that has developed and maintains both BIND DNS Server and ISC-DHCP-Server. Both of these open-source projects are used all over the world and are deployed on many, many servers. While ISC-DHCP-Server is handy and easy to configure, ISC has developed a new DHCP server called Kea that they say will eventually replace ISC-DHCP-Server. ISC claims Kea is better in nearly every way, so I’m going to give it a shot.

While all of ISC’s software is open-source, like many open-source maintainers they generate revenue by selling professional support for the software that they develop. Check out their support page for more info.

Topology

Pretty basic here. Kea will be installed on my Ubuntu 18.04 server and an IPv4 network is configured. We’ll see if we can use DHCP to get an IP address on my Cisco IOSv 15.6 router. I’m doing a packet capture in the middle which is always helpful for troubleshooting and verification.

Installation

Installation was pretty easy, provided you are ok with downloading a script from the internet and running it on your system. This method has some serious security concerns, so if you’re planning to run it on a production system you’ll want to probably take a more cautious approach. However my environment is entirely in GNS3 on a cloned Ubuntu VM so I can destroy the whole thing when I’m done. Please be careful out there, don’t talk to strangers.

It took me some time to find the official documentation, but as usual it was at readthedocs.io.

Running the ISC-Consortium’s bash script went pretty smooth for me, this process installs the package repository that ISC provides:

$ curl -1sLf \
>   'https://dl.cloudsmith.io/public/isc/kea-1-6/cfg/setup/bash.deb.sh' \
>   | sudo bash
Executing the  setup script for the 'isc/kea-1-6' repository ...

  OK: Checking for required executable 'curl' ...
  OK: Checking for required executable 'apt-get' ...
  OK: Detecting your OS distribution and release using system methods ...
 ^^^^: OS detected as: ubuntu 18.04 (bionic)
FAIL: Checking for apt dependency 'apt-transport-https' ...
  OK: Updating apt repository metadata cache ...
  OK: Attempting to install 'apt-transport-https' ...
  OK: Checking for apt dependency 'gnupg' ...
  OK: Importing 'isc/kea-1-6' repository GPG key into apt ...
  OK: Checking if upstream install config is OK ...
  OK: Installing 'isc/kea-1-6' repository via apt ...
  OK: Updating apt repository metadata cache ...
 ^^^^: The repository has been installed successfully - You're ready to rock!

$

Once the repository is successfully installed, you can now install the packages:

$apt install isc-kea-dhcp4-server isc-kea-dhcp6-server isc-kea-admin

Configuration

We’ll start with something really simple. I’ll set the range to be from 10.0.0.100 to 10.0.0.200, the default gateway to 10.0.0.1 (the Ubuntu box) and database set to “memfile” which for Kea means written to a csv file. This is pretty much the exact same functionality as ISC-DHCP-Server.

Ubuntu18.04-1


The default configuration file on my Ubuntu 18.04 box was located at /etc/kea/kea-dhcp4.conf. I deleted all the pre-added stuff and replaced with the following basic configuration:

{
# DHCPv4 configuration starts on the next line
"Dhcp4": {

# First we set up global values
    "valid-lifetime": 4000,
    "renew-timer": 1000,
    "rebind-timer": 2000,

# Next we set up the interfaces to be used by the server.
    "interfaces-config": {
        "interfaces": [ "eth0" ]
    },

# And we specify the type of lease database
    "lease-database": {
        "type": "memfile",
        "persist": true,
        "name": "/var/lib/kea/dhcp4.leases"
    },

# Finally, we list the subnets from which we will be leasing addresses.
    "subnet4": [
        {
            "subnet": "10.0.0.0/24",
            "pools": [
                {
                     "pool": "10.0.0.100 - 10.0.0.200"
                }],
                
            "option-data":[
                {
                     "name": "routers",
                     "data": "10.0.0.1"
                }]
            
            }
        ]
    }
}
# DHCPv4 configuration ends with the next line

And restart Kea with

$systemctl restart isc-kea-dhcp4-server

#and also check status:

$systemctl status isc-kea-dhcp4-server

If everything is good, you should see a green dot from systemd telling you Kea is up and running.


In this step it’s really easy to get hung up because of invalid JSON. Make sure you close all your {‘s and [‘s.

If all is well, Kea should be running and ready to hand out DHCP leases.

CiscoIOSv16.6(2)T-1

Configuration of a DHCP client on an IOS router is dead simple. On the interface just issue “ip address dhcp”. The interface section of your show run will look like this:

interface GigabitEthernet0/0
 ip address dhcp
 duplex auto
 speed auto

Verification

Ubuntu18.04-1

Since we’ve specified in our config that we want to store DHCP leases in /var/lib/kea/dhcp4.leases, my first step is to check to see if there’s a lease in there for the Cisco router.

$cat /var/lib/kea/dhcp4.leases
address,hwaddr,client_id,valid_lifetime,expire,subnet_id,fqdn_fwd,fqdn_rev,hostname,state,user_context
10.0.0.100,0c:b0:c4:ed:c8:00,00:63:69:73:63:6f:2d:30:63:62:30:2e:63:34:65:64:2e:63:38:30:30:2d:47:69:30:2f:30,4000,1583687165,1,1,1,router,0,

It’s a little hard to read, but it’s there.

CiscoIOSv16.6(2)T-1

On the Cisco device you’ll be looking for a log message like this:

*Mar  7 22:39:47.135: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0/0 assigned DHCP address 10.0.0.100, mask 255.255.255.0, hostname

You’ll also want to check IP interfaces for an IP address and issue “show ip interface brief”:

Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         10.0.0.100      YES DHCP   up                    up      
GigabitEthernet0/1         unassigned      YES unset  administratively down down    
GigabitEthernet0/2         unassigned      YES unset  administratively down down    
GigabitEthernet0/3         unassigned      YES unset  administratively down down

You’ll also want to check the packet capture for the standard DHCP set of 4 messages – Discover, Offer, Request, Acknowledge. Wireshark will recognize “bootp” as a filter:

Troubleshooting

The first sign of trouble will be that you’ll see nothing but “Discover” messages coming from the Cisco router, and no response from Kea. At that point, you’ll want to make sure Kea is running:

Then check the syslog file. On Ubuntu 18.04, this is in /var/log/syslog. Make sure to grep for “kea” because the file is usually pretty big, I used “cat /var/log/syslog | grep kea”:

In this particular instance, I had written invalid JSON in the Kea configuration file, so Kea failed to start.

I also had an instance where I was trying using a MySQL server to store leases, and my configuration wasn’t correct. Kea started but simply didn’t respond to DHCP requests. If you’re using SQL, make sure communication is happening between Kea and your SQL server.

Optional – MySQL Server for Leases storage

One of the options that Kea supports is storing lease data in a database. For this you can use a MySQL server (or others). ISC states in their documentation that the SQL database option is really for needs specific to large deployments, so it’s optional of course.

To install MySQL:

$sudo apt update
$sudo apt install mysql-server

You can check to make sure it’s running:

$systemctl status mysql

Status should be green:

Then you need to jump into mysql and create a database for Kea to use:

$mysql -u root -p
Enter password:
mysql>CREATE DATABASE james_kea;
Query OK, 1 row affected (0.00 sec)
mysql> CREATE USER 'james'@'localhost' IDENTIFIED BY 'james';
Query OK, 0 rows affected (0.01 sec)
mysql> GRANT ALL ON james_kea.* TO 'james'@'localhost';
Query OK, 0 rows affected (0.00 sec)
mysql> exit
Bye
$

Then initialize the Kea database with the kea-admin tool:

kea-admin db-init mysql -u james -p james -n james_kea

You’re going to need to change the “lease-database” section of /etc/kea/kea-dhcp4.conf to tell it to store leases in MySQL:

    "lease-database": {
    #    "type": "memfile", #I just commented these out.
    #    "persist": true,#I just commented these out.
    #    "name": "/var/lib/kea/dhcp4.leases" #I just commented these out.
        "type":"mysql",
        "name":"james_kea",
        "host":"127.0.0.1",
        "user":"james",
        "password":"james"
        "password":"james"
    },

Restart Kea for it to take effect.

As I mentioned before, if Kea can’t communicate with MySQL, it will start but simply not respond to DHCP requests. If you’re in that situation, make sure to check that your SQL and Kea configurations are correct.

You can take a look at the database via the mysql CLI to see if leases are being stored there:

$mysql -u root -p
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| james_kea          |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)
mysql> use james_kea
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+-------------------------------+
| Tables_in_james_kea           |
+-------------------------------+
| dhcp4_audit                   |
| dhcp4_audit_revision          |
| dhcp4_global_parameter        |
| dhcp4_global_parameter_server |
####EDITED FOR BREVITY####
| lease4                        |
| lease4_stat                   |
| lease6                        |
| lease6_stat                   |
| lease6_types                  |
| lease_hwaddr_source           |
| lease_state                   |
| logs                          |
| modification                  |
| parameter_data_type           |
| schema_version                |
+-------------------------------+
mysql> SELECT * FROM lease4;
+-----------+--------+-----------------------------+----------------+---------------------+-----------+----------+----------+----------+-------+--------------+
| address   | hwaddr | client_id                   | valid_lifetime | expire              | subnet_id | fqdn_fwd | fqdn_rev | hostname | state | user_context |
+-----------+--------+-----------------------------+----------------+---------------------+-----------+----------+----------+----------+-------+--------------+
| 167772260 | 
              ����      |  cisco-0cb0.c4ed.c800-Gi0/0 |           4000 | 2020-03-09 02:16:09 |         1 |        1 |        1 | router   |     0 | NULL         |
+-----------+--------+-----------------------------+----------------+---------------------+-----------+----------+----------+----------+-------+--------------+
1 row in set (0.00 sec)

mysql> 

It shows up better on a terminal with a lot of width, but there is lease data stored for my Cisco router. Success.

Optional – MySQL Server/REST API for Configuration Storage

I was really hoping to get to test this, however I learned this functionality is provided through external libraries, or “hooks” that the ISC team offers through a premium package.

The “Configuration Backend”, as they call it, offers the ability to store configurations for networks, some options, and other stuff in MySQL and not a JSON config file. Changes to these configurations can be made through REST calls to the kea-ctrl-agent which processes and strips away the HTTP headers and forwards the JSON data to Kea. Essentially, you could make configuration changes from a script or program on the fly without having to restart or reconfigure the server. Pretty cool, I can see why they charge for it.

As of this writing in March of 2020, the ISC website says you need a premium support subscription for the premium hooks package.

DMVPN (Sans-Ipsec) on Linux/Cisco with Free Range Routing

Background

Free Range Routing is a really cool fork of the Quagga project by Cumulus Linux, both are implementations of a number of network routing protocols for Linux. While Quagga implemented the classic protocols such as OSPF, BGP, RIP, and PIMD that have been in use for decades, FRR implements some new ones and adds features to the old protocols.

Free Range Routing is a bit tricky to build from source at the moment (February 2020) due to some newer packages such as libyang not being readily available in major distributions such as Ubuntu. However Ubuntu’s Snap technology makes installing a very recent version of Free Range Routing, well, a snap. I have another post that shows you how to install it and run EIGRP on Linux which is pretty cool because it’s Cisco proprietary (yes, Cisco still owns it despite publishing an RFC).

Another protcol that Free Range Routing implements is NHRP, which is the key component in another Cisco-originated technology, Dynamic Multipoint Virtual Private Network, or DMVPN for short. NHRP serves as the protocol that determines GRE tunnel endpoints.

For this post, I’m going to skip the encryption part of DMVPN that IPsec provides. It’s done with Strongswan on Linux, and it’s tricky to get right and I’ll cover it in another post. But you can get a nice unencrypted overlay network running with NHRP and GRE before you ever start encrypting. I don’t think there’s an official term for DMVPN without IPsec, but I’ve heard networking folk refer to it as “naked” DMVPN. Obviously you don’t want to do this on the Internet (despite the “Internet” node in my topology), but there are some viable use cases for naked DMVPN on a private network and I’ve personally seen it on production networks. Free Range Routing’s NHRP implementation is still experimental, so please be careful.

Topology

I don’t like how cluttered this drawing is, but I’m pretty sure all the info on there is relevant to the lab. The idea here is to get PC’s 1, 2 and 3 to be able to ping each other. The 2 Ubuntu routers and 1 Cisco router can’t establish routing with “The Internets” so they’ll need a tunneling mechanism to learn about the networks of the other routers that have PC’s connected. First I’ll get NHRP connected and get GRE tunnels working. Then I’ll run BGP to advertise 192.168.0.0/24, 172.16.0.0/24 and 172.17.0.0/24 over the “overlay” GRE network.

Configuration

This post assumes the basic IP connectivity between the Ubuntu and Cisco routers has been set up, as well as LAN connectivity for the PC’s. Specifically, networks 11.0.0.0/30, 22.0.0.0/30, 33.0.0.0/30, 192.168.0.0/24, 172.16.0.0/24, and 172.17.0.0/24 have all been configured. With all that boring stuff done, we’ll jump into more interesting things.

Ubuntu18.04-FRR-1

This is the DMVPN hub. First order of business is to create a GRE tunnel interface:

$ip tunnel add james_gre mode gre key 42 ttl 64 dev eth0
$ip addr add 10.0.0.1/32 dev james_gre
$ip link set james_gre up

You might be wondering why the GRE IP address is 10.0.0.1/32 and not 10.0.0.1/24. Well apparently it has something to do with the way the good folks at Cumulus Linux implemented it. I’m sure they had their reasons, and they outline a couple in their documentation and on their Github NHRP readme page. That’s just the way it’s done with FRR, so I don’t argue.

Next we need to jump into the FRR CLI and configure NHRP. Remember to type the full path as it didn’t get added to $PATH when FRR was installed (or you can add it to your $PATH, or cd to /snap/bin):

$/snap/bin/frr.vtysh
Hello, this is FRRouting (version 7.2.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

# conf t
(config)#int james_gre
(config-if)#ip nhrp network-id 1
(config-if)#ip nhrp nhs dynamic nbma 11.0.0.1
(config-if)#ip nhrp registration no-unique
(config-if)#exit
(config)#

I’ve skipped setting up “NHRP redirect” (direct spoke-to-spoke communication) as I was unable to get it working in my lab between a Cisco spoke and Ubuntu spoke. I can make it work between two Ubuntu spokes or two Cisco spokes but that’s not really much fun. As I said this implementation is still a work in progress.

Next I’ll set up BGP routing to advertise the networks where the PC’s are connected. I’m using BGP because it lends itself to DMVPN for a number of reasons and also FRR’s NHRP implementation doesn’t support multicast yet, so no EIGRP or OSPF. The Hub will serve as a BGP “route reflector”, which basically means you can avoid a full mesh of BGP neighborships in iBGP (BGP where the AS is all the same). Sounds fancy but it actually simplifies things here, I promise.

(config)#router bgp 1
(config-router)#neighbor 10.0.0.2 remote-as 1
(config-router)#neighbor 10.0.0.3 remote-as 1
(config-router)#address-family ipv4 unicast
(config-router-af)#network 192.168.0.0/24
(config-router-af)#neighbor 10.0.0.2 route-reflector-client
(config-router-af)#neighbor 10.0.0.3 route-reflector-client
#

Ubuntu18.04-FRR-2

The Ubuntu spoke is pretty similar, except without any route reflector stuff.

$ip tunnel add james_gre mode gre key 42 ttl 64 dev eth0
$ip addr add 10.0.0.2/32 dev james_gre
$ip link set james_gre up
$/snap/bin/frr.vtysh
Hello, this is FRRouting (version 7.2.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

# conf t
(config)#int james_gre
(config-if)#ip nhrp network-id 1
(config-if)#ip nhrp nhs dynamic nbma 11.0.0.1
(config-if)#ip nhrp registration no-unique
(config-if)#exit
(config)#ip route 10.0.0.0/24 10.0.0.1
(config)#router bgp 1
(config-router)#neighbor 10.0.0.1 remote-as 1
(config-router)#address-family ipv4 unicast
(config-router-af)#network 172.16.0.0/24
(config-router-af)#exit
#

You may have noticed an additional command – I added a static route via “10.0.0.0/24 10.0.0.1”. This was for compatibility with the Cisco device. I could avoid this step and use NHRP redirect with another Ubuntu spoke and they would communicate directly. However with a Cisco spoke in play I had to add the route and all traffic goes through the hub. Maybe it’ll be better next release. Or if there’s a configuration I’m missing, please comment or shoot me an email, I’d love to know because I practically broke my keyboard trying different configurations.

CiscoIOSv15.6(2)T-1

This is the tunnel and BGP configuration for my Cisco IOSv router:

interface Tunnel0
 ip address 10.0.0.3 255.255.255.0
 no ip redirects
 ip nhrp network-id 1
 ip nhrp nhs dynamic nbma 11.0.0.1
 ip nhrp registration no-unique
 ip nhrp shortcut
 tunnel source GigabitEthernet0/0
 tunnel mode gre multipoint
 tunnel key 42
!
router bgp 1
 bgp log-neighbor-changes
 network 172.17.0.0 mask 255.255.255.0
 neighbor 10.0.0.1 remote-as 1
!

You probably noticed that the IP address on the tunnel interface is not /32 like the Ubuntu routers, it’s a /24 which is ultimately the size of the DMVPN network. That’s the way Cisco does it, so I don’t argue.

Verification

You can run some useful show commands from the FRR CLI on the Ubuntu routers as well as the Cisco CLI. “show ip route” is really useful, you’ll notice “N” routes on the Ubuntu routers which is how FRR NHRP advertises other spoke IP addresses (Read the FRR docs for more detail):

# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

K>* 0.0.0.0/0 [0/0] via 11.0.0.2, eth0, 1d01h01m
C>* 10.0.0.1/32 is directly connected, james_gre, 1d00h57m
N>* 10.0.0.2/32 [10/0] is directly connected, james_gre, 02:56:29
N>* 10.0.0.3/32 [10/0] is directly connected, james_gre, 12:37:57
C>* 11.0.0.0/30 is directly connected, eth0, 1d01h02m
B>  172.16.0.0/24 [200/0] via 10.0.0.2 (recursive), 01:17:14
  *                         via 10.0.0.2, james_gre onlink, 01:17:14
B>  172.17.0.0/24 [200/0] via 10.0.0.3 (recursive), 01:16:53
  *                         via 10.0.0.3, james_gre onlink, 01:16:53
C>* 192.168.0.0/24 is directly connected, eth1, 12:10:26
#

You can also run “show dmvpn” which should show the other endpoints:

# show dmvpn
Src                      Dst                      Flags  SAs  Identity                <br>
11.0.0.1                 22.0.0.1                 n      0                            <br>
11.0.0.1                 33.0.0.1                 n      0          

And a good old fashioned ping should confirm everything for you. Here I’m pinging from the PC behind the Ubuntu spoke to the PC behind the Cisco spoke:

PC2> ping 172.17.0.2
84 bytes from 172.17.0.2 icmp_seq=1 ttl=62 time=3.523 ms
84 bytes from 172.17.0.2 icmp_seq=2 ttl=62 time=3.828 ms

Troubleshooting

You can apparently send NHRP debugs to syslog, but I didn’t find them particularly helpful when I was troubleshooting. I found a packet capture to be the most useful. The the key is to get NHRP going, in a packet capture you should see a NHRP registration request and reply with Code=Success. If you see registration requests with no request at all, you have a problem. On my cisco I originally forgot to put in the tunnel key command, and it kept sending a request and getting an ICMP error response. Don’t forget your tunnel key.

DMVPN between Ubuntu and Cisco, pretty cool right?