DMVPN (Sans-Ipsec) on Linux/Cisco with Free Range Routing

Background

Free Range Routing is a really cool fork of the Quagga project by Cumulus Linux, both are implementations of a number of network routing protocols for Linux. While Quagga implemented the classic protocols such as OSPF, BGP, RIP, and PIMD that have been in use for decades, FRR implements some new ones and adds features to the old protocols.

Free Range Routing is a bit tricky to build from source at the moment (February 2020) due to some newer packages such as libyang not being readily available in major distributions such as Ubuntu. However Ubuntu’s Snap technology makes installing a very recent version of Free Range Routing, well, a snap. I have another post that shows you how to install it and run EIGRP on Linux which is pretty cool because it’s Cisco proprietary (yes, Cisco still owns it despite publishing an RFC).

Another protcol that Free Range Routing implements is NHRP, which is the key component in another Cisco-originated technology, Dynamic Multipoint Virtual Private Network, or DMVPN for short. NHRP serves as the protocol that determines GRE tunnel endpoints.

For this post, I’m going to skip the encryption part of DMVPN that IPsec provides. It’s done with Strongswan on Linux, and it’s tricky to get right and I’ll cover it in another post. But you can get a nice unencrypted overlay network running with NHRP and GRE before you ever start encrypting. I don’t think there’s an official term for DMVPN without IPsec, but I’ve heard networking folk refer to it as “naked” DMVPN. Obviously you don’t want to do this on the Internet (despite the “Internet” node in my topology), but there are some viable use cases for naked DMVPN on a private network and I’ve personally seen it on production networks. Free Range Routing’s NHRP implementation is still experimental, so please be careful.

Topology

I don’t like how cluttered this drawing is, but I’m pretty sure all the info on there is relevant to the lab. The idea here is to get PC’s 1, 2 and 3 to be able to ping each other. The 2 Ubuntu routers and 1 Cisco router can’t establish routing with “The Internets” so they’ll need a tunneling mechanism to learn about the networks of the other routers that have PC’s connected. First I’ll get NHRP connected and get GRE tunnels working. Then I’ll run BGP to advertise 192.168.0.0/24, 172.16.0.0/24 and 172.17.0.0/24 over the “overlay” GRE network.

Configuration

This post assumes the basic IP connectivity between the Ubuntu and Cisco routers has been set up, as well as LAN connectivity for the PC’s. Specifically, networks 11.0.0.0/30, 22.0.0.0/30, 33.0.0.0/30, 192.168.0.0/24, 172.16.0.0/24, and 172.17.0.0/24 have all been configured. With all that boring stuff done, we’ll jump into more interesting things.

Ubuntu18.04-FRR-1

This is the DMVPN hub. First order of business is to create a GRE tunnel interface:

$ip tunnel add james_gre mode gre key 42 ttl 64 dev eth0
$ip addr add 10.0.0.1/32 dev james_gre
$ip link set james_gre up

You might be wondering why the GRE IP address is 10.0.0.1/32 and not 10.0.0.1/24. Well apparently it has something to do with the way the good folks at Cumulus Linux implemented it. I’m sure they had their reasons, and they outline a couple in their documentation and on their Github NHRP readme page. That’s just the way it’s done with FRR, so I don’t argue.

Next we need to jump into the FRR CLI and configure NHRP. Remember to type the full path as it didn’t get added to $PATH when FRR was installed (or you can add it to your $PATH, or cd to /snap/bin):

$/snap/bin/frr.vtysh
Hello, this is FRRouting (version 7.2.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

# conf t
(config)#int james_gre
(config-if)#ip nhrp network-id 1
(config-if)#ip nhrp nhs dynamic nbma 11.0.0.1
(config-if)#ip nhrp registration no-unique
(config-if)#exit
(config)#

I’ve skipped setting up “NHRP redirect” (direct spoke-to-spoke communication) as I was unable to get it working in my lab between a Cisco spoke and Ubuntu spoke. I can make it work between two Ubuntu spokes or two Cisco spokes but that’s not really much fun. As I said this implementation is still a work in progress.

Next I’ll set up BGP routing to advertise the networks where the PC’s are connected. I’m using BGP because it lends itself to DMVPN for a number of reasons and also FRR’s NHRP implementation doesn’t support multicast yet, so no EIGRP or OSPF. The Hub will serve as a BGP “route reflector”, which basically means you can avoid a full mesh of BGP neighborships in iBGP (BGP where the AS is all the same). Sounds fancy but it actually simplifies things here, I promise.

(config)#router bgp 1
(config-router)#neighbor 10.0.0.2 remote-as 1
(config-router)#neighbor 10.0.0.3 remote-as 1
(config-router)#address-family ipv4 unicast
(config-router-af)#network 192.168.0.0/24
(config-router-af)#neighbor 10.0.0.2 route-reflector-client
(config-router-af)#neighbor 10.0.0.3 route-reflector-client
#

Ubuntu18.04-FRR-2

The Ubuntu spoke is pretty similar, except without any route reflector stuff.

$ip tunnel add james_gre mode gre key 42 ttl 64 dev eth0
$ip addr add 10.0.0.2/32 dev james_gre
$ip link set james_gre up
$/snap/bin/frr.vtysh
Hello, this is FRRouting (version 7.2.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

# conf t
(config)#int james_gre
(config-if)#ip nhrp network-id 1
(config-if)#ip nhrp nhs dynamic nbma 11.0.0.1
(config-if)#ip nhrp registration no-unique
(config-if)#exit
(config)#ip route 10.0.0.0/24 10.0.0.1
(config)#router bgp 1
(config-router)#neighbor 10.0.0.1 remote-as 1
(config-router)#address-family ipv4 unicast
(config-router-af)#network 172.16.0.0/24
(config-router-af)#exit
#

You may have noticed an additional command – I added a static route via “10.0.0.0/24 10.0.0.1”. This was for compatibility with the Cisco device. I could avoid this step and use NHRP redirect with another Ubuntu spoke and they would communicate directly. However with a Cisco spoke in play I had to add the route and all traffic goes through the hub. Maybe it’ll be better next release. Or if there’s a configuration I’m missing, please comment or shoot me an email, I’d love to know because I practically broke my keyboard trying different configurations.

CiscoIOSv15.6(2)T-1

This is the tunnel and BGP configuration for my Cisco IOSv router:

interface Tunnel0
 ip address 10.0.0.3 255.255.255.0
 no ip redirects
 ip nhrp network-id 1
 ip nhrp nhs dynamic nbma 11.0.0.1
 ip nhrp registration no-unique
 ip nhrp shortcut
 tunnel source GigabitEthernet0/0
 tunnel mode gre multipoint
 tunnel key 42
!
router bgp 1
 bgp log-neighbor-changes
 network 172.17.0.0 mask 255.255.255.0
 neighbor 10.0.0.1 remote-as 1
!

You probably noticed that the IP address on the tunnel interface is not /32 like the Ubuntu routers, it’s a /24 which is ultimately the size of the DMVPN network. That’s the way Cisco does it, so I don’t argue.

Verification

You can run some useful show commands from the FRR CLI on the Ubuntu routers as well as the Cisco CLI. “show ip route” is really useful, you’ll notice “N” routes on the Ubuntu routers which is how FRR NHRP advertises other spoke IP addresses (Read the FRR docs for more detail):

# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

K>* 0.0.0.0/0 [0/0] via 11.0.0.2, eth0, 1d01h01m
C>* 10.0.0.1/32 is directly connected, james_gre, 1d00h57m
N>* 10.0.0.2/32 [10/0] is directly connected, james_gre, 02:56:29
N>* 10.0.0.3/32 [10/0] is directly connected, james_gre, 12:37:57
C>* 11.0.0.0/30 is directly connected, eth0, 1d01h02m
B>  172.16.0.0/24 [200/0] via 10.0.0.2 (recursive), 01:17:14
  *                         via 10.0.0.2, james_gre onlink, 01:17:14
B>  172.17.0.0/24 [200/0] via 10.0.0.3 (recursive), 01:16:53
  *                         via 10.0.0.3, james_gre onlink, 01:16:53
C>* 192.168.0.0/24 is directly connected, eth1, 12:10:26
#

You can also run “show dmvpn” which should show the other endpoints:

# show dmvpn
Src                      Dst                      Flags  SAs  Identity                <br>
11.0.0.1                 22.0.0.1                 n      0                            <br>
11.0.0.1                 33.0.0.1                 n      0          

And a good old fashioned ping should confirm everything for you. Here I’m pinging from the PC behind the Ubuntu spoke to the PC behind the Cisco spoke:

PC2> ping 172.17.0.2
84 bytes from 172.17.0.2 icmp_seq=1 ttl=62 time=3.523 ms
84 bytes from 172.17.0.2 icmp_seq=2 ttl=62 time=3.828 ms

Troubleshooting

You can apparently send NHRP debugs to syslog, but I didn’t find them particularly helpful when I was troubleshooting. I found a packet capture to be the most useful. The the key is to get NHRP going, in a packet capture you should see a NHRP registration request and reply with Code=Success. If you see registration requests with no request at all, you have a problem. On my cisco I originally forgot to put in the tunnel key command, and it kept sending a request and getting an ICMP error response. Don’t forget your tunnel key.

DMVPN between Ubuntu and Cisco, pretty cool right?

EIGRP Routing on Linux via Free Range Routing

Background

If you are familiar with networking on Linux, you may know of the Quagga project, which is an old open source project that implemented some routing protocols. You may also have heard that a cool company called Cumulus Linux forked Quagga into an open source project they call “Free Range Routing” (Their github page has the most recent code, the project page has more info, and their documentation is extensive) . They have taken the development of the project into high gear and implemented many new features, new protocols, and generally gone above and beyond the original Quagga project which had seen little to no development activity for years.

While you may be able to find some binary packages out there, building from source is the best way to get the most recent code and the most recent features. However building from source can be time consuming if your not familiar with the project and/or the process of building from source. If you just want to poke around a little bit or do some testing, there is a nice alternative using Ubuntu’s “snap” technology. Snaps are kind of like containers, although they’re different from Docker or LXC. The idea is to package up all the libraries the application needs so it can run on any Linux distribution that can run the snap daemon. Stability, portability, and ease of installation are the goal. You can check out more about snap at the Snapcraft page.

Free Range Routing offers snaps through the snap store and installation is downright easy.

If you’re familiar with Cisco technologies, you may know that their proprietary (and yes, it’s still proprietary) routing protocol EIGRP is only available on Cisco devices. In 2013 Cisco published a draft RFC detailing how to implement EIGRP, opening the gate for anyone interested to implement it. There were a few attempts to add it on to Quagga, however none of them really got past alpha stage. Free Range Routing is the first major attempt to implement it, although it’s still in experimental stage. Installing the Free Range Routing snap will get you the latest EIGRP code, and you can run it on your Linux box.

Installation

There is a readme specific to installing the snap package that comes when you install the snap itself, but it’s at the Github page as well of course.

Ubuntu 18.04

Make sure to update, and install snapd:

sudo apt update
sudo apt install snapd

Then install the frr snap:

sudo snap install frr

Then you’ll need to give it permission to control the network:

snap connect frr:network-control core:network-control

Restart your box to finish.

Topology

Pretty simple idea here. We want PC1 to be able to reach PC2. There are no routes in place, so that’s where EIGRP comes in. The Ubuntu box doesn’t know about 172.16.0.0/24 and the Cisco IOSv Router doesn’t know about 192.168.0.0/24, and we’ll use EIGRP to tell each router about those networks. This blog assumes IPv4 routing has been enabled on the Ubuntu box and IP addresses have been configured on the interfaces shown.

Configuration

Ubuntu 18.04

You’ll find the snap files under “/snap”, so cd to “/snap/bin” where you’ll find all the frr stuff. Most importantly you’ll find frr.vtysh which is the command line interface for frr. Running that will take you into the frr CLI, or you can use an absolute path to it:

sudo /snap/bin/frr.vtysh

You’ll be greeted with the frr CLI which is very Cisco-like and based on the original Quagga CLI. You can configure EIGRP using Cisco-like commands:

#conf t
(config)#
(config-router)#network 192.168.0.0/24
(config-router)#network 172.16.0.0/24
(config-router)#end
#

CiscoIOSv15.6(2)T-1

I’ve abbreviated the output of my show run a bit to make things more readable:

interface GigabitEthernet0/0
 ip address 10.0.0.2 255.255.255.252
!
interface GigabitEthernet0/1
 ip address 172.16.0.1 255.255.255.0
!        
router eigrp 10
 network 10.0.0.0 0.0.0.3
 network 172.16.0.0 0.0.0.255

Verification

Ubuntu 18.04

You can use the same show commands within the frr CLI to see if you’re getting routes:

# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

E   10.0.0.0/30 [90/28160] is directly connected, eth0, 00:43:35
C>* 10.0.0.0/30 is directly connected, eth0, 00:46:38
E>* 172.16.0.0/24 [90/5376] via 10.0.0.2, eth0, 00:42:36
E   192.168.0.0/24 [90/28160] is directly connected, eth1, 00:42:47
C>* 192.168.0.0/24 is directly connected, eth1, 00:45:56
# 

Of course you can always drop out with “exit” to the Bash shell and do “ip route”:

$ip route
10.0.0.0/30 dev eth0 proto kernel scope link src 10.0.0.1 
172.16.0.0/24 via 10.0.0.2 dev eth0 proto 192 metric 20 
192.168.0.0/24 dev eth1 proto kernel scope link src 192.168.0.1 
$ 

CiscoIOSv15.6(2)T-1

It’s nice when the commands are the same (similar) everywhere.

Router#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        10.0.0.0/30 is directly connected, GigabitEthernet0/0
L        10.0.0.2/32 is directly connected, GigabitEthernet0/0
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.0.0/24 is directly connected, GigabitEthernet0/1
L        172.16.0.1/32 is directly connected, GigabitEthernet0/1
D     192.168.0.0/24 [90/28416] via 10.0.0.1, 00:44:51, GigabitEthernet0/0
Router#

Finally, make sure PC1 can ping PC2:

PC1> ping 172.16.0.2
84 bytes from 172.16.0.2 icmp_seq=1 ttl=62 time=1.843 ms
84 bytes from 172.16.0.2 icmp_seq=2 ttl=62 time=1.879 ms
84 bytes from 172.16.0.2 icmp_seq=3 ttl=62 time=1.904 ms
84 bytes from 172.16.0.2 icmp_seq=4 ttl=62 time=1.906 ms

EIGRP on Linux!