How To Install (and lab) Keepalived on Ubuntu 20.04 and Rocky Linux 8.5

Keepalived is an open source software project that can do many things related to high availability. One of these many things is the Virtual Router Redundancy Protocol, which provides for high availability for IP networking. In other words, you can have two routers and if one goes down, the second one kicks in automatically.

The way this works is two or more routers exchange VRRP messages on a subnet and based on their configuration, decide who is the master and who is the backup. Once this is decided they will agree on a pre-configured “virtual” IP, or an IP that is not configured on an interface, but a floating one that either router can assume responsibility for should the other one fail for some reason and VRRP messages stop flowing.

Topology

VRRP lab in GNS3

The relevant network here is on the bottom half, where a subnet of 192.168.0.0/24 is configured. The Ubuntu server has 192.168.0.2/24 on its ens3 interface, while Rocky has 192.168.0.3/24 on its ens3 interface. They will both have keepalived installed and through VRRP share virtual IP of 192.168.0.1/24. The Alpine linux “PC” at the bottom which is acting as a workstation or desktop computer will have its default route configured to point to 192.168.0.1, the VRRP address.

Ubuntu 20.04 configuration

On the Ubuntu server we’ll install the keepalived available from the package manager with this:

apt-get install keepalived

Once that’s installed, we’ll write the configuration file which is in /etc/keepalived/keepalived.conf. You’ll need to create the keepalived.conf file:

vrrp_instance VI_1 {
	state MASTER
	interface ens3
	virtual_router_id 51
	priority 100
	advert_int 1
	authentication {
		auth_type PASS
		auth_pass james
	}
	virtual_ipaddress {
		192.168.0.1/24
	}
}

I’ll go through the parameters here:

  • state MASTER: the state that the router will start in.
  • interface ens3: VRRP protocol messages should flow is ens3.
  • virtual_router_id: An integer that both routers should have configured to the same thing.
  • priority: who wins the master/backup election – higher numerical means higher priority.
  • advert_int: backup waits this long (multiplied by 3) after messages from master fail before becoming master
  • authentication: a clear text password authentication.
  • virtual_ipaddress: the agreed-upon virtual IP that the routers will share

Restart keepalived to load the config:

systemctl restart keepalived

Rock Linux configuration

Since Rocky uses yum for package management, we install keepalived like this:

yum install keepalived

And in the /etc/keepalived/keepalived.conf file we’ll write this:

vrrp_instance VI_1 {
	state BACKUP
	interface ens3
	virtual_router_id 51
	priority 99
	advert_int 1
	authentication {
		auth_type PASS
		auth_pass james
	}
	virtual_ipaddress {
		192.168.0.1/24
	}
}

Make sure to restart keepalived to load the config:

systemctl restart keepalived

The only parameters that are different are the state and the priority.

Verify keepalived and VRRP

The first thing you can check on Ubuntu is the /var/log/syslog file to make sure keepalived started and is in the correct state:

tail /var/log/syslog
---
Dec  1 12:23:47 u20vm Keepalived[12349]: Opening file '/etc/keepalived/keepalived.conf'.
Dec  1 12:23:47 u20vm Keepalived[12349]: Starting VRRP child process, pid=12360
Dec  1 12:23:47 u20vm Keepalived_vrrp[12360]: Registering Kernel netlink reflector
Dec  1 12:23:47 u20vm Keepalived_vrrp[12360]: Registering Kernel netlink command channel
Dec  1 12:23:47 u20vm Keepalived_vrrp[12360]: Opening file '/etc/keepalived/keepalived.conf'.
Dec  1 12:23:47 u20vm Keepalived_vrrp[12360]: Registering gratuitous ARP shared channel
Dec  1 12:23:47 u20vm Keepalived_vrrp[12360]: (VI_1) Entering BACKUP STATE (init)
Dec  1 12:23:48 u20vm Keepalived_vrrp[12360]: (VI_1) received lower priority (99) advert from 192.168.0.3 - discarding
Dec  1 12:23:51 u20vm Keepalived_vrrp[12360]: message repeated 3 times: [ (VI_1) received lower priority (99) advert from 192.168.0.3 - discarding]
Dec  1 12:23:51 u20vm Keepalived_vrrp[12360]: (VI_1) Entering MASTER STATE

On Rocky the journalctl -e command showed me the keepalived logs.

Once you’ve confirmed that keepalived is in the right state, you can prove it further with a wireshark capture. If you’re doing this lab in GNS3 like I am, it’s easy, just right click the link (in this case, between the Ubuntu server and the switch) and capture on it. Otherwise you can use something like tcpdump on the Ubuntu/Rocky routers themselves. Right when you restart the keepalived process you will see packets going back and forth. That’s the master election process and exchange of parameters/neighbor establishment:

VRRP election in wireshark

But once the routers are in agreement and VRRP is working, packets will only flow from the master to the designated (in the RFC) VRRP multicast address at 224.0.0.18:

VRRP keepalives in wireshark

We can also further prove that when we initiate a ping to 8.8.8.8 from the Alpine “PC” (it’s a docker container), we can see that traffic is flowing through the Ubuntu router.

Simulate a failure

We’ll simulate a failure by shutting the ens3 interface on the Ubuntu router, like so:

ip link set ens3 down

The Rocky router will observe that VRRP “hello” messages are no longer going to 224.0.0.18, and quickly assume the role of master and take over for 192.168.0.1. I did a continuous ping on the Alpine PC and it didn’t actually show any failed pings! We can see that traffic is now flowing through the Rocky router:

Now let’s “fail back” to the Ubuntu router by re-enabling the ens3 interface:

ip link set ens3 up
ip addr add 192.168.0.1/24 dev ens3

And we should be able to see keepalived resuming master responsibilities in the Ubuntu /var/log/syslog:

Dec  1 12:44:53 u20vm Keepalived_vrrp[12379]: Netlink reports ens3 up
Dec  1 12:44:53 u20vm systemd-networkd[602]: ens3: Gained carrier
Dec  1 12:44:55 u20vm systemd-networkd[602]: ens3: Gained IPv6LL
Dec  1 12:45:04 u20vm Keepalived_vrrp[12379]: (VI_1) Entering BACKUP STATE
Dec  1 12:45:04 u20vm Keepalived_vrrp[12379]: (VI_1) received lower priority (99) advert from 192.168.0.3 - discarding
Dec  1 12:45:07 u20vm Keepalived_vrrp[12379]: message repeated 3 times: [ (VI_1) received lower priority (99) advert from 192.168.0.3 - discarding]
Dec  1 12:45:08 u20vm Keepalived_vrrp[12379]: (VI_1) Entering MASTER STATE

While the Rocky log will show a similar message about becoming backup. We can see that pings are once again flowing through the Ubuntu router:

Hope you liked it.

How To Check If A UDP Port Is Listening Using Netcat

Netcat is a super cool tool to perform a great number of different testing and troubleshooting tasks. On Ubuntu Linux, netcat comes built-in so you don’t even have to install it. A common question I get from people trying to troubleshoot a UDP-based application is how to check if a UDP port is “listening” or not. Most folks are familiar with the telnet-based method of checking if a TCP port is listening (e.g., telnet google.com 80), but UDP remains elusive.

Today let’s take a look at why UDP is tricky to check and a method using the netcat tool to see if a server is listening on a particular UDP port.

Topology

We’re using a very simple topology here, my PC is at 192.168.122.1 and the server we’re going to check for an open UDP port is at 192.168.122.252. My PC runs an Ubuntu 20.04 Desktop OS, while the server is running the Ubuntu Server version.

TCP and UDP are different protocols

TCP is designed to establish “reliable” connections, meaning packets are either delivered on-time and in order, or the connection fails. To guarantee delivery, it must first establish a “socket”, which is where the TCP handshake comes in. All TCP sockets start like this:

Client –> [SYN] –> Server
Client <– [SYN/ACK] <– Server
Client –> [ACK] –> Server

Once this handshake is done, a socket and connection is established.

A telnet test will succeed if this TCP handshake completes properly, letting you know that the port you’re testing is open.

The problem with UDP is that no such handshake exists, because it was designed specifically to not have connections or reliability. Neither is “better”, they’re just used for different things.

So how do you test for an open UDP connection if there’s not a protocol handshake or guaranteed response? If an application is NOT listening on a particular UDP port, and a UDP packet arrives on that port, the OS will send back an ICMP message of type 3 and code 3, meaning “destination unreachable” and “port unreachable”, respectively. Because UDP lacks a protocol message for this purpose, the ICMP message serves to notify the sender that they’re barking up the wrong tree.

Set up a netcat UDP server

Setting up a server to listen on a UDP port of our choosing is easy using netcat. Just run this command on the Ubuntu server:

netcat -ul 2000 &

The -ul switches specify UDP and “listen” mode, respectively. The ampersand at the end puts the process in the background so we can get our shell back. After running this command, we can do a quick netstat to see that there is indeed a UDP server listening on port 2000:

netstat -an | grep 2000
udp        0      0 0.0.0.0:2000            0.0.0.0:*

Use netcat on the client to test the UDP port

From “James’s PC” on the diagram, I’ll run the following command:

nc -vz -u 192.168.122.252 2000

The -v and -u options specify verbose and UDP mode, respectively. The -z option tells netcat not to send any data with the packet. At the end are the IP address of the server and to UDP port 2000. If the port is open, we should see output like this:

Connection to 192.168.122.252 2000 port [udp/*] succeeded!

The message is well and good, but well all know packets don’t lie. If we take a wireshark capture of the interaction, we can see why this success message is produced:

Five UDP packets were sent by netcat on James’s PC to make sure no response came back from the server. If there’s no ICMP response, we can infer that the port is indeed open.

The netcat UDP server will shut down after hitting it once with UDP packets, so it’s no longer listening on port 2000. If we run our command again, we’ll get no output from netcat and wireshark clearly shows the ICMP type 3 code 3 message being sent back from the server:

UDP port testing is not like TCP at all, since we can only infer the port’s open state, not guarantee it like TCP.

I hope this one was helpful!

Wireguard VPN on Ubuntu 20.04

Wireguard is an attempt to improve VPN tunnels in a number of ways – simpler code, less compute, easier configuration, the list goes on. If we’re comparing it to IPsec, I would say that yes, it’s a bit easier to configure. One of the main differences is that it does not rely on the classic two IPsec options for keys – PSK and X.509 certificates. Instead, it relies on public key/private key similar to SSH.

Today, we’re going to configure a very simple policy-based site-to-site Wireguard VPN. By “policy-based” I mean that tunneled traffic is determined by a pre-written configuration in the Wireguard configuration file, not by static or dynamic routes. By site-to-site, I mean this is not a remote-access (road warrior) VPN, it’s designed to connect the subnets that sit behind the two VPN peers.

Topology

The two Ubuntu20.04 machines are serving as routers and VPN peers. No routing is in place for PC1 and PC2 to talk to each other. While the two Ubuntu machines can connect to each other, there is no network-level encryption. That’s where Wireguard comes in.

Generate key pairs

If you haven’t installed it yet, wireguard can be installed with apt-get:

apt-get install wireguard

We’ll use the wg command to generate keys on Ubuntu20.04-1. These two commands will generate a private and public key pair:

wg genkey > private1.key
wg pubkey < private1.key > public1.key

We’ll do the same on Ubuntu20.04-2

wg genkey > private2.key
wg pubkey < private2.key > public2.key

Tunnel configurations

Then we’ll write a tunnel config file for Ubuntu20.04-1 in /etc/wireguard/wg0.conf:

[Interface]
PrivateKey = uOnrdfW/ANcU+fh+RUjlb3TQlFWIdwbOpDpAA+NkonY=
Address = 10.0.0.1/30
ListenPort = 8888

[Peer]
PublicKey = JjSDqdzjPOX+iIAaWCjxxg1yZ76jAd6jfSfv/1AlojI=
Endpoint = 12.0.0.2:8888
AllowedIPs = 10.0.0.0/30, 172.16.0.0/24

You’ll notice I took the keys out of the key files and pasted them into the config file. Under “Interface” we have Ubuntu20.04-1’s own private key, while the public key of Ubuntu20.04-2 is under “Peer”. The “Address” parameter is the “glue network” of the tunnel. This will be the virtual subnet that exists inside the tunnel encryption. The “AllowedIPs” parameter is where tunneled traffic is specified (interesting traffic). You need to put the destination subnet here. Since this is Ubuntu20.04-1 and 172.16.0.0/24 is the destination on the other side of the tunnel, we put 172.16.0.0/24 as part of the “AllowedIPs” parameter. Also put the glue network in here as well.

We can write a similar one for Ubuntu20.04-2 in /etc/wireguard/wg0.conf. For Ubuntu20.04-2, 192.168.0.0/24 is on the other side of the tunnel so 192.168.0.0/24 will be in “AllowedIPs”. Under “Interface”, we have its own private key, while the public key of Ubuntu20.04-1 is pasted under “Peer”.

[Interface]
PrivateKey = KJgjkPQVhOX5CyYYWr7B6v1AbI7H2kEtBi4wdhAES2g=
Address = 10.0.0.2/30
ListenPort = 8888

[Peer]
PublicKey = +GOlIMgLAnLZujraI8m4F6JyWZOpxWGRAPSUqkwrZyg=
Endpoint = 11.0.0.2:8888
AllowedIPs = 10.0.0.0/30, 192.168.0.0/24

To bring the tunnels up, on each Ubuntu machine run this command:

wg-quick up wg0

To verify the configuration is loaded, use “wg show”. I ran this one from Ubuntu20.04-2:

wg show
---
interface: wg0
  public key: JjSDqdzjPOX+iIAaWCjxxg1yZ76jAd6jfSfv/1AlojI=
  private key: (hidden)
  listening port: 8888

peer: +GOlIMgLAnLZujraI8m4F6JyWZOpxWGRAPSUqkwrZyg=
  endpoint: 11.0.0.2:8888
  allowed ips: 10.0.0.0/30, 192.168.0.0/24
  latest handshake: 28 minutes, 48 seconds ago
  transfer: 1.09 KiB received, 2.63 KiB sent

We should be able to ping the tunnel glue network IPs (here from Ubuntu20.04):

root@u20vm:/home/james# ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=5.22 ms

And PC1 and PC2 can ping each other too:

PC1> ping 172.16.0.2

84 bytes from 172.16.0.2 icmp_seq=1 ttl=62 time=4.942 ms

You’ll notice we did not have to add any routes for 192.168.0.0/24 and 172.16.0.0/24 to be able to reach each other. The Wireguard configuration added routing automatically, which is why I am calling this type of tunnel “policy-based”.

And the most fun part, Wireshark. We can see the traffic going back and forth, and the protocol is labeled “Wireguard”. Pretty cool, right?

Wireshark capture of Wireguard traffic

Hope you liked this one.

Basic Firewall With Iptables on Ubuntu 20.04

Ubuntu comes with iptables, a configuration utility that allows you to manage rules for Netfilter, the Linux Kernel firewall. Using iptables you can manipulate packets as they leave, enter, or are forwarded across network interfaces on a Linux operating system. Today we’ll look at how to block SSH traffic going through an Ubuntu 20.04 system acting as a router.

Topology

I have a basic configuration here with three IP subnets – 192.168.0.0/24 where the SSH client lives, 10.0.0.0/30 is a transit network between routers, and 172.16.0.0/24 where the SSH server lives. We will be configuring Ubuntu20.04-Firewall with iptables to block SSH traffic.

Installation

Iptables requires no external package installation with apt-get or otherwise, it comes stock-and-standard with a fresh Ubuntu 20.04 Server or Desktop OS. You can verify the status of your iptables rules like so:

iptables -S

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

By default, iptables is configured to pretty much do nothing. No packets are filtered and no NAT (network address translation) is configured.

Iptables syntax

The syntax for iptables can be quite confusing, and since I myself am not configuring them on a daily basis, I always need to reference documentation (or someone else’s blog) for how to configure something specific. It’s a good idea to take a quick look at the basic syntax though. The command structure looks like this, taken straight from the iptables manual page:

iptables [-t table] [mode] [chain] [rulenum] [rule-specification] [options]
  • table is which type of table you want to use. Usually it’s filter for dropping disallowed packets, or nat for translating packets. Filter is default if none is specified.
  • mode is an action, -A (append), -I (insert), -D (delete), -R (replace), -L (list), -P (set policy, is default action) are all valid modes.
  • chain is the part of the routing process to which your rule applies, there are five chains – PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING.
  • rulenum gives the rule a sequence, rules are applied one-by-one and when there is a match, the corresponding action is taken and all subsequent rules are ignored.
  • rule-specification is the actual rule itself. There are many parameters that can be specified here, like protocol, source, destination, etc.
  • options give you some customization – for example, you can add -v for verbose output.

Now that we have a basic idea of what iptables does, let’s drop some SSH packets.

Verifying what we’re blocking

On Ubuntu20.04-Firewall in my topology, I can see that Ubuntu20.04-SSH_Client at 192.168.0.2 has SSH access to Ubuntu20.04-SSH_Server at 172.16.0.2:

james@client$ssh james@172.16.0.2
james@172.16.0.2's password:

james@server$

SSH is an network application protocol that usually uses TCP port 22 to establish an encrypted command-line session between two computers. If we do a quick wireshark betwen the Ubuntu20.04-Firewall and CiscoIOSv15.6(1)T-1 router, we can see this SSH traffic traversing the link:

Wireshark capture of SSH traffic in GNS3

Starting at the top you can see the TCP handshake on port 22 starting with the SYN flag. A few packets later SSHv2 packets begin. All packets use a randomized source TCP port (source is different per session) and a destination TCP port of 22. So we can safely say that in this experiment, if we drop TCP destination port 22, we will effectively block SSH traffic between the client and server.

Configure the iptables rule

A rule to drop SSH packets on TCP 22 can be configured in one line on Ubuntu20.04-Firewall:

iptables -A FORWARD -p tcp --dport 22 -j DROP
  • -A specifies we are appending a rule.
  • FORWARD applies the rule to packets being forwarded from one interface to another
  • -p tcp is a rule-specification to apply the rule to TCP packets
  • --dport 22 applies the rule to destination TCP port 22
  • -j DROP tells iptables to drop the packet if the previous conditions match

Verify

Let’s see if the client can connect now:

james@client$ ssh james@172.16.0.2
ssh: connect to host 172.16.0.2 port 22: Connection timed out

It worked! Packets on TCP port 22 are being dropped at the Ubuntu firewall.

Please keep in mind – this is a simple block on TCP destination port 22. TCP and UDP ports use the concept of “well-known” which means servers running protocols use a port that everyone “knows”. This is so that someone connecting to a server for the first time with no previous knowledge of its configuration will be able to connect, since they both assume that TCP port 22 is used for SSH. However if someone configures SSH on the client and server in this example to use any other port, it will go right around the filter.

Reach out if you have issues or something to share!

Create Your Own Certificate Authority With OpenSSL and NGINX On Ubuntu 20.04

OpenSSL is probably the most widely used cryptographic tool in existence. Deployed on millions of machines, from cloud servers and containers to small embedded devices like network routers, it performs many functions to secure communications between devices. Security in computing and networking is a gigantic topic. So gigantic in fact, that no single blog post can come even close covering everything. To keep things simple, I’ll narrow things to just getting our own environment created, including a “certificate authority” (CA), a web server and a web client. These days, SSL is synonymous with TLS. TLS is just the name of the newer versions of the SSL protocol.

Topology

The network topology in this post isn’t very important, we’re going to be looking at how a certificate authority works with SSL encryption. But I’ve created a basic IP subnet 10.0.0.0/24 so we can watch the SSL traffic flow in Wireshark. All three machines are running Ubuntu 20.04. All SSL-related functions will be performed with OpenSSL.

A (very) basic explanation of a certificate authority

Certificate authorities, SSL, and Public Key Infrastructure (PKI) are a complex topic. To explain the whole thing would take lots of different explanations to break down each piece of the bigger picture and explain the part it plays. So to avoid having a gigantic explanation here, I’m going to sum up the function of a CA with a single sentence:

“A certificate authority provides a way for computers to verify that other computers they communicate with are in fact the domain name (i.e., example.com) they claim to be, and not a fraud.”

I hope that helps. I’ll take a crack at explaining PKI in another post.

Create the CA

A CA can be created with OpenSSL with a couple quick commands. When this process is complete, we will have two files, a private key and a digital certificate. First create a private key on our CA Ubuntu machine (at 10.0.0.3):

openssl genrsa -aes128 -out james_ca.key 2048

Now we’ll create a certificate. The private key’s corresponding public key will be on this certificate. We’ll be asked some questions after issuing the command, most of them are not important. The Common Name field should ideally be unique:

openssl req -x509 -new -nodes -key james_ca.key -sha256 -days 365 -out james_ca.pem
----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Idaho
Locality Name (eg, city) []:Boise
Organization Name (eg, company) [Internet Widgits Pty Ltd]:James
Organizational Unit Name (eg, section) []:James
Common Name (e.g. server FQDN or YOUR name) []:James_CA
Email Address []:someone@example.com

Our server is now a CA! The world’s CA’s are considered “trusted” because web browsers everywhere come with their certificates pre-installed. The CA we just created is trusted by no one and pre-installed in nothing, but we can get our Ubuntu client machine to trust the CA by manually loading the CA certificate into the client’s certificate store.

Load the CA certificate on the client

We’ll use secure copy (scp) which runs over SSH to copy the CA certificate to the Ubuntu client. The command looks like this:

scp james@10.0.0.3:james_ca.pem .

james_ca.pem                                  100% 1456   605.0KB/s   00:00

The first line above is the command, the second is the output. Now we need to add this to Ubuntu’s trusted certificate store. A couple commands will do the trick.

cp james_ca.pem /usr/local/share/ca-certificates/james_ca.crt
update-ca-certificates 

Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

The first two lines are the commands, the rest is output. We’ve copied james_ca.pem (and changed its extension to .crt) to /usr/local/share/ca-certificates, which is where Ubuntu keeps certificates that it trusts. The update-ca-certificates command loads the new certificate for use in applications, like a web client or browser.

We have done manually what the world’s web browsers have built-in. We have loaded the certificate of a CA we trust. Now any other certificate presented to this client will be trusted if it has been signed by the CA.

Create a signing request on the server

Now we need to create what is called a “signing request”, which is actually just another file in a specific format. We take this signing request file to the CA, which it will use to create a signed certificate, which can be returned to the server and loaded into nginx web server for use. We’ll be asked the same set of questions this time since a certificate is going to be created. The important field is the Common Name, which is usually the server’s domain name, but in this simple topology it’s the IP address (since I haven’t set up DNS) that the client will use to access the nginx web server. The signing request is created with SSL like this:

openssl req -newkey rsa:2048 -keyout private.key -out james.csr

Generating a RSA private key
.................+++++
..............................................................................................+++++
writing new private key to 'private.key'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Idaho
Locality Name (eg, city) []:Boise
Organization Name (eg, company) [Internet Widgits Pty Ltd]:James
Organizational Unit Name (eg, section) []:James
Common Name (e.g. server FQDN or YOUR name) []:10.0.0.2
Email Address []:someone@example.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

The first line is the command, the rest is output. The passphrase you use for the private key will have to be removed later, I’ll show you how at that time.

Now we can take the file james.csr to the CA and produce a signed certificate.

Load the signing request on the CA

Let’s again use secure copy to bring the signing request file to the CA. On the CA’s cli:

scp james@10.0.0.2:james.csr .

The signing request file is now on the CA. We can create a certificated (server.pem) signed with the CA’s private key like this:

openssl x509 -req -in james.csr -days 365 -CA james_ca.pem -CAkey james_ca.key -out server.pem

Just a quick recap of the different files in use here:

  • james.csr is the signing request from the server
  • james_ca.pem is the CA’s certificate
  • james_ca.key is the CA’s private key
  • server.pem is the CA-signed certificate that we can now return to the server for use in nginx

Great! We’ve got a freshly signed certificate ready to go. Let’s copy it to the server and load it into nginx.

Load the signed certificate into nginx

Let’s use scp again to copy server.pem to the server. On the server’s cli:

scp james@10.0.0.3:server.pem .

Now we have the signed certificate on our server. By the way, nginx can be quickly installed like this:

apt-get install nginx

And check it’s working with systemctl status nginx:

As I said earlier, we’ll need to remove the passphrase encryption from the server’s private key before it can be used in nginx. It’s one command, like this:

openssl rsa -in private.key -out private_unencrypted.pem

Now that the private key has been unencrypted into a new file called private_unencrypted.pem, it can be used in the nginx config file. The default nginx config is in /etc/nginx/sites-available/default. I added these lines, they’ll be a bit different for yours if the folders and/or files are differently named:

listen 443 ssl default_server;
ssl_certificate /home/james/server.pem;
ssl_certificate_key /home/james/private_unencrypted.pem;
server_name 10.0.0.2;

Restart nginx:

systemctl restart nginx

If you don’t get any errors, you are good to go!

Verify SSL is working from the client

There are a lot of commands to get the certificate presented from the server, view it, verify it, etc. But for now, we can just validate the SSL connection. You can do that with curl really easy from the client:

curl https://10.0.0.2

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

.... omitted for brevity

We can see we properly got an HTML document back from the server. We can also see it in Wireshark if you capture traffic going to and from the client and server. We can see the entire SSL (TLS) handshake go through without any errors:

Ironically, the CA has not had its own certificate installed in its certificate store (unless you did that on your own). So we can actually see using curl from the command line of the CA machine what it would look like to try to access the server without the certificate installed. From the CA:

curl https://10.0.0.2

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This is the command line equivalent of this screen in your browser:

Since the CA’s certificate hasn’t been installed in it’s certificate store, the connection to the server at 10.0.0.2 can’t be verified and so you get a nasty message.

This was a long one but I hope it helped you understand keys, certificates and certificate authorities! It sure helped me!

SSH IP VPN Tunnel on Ubuntu 20.04

Today we will create an virtual interface to which you can assign an IP address and use like any other IP interface on Ubuntu. It’s transmissions are encrypted by SSH. This is not SSH port-forwarding. I repeat, this is not layer 4 SSH port-forwarding or what is commonly known as SSH-tunneling. This is full layer-3 connectivity on top of SSH.

SSH is a common tool for network engineers and systems administrators to securely access the CLI (command-line interface) of various systems. OpenSSH is an open-source implementation of the protocol and is included or available to install on most Linux distributions. While it’s a great tool for CLI access, SSH has other, darker powers that some consider to be hacking tools or black magic.

One of OpenSSH’s tools that is somewhat well known is the “SSH Tunnel”, and is basically a port forwarding technique that allows the sending of a single TCP or UDP port through an SSH connection. A Much less known feature is OpenSSH’s ability to create a virtual Ethernet adapter on top of an SSH connection. This allows full layer-3 IP connectivity, not just a single layer-4 TCP or UDP port. You can add routes that point through this virtual connection, just like you would any other Ethernet interface. You can even run a routing protocol across it.

Topology

We are setting up an SSH IP tunnel from Ubuntu20.04Server-1 on the left side at private physical IP 192.168.0.2 to Ubuntu20.04Server-3 on the right with a public physical IP of 12.0.0.2. The SSH tunnel will use a network of 10.0.0.0/30. One fun fact here is that this tunnel is traversing a NAT (PAT) that I set up on the Cisco router that is connecting Ubuntu20.04Server-1 to the Internet. No issues traversing NAT for SSH IP tunnel. Finally, we will add some static routes to allow Ubuntu20.04Server-4 to ping Ubuntu20.04Server-3 through the SSH tunnel.

Installation

Ubuntu20.04Server-3

You probably installed Ubuntu’s OpenSSH server when you installed the OS but you’ll also need a tool called autossh, so run this command on Ubuntu20.04Server-3:

apt-get install openssh-server

Now we’re going to make some changes to the SSH server configuration. Root login is required from the client in order to create a TUN adapter, so we’ll be enabling that. Edit the /etc/ssh/sshd_config file. You will make these changes:

  • Uncomment and change “PermitRootLogin prohibit-password” to “PermitRootLogin without-password”
  • Uncomment and change “PermitTunnel no” to “PermitTunnel yes”
vi /etc/ssh/sshd_config
PermitRootLogin without-password
PermitTunnel yes

The above configuration is confusing – it will allow login as root, but not without a key. It’s relatively secure. Then restart the OpenSSH server:

systemctl restart sshd

Ubuntu20.04Server-1

On Ubuntu20.04Server-1, you’ll need a tool called “autossh” that watches SSH sessions and restarts them if they die. Run this command:

apt-get install autossh

Let’s set up key authentication, so we can log in as root to the server. :

ssh-keygen -t rsa #create an RSA key
cat ~/.ssh/id_rsa.pub | ssh james@12.0.0.2 "mkdir -p ~/.ssh && cat >>  ~/.ssh/authorized_keys" #Copy key to server

Connecting the tunnel

We’re ready to build our tunnel! From Ubuntu 20.04Server-1 (the client at 192.168.0.2), run the following magical command:

autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -NTC -o Tunnel=point-to-point -w 0:0 12.0.0.2 &

There’s a fair amount going on here, I’ll break it down:

  • ‘-M 0’ refers to monitoring tcp port, do not use
  • ‘-o “ServerAliveInterval 30”’ sends a keepalive every 30 seconds
  • ‘-o “ServerAliveCountMax 3″’ retries keepalive a maximum of 3 times. Autossh ends here, SSH native commands start from next option.
  • ’-N’ instructs SSH not to execute a remote command
  • ’-T’ disables pseudo-tty allocation
  • ’-C’ compression, may improve performance, may degrade
  • ‘-o Tunnel=point-to-point’ creates a virtual interface
  • ’-w 0:0’ gives the local and remote tun adapters a number, in this instance 0. Left side of ‘:’ is local, right side is remote.
  • 12.0.0.2 is the tunnel destination
  • The final ampersand runs the command in the background so you can get your shell back.

If you have done everything correctly, you now have a “tun0” device on both the server and client:

ip link

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 0c:88:6c:69:00:00 brd ff:ff:ff:ff:ff:ff
9: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 500
    link/none 

Now you can configure it with IP settings. You can use “ip route” commands for testing, or netplan to survive reboot. On the server, we’ll add a static route for 192.168.0.0/24 pointing through the newly created tunnel so it can access that network. And on Ubuntu 20.04Server-4 (an innocent bystander in the 192.168.0.0/24 network at 192.168.0.3), we’ll also add a route for the 10.0.0/30 network pointing to 192.168.0.2 so we can see that the tunnel works to route all IP traffic, not just traffic between the client and server. Make sure the client has IP forwarding enabled or that won’t work.

#On Ubuntu 20.04Server-1 (the client)

ip addr add 10.0.0.1/30 dev tun0
ip link set tun0 up

#On Ubuntu 20.04Server-3 (the server)
ip addr add 10.0.0.2/30 dev tun0
ip link set tun0 up
ip route add 192.168.0.0/24 via 10.0.0.1

#On Ubuntu 20.04Server-4 (innocent bystander at 192.168.0.3)
ip route add 10.0.0.0/30 via 192.168.0.2

Let’s trying pinging from Ubuntu 20.04Server-4 to Ubuntu 20.04Server-3:

james@u20vm:~$ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=63 time=3.87 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=63 time=3.73 ms

It works! Now let’s try to ping from the server Ubuntu 20.04Server-3 all the way through to Ubuntu 20.04Server-4, going right through that NAT at the Cisco router:

james@u20vm:~$ ping 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
64 bytes from 192.168.0.3: icmp_seq=1 ttl=63 time=3.30 ms
64 bytes from 192.168.0.3: icmp_seq=2 ttl=63 time=3.69 ms

It works!

Hope you enjoyed this one, SSH IP tunnels are one of my favorite Linux hacks.

Dockerized phpipam in GNS3

If you’re keeping track of all your IP addresses from your environment in a really big and messy Excel file, you may want to consider switching to an IP address management tool. One such tool is phpipam, which is a web-based tool that allows you to store your IP addresses in a central database (a SQL database, to be specific). The reasons why that approach would be far superior to an Excel file are pretty clear – first of all no more emailing a million different copies of that Excel file. But it has other advantages as well, for example if your software development team wants to check the availability/reserve a new IP address, subnet or vlan from code, they can do it via the phpipam API without ever clicking on anything.

A testing instance of phpipam can be brought into your GNS3 environment quickly using Docker! It requires a little hacking, but nothing too ambitious. If you haven’t got GNS3 or Docker installed or you don’t know how to add a Docker image to GNS3, check out my post on that topic.

Topology

We’re not doing anything fancy here, just the phpipam docker container connected to the “NAT” cloud node. By default, the NAT cloud node uses a virtual adapter with IP subnet 192.168.122.0/24. The NAT adapter is at .1, and I’ll set my phpipam container to use .2. This setup will allow us to access the phpipam web server in the container at 192.168.122.2 via a web browser from our desktop computer that runs GNS3.

Build a custom docker image

We’re going to quickly build a custom docker image from the official phpipam image on Docker Hub. If you’re using a GNS3 VM, you can do this via a cli session on the VM. If you’re using Linux, just do this from any terminal. Make a directory for your Dockerfile:

mkdir jamesphpipam
vi Dockerfile

Now we’re going to write the docker commands for our custom image. MySQL server needs to be installed, and also the directory “/run/mysqld” needs to be created as well so MySQL can create a Unix socket there:

FROM phpipam/phpipam-www

RUN apk add mysql mysql-client
RUN mkdir /run/mysqld

Now we have an image (it’s based on Alpine Linux) ready to fire up in GNS3. You’ll need to add it from GNS3 preferences -> docker containers -> new. Go through all the screens and use defaults, except you’ll want to set the “start command” to “/bin/sh” to give you command line access when you double click on it from the GNS3 canvas.

Configure MySQL and Apache

First we need to open up the cli on the container and set its IP address to 192.168.122.2 (ip addr add 192.168.122.2/24 dev eth0). Start up both mysqld and httpd (MySQL Server and Apache Web Server), like this:

httpd
mysqld --user=root &

Make sure you use the ampersand at the end of your mysqld command, so it runs in the background.

To set the MySQL user and password, I had to login to the MySQL cli and run these commands in the phpipam docker container:

mysql -u root
ALTER USER 'root'@'localhost' IDENTIFIED BY 'SomeSecret';

Now we should be able to access the phpipam page at 192.168.122.2 from any web browser!

Configure new phpipam installation

If you click on “New phpipam installation”, it will take you to a page to select the SQL database installation type:

Let’s select “Automatic database installation”. Then we just put in the user “root” and password “SomeSecret” that we entered in our mysql cli earlier:

And our database is installed! Now we just need to set the admin password on the next screen:

Click on “Proceed to login”, login with user “admin” and the password you just set. You’ll be taken to the main phpipam page!

Hit me up if you run into any snags!

Add Dockerized Bind DNS Server to GNS3

I posted a while ago on how to install and configure the Bind DNS server on Ubuntu 18.04, and got a request from a reader with help on getting Dockerized Bind into GNS3. This post is the result of my tinkering with that lab.

The organization that oversees the Bind open source project also releases an official Docker image through the Docker hub that anyone can access. Docker container technology can be a tricky at first for systems and network engineers to wrap their heads around. Docker containers are not an entire operating system – full operating systems are designed to run many processes at once. Docker containers are designed to run one process and one process only. They only contain the software and libraries needed to run that process.

GNS3 has a very cool integration with Docker, however. It allows you to add full network adapters to your containers and copies in some handy tools to make the command line environment usable. But since many of the familiar OS tools are not included in most Docker containers like they would be with a standard OS, it can be challenging to get things working right.

If you are using Ubuntu Linux, feel free to check out my guide on installing GNS3 and Docker on Ubuntu 20.04. If you are using Windows or Mac with the GNS3 VM, Docker is already installed on the VM.

Topology

My topology is simple – a single vlan and IP subnet of 10.0.0.0/24. My Bind DNS server will reside at 10.0.0.3, with two Alpine Linux containers at 10.0.0.1 and 10.0.0.2. I walk through getting Alpine Linux containers installed on the post I linked above, if you need help.

Build your own image based off the official ISC Bind image

First open up a shell or terminal on the GNS3 VM or wherever the GNS3 server is located. If you don’t know how to open a shell, they walk you through it on the official GNS3 docs:

https://docs.gns3.com/docs/emulators/create-a-docker-container-for-gns3/

Create a directory where you can write your Dockerfile and build the image:

mkdir jamesbind
cd jamesbind

vi Dockerfile

Feel free to use whatever text editor you like, I’m a vi person. We’re going to write a Dockerfile that looks like this:

FROM internetsystemsconsortium/bind9:9.11
RUN apt-get update
RUN apt-get install vim -y

Basically all this does is pull the official Bind Docker image, and run some commands on the image. Namely we are updating apt-get and installing vi. We need to do this because this docker image does not have a text editor installed, and we have to edit the Bind configuration files.

Full disclosure: there is another, much better way than manually editing config files from inside the container. You can write the config files in the same folder as the Dockerfile, and add them to the Docker image when you build it. However, I think it’s best for learning and troubleshooting purposes to manually edit the text files, so that’s the route I’m going.

Build your image(-t switch gives it a “tag”, which is basically a name):

docker build -t jamesbind .

Don’t forget the period at the end, that’s important. You should now have a fresh docker image with bind and vi installed in it.

Add your image to GNS3

From the GNS3 preferences window, you can now add your image to the list of devices available.

Click through and use the defaults except when you get to the “Start command” window. You’ll want to set that to /bin/bash:

Now you’re ready to use your image in GNS3!

Fire up Bind

Drag all the containers out, connect and double click on them to get a terminal. You should be able to configure IP settings normally using iproute2 commands (ip addr add 10.0.0.1/24 dev eth0, etc). For the Bind container, let’s write our config files. As I mentioned many cycles ago in my Bind server post, there are three Bind config files:

/etc/bind/named.conf.options –> Configures BIND9 options
/etc/bind/named.conf.local –> Sets zone file name and gives its location
/etc/bind/zones/db.jamesmcclay.com –> The actual zone file with DNS records.

First let’s hop into our Bind container (just double click on it) and configure named.conf.options. Mine looks like this:

options {
        directory "/var/cache/bind";
        listen-on { any; };
};

Now on to named.conf.local. This is where you declare your zone. Mine is going to be jamesmcclay.com, I just made it up.

zone "jamesmcclay.com" {
    type master;
    file "/etc/bind/zones/db.jamesmcclay.com";
};

Now for the zone file that we indicated above. It needs to be created, so lets create both the zones folder and jamesmcclay.com zone file:

mkdir zones
cd zones
vi db.jamesmcclay.com
@               IN      SOA     ns.jamesmcclay.com.    root.jamesmcclay.com. (
                                2               ; Serial
                                604800	        ; Refresh
                                86400           ; Retry
                                2419200         ; Expire
                                604800 )        ; Negative Cache TTL
;
@               IN      NS      ns.jamesmcclay.com.
ns              IN      A       10.0.0.3
alpine1         IN      A       10.0.0.1
alpine2         IN      A       10.0.0.2

Finally, fire up Bind by running the “named -g” command. This will run it in the foreground, with debug output which will be handy. Alternatively, you can just run “named” and it’ll go in the background. When you run it, you’ll be looking for a line that says your zone file was loaded. “all zones loaded” seems to be a lie, if there’s errors on your zone, it’ll say that and then say all zones were loaded. Make sure you read the output carefully:

named -g
<...removed for brevity...>
26-Oct-2021 23:49:14.231 zone jamesmcclay.com/IN: loaded serial 2
26-Oct-2021 23:31:49.828 all zones loaded
26-Oct-2021 23:31:49.829 running

In your Alpine containers, add “nameserver 10.0.0.3” to resolv.conf to tell them to use the Bind server for DNS resolution:

echo "nameserver 10.0.0.3" > /etc/resolv.conf

Testing your setup

First let’s ping ns.jamesmcclay.com (the Bind container) from alpine-1:

ping ns.jamesmcclay.com

PING ns.jamesmcclay.com (10.0.0.3): 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=1.080 ms
64 bytes from 10.0.0.3: seq=1 ttl=64 time=1.073 ms

It works! We can see in a wireshark packet capture the DNS request from 10.0.0.1 and response from 10.0.0.3:

Pinging to alpine2.jamesmcclay.com also works:

ping alpine2.jamesmcclay.com

PING alpine2.jamesmcclay.com (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: seq=0 ttl=64 time=0.999 ms
64 bytes from 10.0.0.2: seq=1 ttl=64 time=1.087 ms

Troubleshooting

The Bind configuration files are really sensitive to anything that’s left out. Be sure and check to see if you forgot a semicolon or that your zone file is properly formatted with all required entries in place. And again, I highly recommend using the “named -g” when you are testing, it’ll give you some big hints as to what is wrong with your configuration.

If your Bind server is running with no config errors and something still isn’t working, it could be a network issue. Make sure and do a packet capture to see if packets are actually flowing and they’re what you expect! Sometimes after troubleshooting for a long time I do a packet capture only to find packets were never leaving the network interface due to something I forget, like adding an IP address or route somewhere.

Good luck! Feel free to reach out with questions about your lab, I’m always happy to help.

Install GNS3 and Docker on Ubuntu 20.04 for Cisco and Linux Network Labs

Every major OS has its place, so I’m not hoping to get into that discussion, but I find that Ubuntu Linux works really well for creating network labs in GNS3. If you’re not familiar with GNS3, you’re missing out. It allows you to pull in real VM’s, and even Docker containers into an emulated network environment for testing and experimentation. You can run Cisco routers and switches, other vendor network vendors, Windows Desktop and Server, Linux and any other OS that is supported by Linux’s QEMU/KVM hypervisor which is pretty much anything. GNS3 has many features, but today we’ll just look at getting it installed, along with Docker.

Why is Ubuntu better to run GNS3? You may have noticed that on Windows or Mac versions of GNS3, the server has to run on a VM to work properly. That server VM runs a Linux OS, specifically Ubuntu. So using Ubuntu as your desktop OS means you’re cutting out all of that complexity with the server VM, not to mention the additional RAM consumed. Simply put, GNS3 runs the way it’s supposed to on Ubuntu. Not to knock the Windows and Mac versions, the GNS3 team worked hard on those. But in my humble and honest opinion, Ubuntu just works better for GNS3.

Most folks stick to using VM’s in GNS3, but the Docker integration is pretty awesome and has some very real benefits over VM’s. Any docker container you have installed on the same system as the GNS3 server can be pulled into GNS3, although whether it will work properly depends somewhat on what the container has installed in it.

GNS3 Installation

The official GNS3 Ubuntu releases can be found at their PPA at:

https://launchpad.net/~gns3/+archive/ubuntu/ppa

The PPA can be added and GNS3 installed with just a few quick commands, although it’s a relatively big download:

sudo add-apt-repository ppa:gns3/ppa
sudo apt-get update
sudo apt-get install gns3-gui

When you first run GNS3, you’ll notice that the default option is not a VM, it’s to run the server locally. No VM needed!

At this point, GNS3 is installed, although you may have to run this command to get wireshark captures working:

chmod 755 /usr/bin/dumpcap

Docker Installation

I’ll just be following the official Docker instructions here, they work great:

https://docs.docker.com/engine/install/ubuntu/

These are to install the repository, which is probably the “best” option. There is a convenience one-liner script, but we all know that’s not a good habit to get into, so we’ll avoid that.

First install dependencies:

 sudo apt-get update
 sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

Add the Docker official GPG key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Add the stable repository:

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

And install:

 sudo apt-get update
 sudo apt-get install docker-ce docker-ce-cli containerd.io

To avoid getting permissions errors in GNS3, you’ll need to add your user to the docker group. You’ll need to log out/log in or restart for this to take effect:

sudo usermod -aG docker ${USER}

Add a Docker container to your GNS3

Now that Docker is installed, pulling a Docker image from the Docker Hub is easy. A popular one is Alpine Linux because it’s so small, but packs lots of popular tools and libraries:

docker pull alpine

Now you should be able to add this image to GNS3. Go into GNS3, go to preferences, and all the way at the bottom where it says “Docker containers”. Click on “new”, and you should be able to select the Alpine Linux image from the drop-down menu:

Click through and leave the defaults, but you might want two network adapters instead of one, in case you want it to be a router. Now just drag a couple containers out onto the canvas:

At this point, you should be able to double click on these and get a busybox shell, which will let you configure IP settings and the like. You may have noticed that the startup of these containers is near-instantaneous, and they consume very little RAM. One of the many perks of the lightweight nature of Docker containers. Enjoy!

IPsec on Linux – Strongswan Configuration w/Cisco IOSv (IKEv2, Route-Based VTI, PSK)

IPsec is a cool tool for encrypting connections between network nodes, usually over the Internet (but not always). There are many different ways to configure an IPsec tunnel. Many tunnels use a policy-based approach which means the traffic that is sent through the tunnel is pre-defined using a “policy” that is part of the configuration. That doesn’t always work, you may need to dynamically change what traffic goes through using a routing protocol like OSPF or EIGRP without having to bring the tunnel down and reconfigure it . Thus there is another option, a “route-based” tunnel.

There are two different methods for creating a route-based IPsec tunnel, the first using GRE, which inserts a second GRE IP header into packets going into the tunnel. The second is VTI, which operates in a similar manner to GRE but under the hood it’s quite a different implementation.

VTI was originally way to save IP space on point-to-point links in the early networking days before subnetting. It was adapted as a way to assign routes to an IPsec tunnel. Part of its legacy is a “numbered” or “unnumbered” mode. Originally, VTI could inherit an IP from another interface and save IP address space. This unnumbered mode is pretty strange, so today we’ll take a look at numbered mode to keep things familiar.

Topology

Pretty simple, we’re trying to get the Window10 box at the bottom left to ping the Ubuntu Server 18.04 at the bottom right to ping each other. We’ll configure a tunnel between the Ubuntu box at the top left and the Cisco IOSv router at the top right. The underlying 1.1.1.0/30 network serves to look like a WAN network, while the network inside the VTI tunnel will be 10.0.0.0/30.

Installation

Ubuntu18.04-FRR-1


Strongswan on Ubuntu 18.04 is pretty easy with apt-get:

apt-get install strongswan

Configuration

Ubuntu18.04-FRR-1

First you need to add a config to /etc/ipsec.conf, something that looks like this:

conn tunnel 
        leftupdown=/usr/local/sbin/ipsec-notify.sh #run this script on start
        left=166.0.0.5
        leftsubnet=0.0.0.0/0 #all traffic
        right=166.0.0.1
        rightsubnet=0.0.0.0/0 #all traffic
        ike=aes256-sha2_256-modp1024!
        esp=aes-sha2_256!
        authby=secret
        auto=start
        keyexchange=ikev2
        mark=32 # only accepts packets with this mark
        type=transport

Then configure the PSK in /etc/ipsec.secrets:

1.1.1.2 1.1.1.1 : PSK '12345'

Then create the tunnel script that is referenced in the config. For me the file will be located at /home/james/ipsec-notify.sh:

ip link add vti0 type vti local 1.1.1.2 remote 1.1.1.1 key 32
ip link set vti0 up
ip addr add 10.0.0.2/30 dev vti0
ip lin set dev vti0 mtu 1400

Note the “key 32” in the first line above. That identifies what traffic strongswan should encrypt and corresponds to the “mark” in the strongswan config. It’s important.

Next you need to add a line for your VTI interface in /etc/sysctl.conf that looks like this to disable kernel policy lookups, this is a routed interface:

net.ipv4.conf.vti0.disable_policy=1

Finally, you need to tell Charon (Strongswan’s IKE daemon) to not handle routing. We’ll handle routing on our own. In /etc/strongswan.d/charon.conf, find this line:

install_routes = no

Make sure it’s set to no. Tell strongswan to restart and the tunnel should attempt a connection:

ipsec restart

CiscoIOSv15.6(2)T-1

My running-config is abbreviated, but it looks like this:

crypto ikev2 proposal james-proposal 
 encryption aes-cbc-256
 integrity sha256
 group 2
!
crypto ikev2 policy james-policy 
 proposal james-proposal
!
crypto ikev2 keyring james-ring
 peer remote-router-james
  address 1.1.1.2
  pre-shared-key 12345
!   
crypto ikev2 profile james-profile
 match identity remote address 1.1.1.2 255.255.255.255 
 authentication local pre-share
 authentication remote pre-share
 keyring local james-ring
!
crypto ipsec transform-set james-trans esp-aes esp-sha256-hmac 
 mode transport
!
crypto ipsec profile james-protect-vti
 set transform-set james-trans 
 set ikev2-profile james-profile
!
interface Tunnel0
 ip address 10.0.0.1 255.255.255.252
 ip mtu 1400
 tunnel source 1.1.1.1
 tunnel mode ipsec ipv4
 tunnel destination 1.1.1.2
 tunnel protection ipsec profile james-protect-vti

Static routing

Ubuntu18.04-FRR-1

Pretty simple, just add a route for 172.16.0.0/24 pointing to 10.0.0.1

ip route add 172.16.0.0/24 via 10.0.0.1

CiscoIOSv15.6(2)T-1

Simple here too:

ip route 192.168.0.0 255.255.255.0 10.0.0.2

Optional – EIGRP Routing

Remove those static routes and we’ll try a protocol to achieve the aforementioned dynamic routing.

Ubuntu18.04-FRR-1

Checkout my post on setting up EIGRP with Free Range Routing on Linux for the installation. Assuming you have that done, just log into the FRR shell on Ubuntu, it’s in /snap/bin, or just use the full path:

/snap/bin/frr.vtysh
Hello, this is FRRouting (version 7.2.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
james# 
james# conf t
james(config)# router eigrp 10
james(config-router)# network 192.168.0.0/24
james(config-router)# network 10.0.0.0/30
james(config-router)# end
james# 

CiscoIOSv15.6(2)T-1

On a Cisco IOSv router, it’s pretty simple:

Router(config)#router eigrp 10
Router(config-router)#network 172.16.0.0 0.0.0.255
Router(config-router)#network 10.0.0.0 0.0.0.3
Router(config-router)#end

Verification

Ubuntu18.04-FRR-1

Verifying the tunnel on Ubuntu is done with “ipsec statusall”, although there are more specific commands. This will do since we only have one tunnel. I’ve abbreviated the output, but these lines say it all:

Security Associations (1 up, 0 connecting):
      tunnel[1]: ESTABLISHED 58 minutes ago, 1.1.1.2[1.1.1.2]...1.1.1.1[1.1.1.1]
      tunnel[1]: IKEv2 SPIs: 9a137e96ee332b6b_i* 1599c6291be65825_r, pre-shared key reauthentication in 110 minutes
      tunnel[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1024
      tunnel{11}:  INSTALLED, TRANSPORT, reqid 1, ESP SPIs: c72f4e27_i de8fcbc0_o
      tunnel{11}:  AES_CBC_128/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 30 minutes
      tunnel{11}:   1.1.1.2/32 === 1.1.1.1/32

For routing, just issue “ip route” and you’ll see your static or EIGRP routes:

root@james:/snap/bin# ip route
1.1.1.0/30 dev eth1 proto kernel scope link src 1.1.1.2  
10.0.0.0/30 dev vti0 proto kernel scope link src 10.0.0.2  
172.16.0.0/24 via 10.0.0.1 dev vti0 proto 192 metric 20  
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.1

CiscoIOSv15.6(2)T-1

Lots of ways to show what’s going on with an IPsec tunnel on an IOS device, and “show crypto ipsec sa peer” is one of them. I usually use “i” to just include the status:

Router#show crypto ipsec sa peer 1.1.1.2 | i Status
        Status: ACTIVE(ACTIVE)
        Status: ACTIVE(ACTIVE)
        Status: ACTIVE(ACTIVE)
        Status: ACTIVE(ACTIVE)

Also “show ip route” will show your static or EIGRP routes (D for EIGRP):

      1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        1.1.1.0/30 is directly connected, GigabitEthernet0/1
L        1.1.1.1/32 is directly connected, GigabitEthernet0/1
      10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        10.0.0.0/30 is directly connected, Tunnel0
L        10.0.0.1/32 is directly connected, Tunnel0
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.0.0/24 is directly connected, GigabitEthernet0/0
L        172.16.0.1/32 is directly connected, GigabitEthernet0/0
D     192.168.0.0/24 [90/26882560] via 10.0.0.2, 00:07:56, Tunnel0

Finally, don’t forget to ping from Windows:

Troubleshooting

There are lots of tools here, including the strongswan “ipsec statusall”, Cisco debug commands, and others. But the one that always let’s me know what’s wrong the fastest is a packet capture. Look for IKE negotiation packets (ISAKMP filter in Wireshark) if you’re tunnel isn’t coming up, and make sure traffic goes through the tunnel (ESP filter in Wireshark) when it’s up: