OpenBSD bgpd
Setting up bgpd on OpenBSD (and dn42)
So this is the third part of the series, with the other two focusing on ospf and iked.
We’re using OpenBSD, and OpenBGPD which comes as part of the OS.
The aim of this was to hook up to DN42, a decentralised network which participants join and create wireguard tunnels to peers, and then exchange routes via BGP. It’s an awesome playground if you wanted to dip into learning BGP. Having 0 experience with BGP, I muddled my way through, so consider this - at best - a starting point for configuring BGP.
If you wanted to sign up to DN42 then the tl’dr is to create and account, git fork their registry repo, add some new stuff for your registration, and then create a PR. Details are here. I’ll start the article by setting the scene, including the stuff for KAIZO-MNT in their registry.
The first step is to instantiate and identify who you are, this is done by creating a new file:
zsh 903 (git)-[master]-% more data/mntner/KAIZO-MNT
mntner: KAIZO-MNT
admin-c: KAIZO-DN42
abuse-mailbox: simonb@kaizo.org
tech-c: KAIZO-DN42
mnt-by: KAIZO-MNT
source: DN42
auth: sk-ssh-ed25519@openssh.com AAAAsomepubkey
- The auth field is set to the mechanism you will use to sign git commit messages - all git commits are required to be signed. I chose to use my yubikey’s ssh key for this.
Glossing over the details for the rest of it, you’ll self-assign an IPv4 network range:
zsh 908 (git)-[master]-% cat data/inetnum/172.23.132.224_27
inetnum: 172.23.132.224 - 172.23.132.255
cidr: 172.23.132.224/27
netname: KAIZO-NETWORK
descr: Kaizo Development Network
admin-c: KAIZO-DN42
tech-c: KAIZO-DN42
mnt-by: KAIZO-MNT
status: ASSIGNED
nserver: reachout.kaizo.dn42
nserver: outreach.kaizo.dn42
ds-rdata: 33219 15 1 61eeb7b3c68b1d440d0a23273567d8fcb9f9e900
ds-rdata: 33219 15 2 e7eaf0225eff6ac0a247c0a8ccbd5e028c6efaa171bd122420375a68eee49132
source: DN42
and, an IPv6 network range:
zsh 907 (git)-[master]-% cat data/inet6num/fd1f:abc9:4ab::_48
inet6num: fd1f:abc9:04ab:0000:0000:0000:0000:0000 - fd1f:abc9:04ab:ffff:ffff:ffff:ffff:ffff
cidr: fd1f:abc9:4ab::/48
netname: KAIZO-NETWORK
descr: Kaizo Development Network
country: GB
admin-c: KAIZO-DN42
tech-c: KAIZO-DN42
mnt-by: KAIZO-MNT
status: ASSIGNED
nserver: outreach.kaizo.dn42
nserver: reachout.kaizo.dn42
ds-rdata: 6952 15 2 2aec218d283e03bc92f64497a8555f2c1091121b5b0462966dce1979df57585d
ds-rdata: 6952 15 1 ddc439a3f5e171ae64aca6c14cc9c5cb0dbc5d29
source: DN42
So, now we have our IP ranges we can create some connectivity - you do this by establishing peers. In essence, this is done by creating some wireguard interfaces, and assigning one of your IP addresses from the range to either this interface, or to an `lo’ interface.
OpenBGPD
One of the biggest challenges I faced was getting BGP up and running. Most participants use Linux, and BIRD. The DN42 wiki has only some very brief notes on OpenBGPD, and those only cover IPv6 connectivity and no complete bgpd.conf example.
Let’s dip into the config file, line by line and section by section.
ASN
When you register with DN42, you will choose a new Autonomous System Number (ASN), which will be in 424242XXXX range. This range is defined as ‘Reserved for private use’ in RFC6996. The KAIZO-MNT ASN is 4242423138.
The first thing we will need to define is our ASN in the config file. As usual, macro’s are used in the configuration:
# define our own ASN as a macro
ASN="4242423138"
And then, we need to listen on some IP addresses. I chose to bind mine to specific addresses, rather than listening on the wildcarded addresses:
listen on fd1f:abc9:4ab:b000::1
listen on fd1f:abc9:4ab:c000::2
listen on fe80::ade1%wg300
listen on fe80::100%wg200
listen on fe80::100%wg100
listen on 172.23.132.253
listen on 172.23.132.254
Some of those addresses are IPv6 in the fd1f range - which is a private range that is asssigned to DN42. A few more addresses are link-local addresses specific to the interface (wg100, wg200, wg300) that the address is bound to. As an example, here’s one of the /etc/hostname.wg100 configuration files.
wgkey $PRIVKEY
wgport 10001
wgpeer REMOTE_PUBKEY wgendpoint fr1.g-load.eu 23138 wgaip 0.0.0.0/0 wgaip ::/0 wgdescr fr1.g-load.eu
inet 172.23.132.253/32
inet6 fe80::100/64
inet6 fd1f:abc9:4ab:2000:100::2/128
group dnfourtwo
!route add 172.20.53.102 172.23.132.253
!sleep 3 && ping -c 3 172.20.53.102
- The !route command ensures that the remote endpoint is accessible via this interface
- The ping command sends some traffic to that host so that the interface comes up in an
activemode
Next, I decided that I wanted to run a BGP ’looking glass’. This is all covered in the bgplg manpage, so I’m not going to cover this in depth, but it requires access to a restricted socket to be able to communicate with the daemon. There’s no harm in doing this, so for completeness I’ll include the line:
socket "/var/www/run/bgpd.rsock" restricted
We then need to define our AS number, as well as a router-id:
# global configuration
AS $ASN
router-id 172.23.132.254
After that, we need to choose what networks we are going to announce to the rest of the world - if you don’t announce any, then you’re not going to get any traffic :-)
prefix-set dn42 {
fd1f:abc9:4ab::/48
172.23.132.224/27
}
Those network ranges are the ones you can see above, that I have registered in DN42.
Next, we need to set the large-community on that prefix-set:
network prefix-set dn42 set large-community $ASN:1:1
At this point, it’s worth having a read around iBGP and eBGP - the very terse summary is if you have multiple nodes (which we do in this example) then you will be running internal bgp (iBGP) if all the nodes have the same AS number. If they don’t then they are external bgp (eBGP). I have 3 nodes in my configuration, meaning each node will reference 2 other nodes as part of the ibgp mesh. To wit:
group "ibgp mesh" {
remote-as $ASN
neighbor fd1f:abc9:4ab:b000::2 {
descr fw0
announce IPv4 unicast
announce IPv6 unicast
#set nexthop fd1f:abc9:4ab:b000::2
#set nexthop self
}
#neighbor 100.64.1.2 {
# descr fw0
# set nexthop self
#}
neighbor fd1f:abc9:4ab:c000::1 {
descr reachout
announce IPv4 unicast
announce IPv6 unicast
#set nexthop fd1f:abc9:4ab:c000::1
#set nexthop self
}
#neighbor 192.168.168.2 {
# descr reachout
# set nexthop self
#}
}
I’ve left in the IPv4 connections, but we will advertise both the IPv4 and IPv6 routes over a single IPv6 connection. This is achieved via these two lines:
announce IPv4 unicast
announce IPv6 unicast
We could just as eassily add those two lines to an IPv4 configuration and achieve the same outcome.
Two very interesting lines (commented in this example) are:
#set nexthop fd1f:abc9:4ab:c000::1
#set nexthop self
Changing those settings will affect the nexthop settings, which you may or may not need depending on your particular set up. As mentioned, I won’t go into detail, but setting next hop self means that other hosts will try and contact an upstream network via the IP address of this host. Let’s look into that a little…
With BGP, you don’t have the concept of a default gateway - it’s simply you, and who you know how to reach. Let’s imagine we have:
- a host with an IP address of
192.168.254.254/32, - which knows that to reach a network on
10.0.100.0/24 - it needs to send packets via
10.0.0.1/32. - and this box knows how to get packets to
10.0.0.1/32.
Now imagine we have another box on 192.168.1.1/32, which knows how to get to 192.168.254.254.
This box will see a BGP route that tells it to talk to 10.0.100.0/24 via the gateway at 10.0.0.1/32 - but this box has no idea how to get to that 10.0.0.1 address. As a result, traffic will simply be dropped.
However, by adding the line set nexthop self line, the route will appear in BGP as going via 192.168.254.254, thus making traffic routable! You can also define other IP addresses arbitarily, as you can see in the first line of the config. You may not need them, I include them for completeness.
At this point, we’re only connected internally - we have “physical” connectivity but we have no BGP sessions to the rest of the world. We need to add a neighbor out there somewhere. Let’s do that now:
group "peering dn42" {
neighbor fe80::ade0%wg100 {
remote-as 4242423914
descr fr1.g-load.eu-6
#set localpref 300
#set prepend-self 5
set nexthop fd1f:abc9:4ab:2000:100::2
}
neighbor 172.20.53.102 {
remote-as 4242423914
descr fr1.g-load.eu-4
#set localpref 300
#set prepend-self 5
}
}
Here we define two connections, to our peer. One is for IPv4 traffic, and one is for IPv6 traffic. We need to explicitly connect to both as we can’t advertise both IPv4 and IPv6 routes over a single connection, as OpenBGPD doesn’t currently support RFC8950. In order to make this a little more obvious, we will create two connections, one for each of the address families.
Let’s examine one of the connections in a little more depth:
neighbor 172.20.53.102 {
remote-as 4242423914
descr fr1.g-load.eu-4
#set localpref 300
#set prepend-self 5
}
The remote-as will dictate what AS you are connecting to, along with a descr which is an arbitary field for your reference. The next two lines are more interesting:
set localpref 300
The localpref setting is a number I chose arbitrarily - and it’s used as a weighting for how traffic is going to egress your network. In the old days of shitty connectivity, you would use this setting to make traffic prefer going over a 1Gb/s link versus the 100Mb/s link if you have two upstream providers. Using this setting you can choose the outbound path that traffic takes.
Return traffic is a little different, and at this point we need to step into the world of routing. BGP routes use the AS number, for example this is an extract of the routing table:
outreach# bgpctl show rib inet | head -10
flags: * = Valid, > = Selected, I = via IBGP, A = Announced,
S = Stale, E = Error
origin validation state: N = not-found, V = valid, ! = invalid
aspa validation state: ? = unknown, V = valid, ! = invalid
origin: i = IGP, e = EGP, ? = Incomplete
flags vs destination gateway lpref med aspath origin
*> V-? 10.37.0.0/16 172.20.53.97 100 0 4242423914 4242422923 65026 65037 i
*m V-? 10.37.0.0/16 172.20.53.102 100 0 4242423914 4242422923 65026 65037 i
*m V-? 10.37.0.0/16 172.23.220.192 100 0 4242421588 4242422923 65026 65037 i
As the flags indicate above, we can see that the selected route is via the gateway at 172.20.53.97. We can also see that we have a local preference setting of 100 for all of the routes, meaning BGP will choose one as they are all of equal weighting - and how it does that is interesting in itself!
However for now, we’re interested in aspath. The path is the number of BGP hops to get to a destination, the less hops the better. In that example above, you see the hop count is the same for each of the routes. Now, consider this one:
*> V-? 172.20.18.64/27 172.20.53.97 100 0 4242423914 64600 4242420101 4242420241 4242423754 i
*m V-? 172.20.18.64/27 172.20.53.102 100 0 4242423914 64600 4242420101 4242420241 4242423754 i
I* V-? 172.20.18.64/27 172.20.53.104 100 0 4242423914 64600 4242420101 4242420241 4242423754 i
* V-? 172.20.18.64/27 172.23.220.192 100 0 4242421588 4242423914 64600 4242420101 4242420241 4242423754 i
We can see that the route via 172.23.220.192 has one extra hop - and won’t be used. Now, back to our example, we can understand what is possibly going to happen if we enable this line:
set prepend-self 5
Let’s assume that the example above for 172.20.18.64/27, we actually are 172.20.18.64/27 and we want to influence the path that traffic is taking to us from the outside world. If we use the prepend-self option on the bgp settings, whichever neighbour we apply it to will do something interesting. Let’s assume we are the host 172.20.53.97, with the AS 4242423754. Using prepend-self 5, we’d end up controlling everybody elses routing tables thusly:
*> V-? 172.20.18.64/27 172.20.53.97 100 0 4242423914 64600 4242420101 4242420241 4242423754 4242423754 4242423754 4242423754 4242423754 i
As a result our hop count is now conssiderably longer, and bgp routers won’t want to select this route, meaning it will flow to the other routers. By fiddling with this on all your routes, as well as controlling your outbound traffic via the local preference setting we looked at above, we can force interesting routing to take place, as in we can have all traffic egress via one router yet having all the return traffic come back via another.
We’ve taken quite a diversion away from our configuration, so let’s get back to it. After the neighbour group above, we can start to define some behaviours, which are similar in logic to firewall rules, and unsurprisingly have the same logical definitions as they do in pf.
# deny more-specifics of our own originated prefixes
deny quick from ebgp prefix-set dn42 or-longer
#deny quick from ebgp prefix-set internal or-longer
#deny quick to ebgp prefix-set internal or-longer
The uncommented line tells bgpd to drop any prefixs that it sees for it’s own networks, nobody else should be routing our traffic for us!
The other two commented lines could be used if you were using bpg to advertise internal routes also, e.g.
prefix-set internal {
10.0.0.0/10
192.168.0.0/16
}
Thiss would stop you leaking out your internal networking to the outside world. We explicitly don’t want to tell anyone else about it, as well as not wanting to have a third party being able to tell us they are the best way to get to your internal network!
Obviously there are many more things you can do with these statements to have fine grained control over behaviour. These, naturally, are covered in more depth in the manpages.
The rest of the configuration is a more stripped down version of the example, based on the information at the dn42 openbgpd page.
# filter out too long paths, establish more peerings instead
deny quick from any max-as-len 8
# Outbound EBGP: only allow self originated networks to ebgp peers
# Don't leak any routes from upstream or peering sessions. This is done
# by checking for routes that are tagged with the large-community $ASN:1:1
allow to ebgp prefix-set dn42 large-community $ASN:1:1
# enforce ROA
allow from ebgp ovs valid
# IBGP: allow all updates to and from our IBGP neighbors
allow from ibgp
allow to ibgp
# Scrub normal and large communities relevant to our ASN from EBGP neighbors
# https://tools.ietf.org/html/rfc7454#section-11
match from ebgp set { large-community delete $ASN:*:* }
# Honor requests to gracefully shutdown BGP sessions
# https://tools.ietf.org/html/rfc8326
match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
# include the roa-set (https://dn42.eu/howto/OpenBGPD)
# defines roat-set, see _rpki-client crontab
include "/etc/bgpd/dn42.roa-set"
There’s only thing worth mentioning - specific to dn42 - which is the ROA set. A Route Origination Authorization details which AS is authorised to advertise which originating IP prefixes. This is generated by dn42 themselves, and to automate the download of this, I just use a crontab entry like this:
15 7 * * * /usr/local/bin/wget -O /etc/bgpd/dn42.roa-set -q https://dn42.burble.com/roa/dn42_roa_obgpd_46.conf && bgpctl reload >/dev/null 2>&1
With all that in place, you should be able to ‘Go Explore’, which incidentally is the OpenBGP motto!
Full Configuration
# $OpenBSD: bgpd.conf,v 1.22 2023/09/27 10:49:21 claudio Exp $
# example bgpd configuration file, see bgpd.conf(5)
# define our own ASN as a macro
ASN="4242423138"
#listen on fe80::bbbb%sec0
listen on fd1f:abc9:4ab:b000::1
listen on fd1f:abc9:4ab:c000::2
#listen on fe80::bbbb%wg0
listen on fe80::ade1%wg300
listen on fe80::100%wg200
listen on fe80::100%wg100
#listen on 192.168.168.1
#listen on 100.64.1.1
#listen on 172.23.132.252
listen on 172.23.132.253
listen on 172.23.132.254
socket "/var/www/run/bgpd.rsock" restricted
#log updates
# global configuration
AS $ASN
router-id 172.23.132.254
prefix-set dn42 {
fd1f:abc9:4ab::/48
172.23.132.224/27
}
network prefix-set dn42 set large-community $ASN:1:1
group "ibgp mesh" {
remote-as $ASN
neighbor fd1f:abc9:4ab:b000::2 {
descr fw0
announce IPv4 unicast
announce IPv6 unicast
#set nexthop fd1f:abc9:4ab:b000::2
#set nexthop self
}
#neighbor 100.64.1.2 {
# descr fw0
# set nexthop self
#}
neighbor fd1f:abc9:4ab:c000::1 {
descr reachout
announce IPv4 unicast
announce IPv6 unicast
#set nexthop fd1f:abc9:4ab:c000::1
#set nexthop self
}
#neighbor 192.168.168.2 {
# descr reachout
# set nexthop self
#}
}
group "peering dn42" {
#neighbor 172.22.119.10 {
# remote-as 64719
# descr us-nyc.dn42.lutoma.org-4
# set prepend-self 5
# set localpref 50
#}
#neighbor fe80::acab%wg200 {
# remote-as 64719
# descr us-nyc.dn42.lutoma.org-6
# set nexthop fd1f:abc9:4ab:2000:200::1
# set prepend-self 5
# set localpref 50
#}
#neighbor fe80::d299:909a%wg100 {
# remote-as 4242423847
# descr de-flk-dn42.0011.de-6
# set nexthop fd1f:abc9:4ab:2000:100::2
#}
neighbor fe80::ade0%wg100 {
remote-as 4242423914
descr fr1.g-load.eu-6
#set localpref 300
set prepend-self 5
set nexthop fd1f:abc9:4ab:2000:100::2
}
neighbor 172.20.53.102 {
remote-as 4242423914
descr fr1.g-load.eu-4
#set localpref 300
set prepend-self 5
}
neighbor fe80::1588%wg200 {
remote-as 4242421588
descr nl-ams01.dn42.tech9.io-6
set nexthop fd1f:abc9:4ab:2000:200::1
set prepend-self 5
}
neighbor 172.23.220.192 {
remote-as 4242421588
descr nl-ams01.dn42.tech9.io-4
set prepend-self 5
}
neighbor 172.20.53.97 {
remote-as 4242423914
descr de2.g-load.eu-4
#set localpref 400
set prepend-self 5
}
neighbor fe80::ade0%wg300 {
remote-as 4242423914
descr de2.g-load.eu-6
set nexthop fd1f:abc9:4ab:2000:300::2
#set localpref 400
set prepend-self 5
}
}
# deny more-specifics of our own originated prefixes
deny quick from ebgp prefix-set dn42 or-longer
#deny quick from ebgp prefix-set internal or-longer
#deny quick to ebgp prefix-set internal or-longer
# filter out too long paths, establish more peerings instead
deny quick from any max-as-len 8
# Outbound EBGP: only allow self originated networks to ebgp peers
# Don't leak any routes from upstream or peering sessions. This is done
# by checking for routes that are tagged with the large-community $ASN:1:1
allow to ebgp prefix-set dn42 large-community $ASN:1:1
# enforce ROA
allow from ebgp ovs valid
# IBGP: allow all updates to and from our IBGP neighbors
allow from ibgp
allow to ibgp
# Scrub normal and large communities relevant to our ASN from EBGP neighbors
# https://tools.ietf.org/html/rfc7454#section-11
match from ebgp set { large-community delete $ASN:*:* }
# Honor requests to gracefully shutdown BGP sessions
# https://tools.ietf.org/html/rfc8326
match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
# include the roa-set (https://dn42.eu/howto/OpenBGPD)
# defines roat-set, see _rpki-client crontab
include "/etc/bgpd/dn42.roa-set"