Running FRR in K8s/OpenShift

FRR is a fully featured, high performance, free software IP routing suite included in RHEL 8.

We will be using this FRR image for establishing Bidirectional Forwarding Detection (BFD) sessions.

Config map definition

First, let’s define the proper config map:

apiVersion: v1
data:
  daemons: |
    bgpd=no
    ospfd=no
    ospf6d=no
    ripd=no
    ripngd=no
    isisd=no
    pimd=no
    ldpd=no
    nhrpd=no
    eigrpd=no
    babeld=no
    sharpd=no
    pbrd=no
    bfdd=yes
    fabricd=no
    vrrpd=no
    vtysh_enable=yes
    zebra_options="  -A 127.0.0.1 -s 90000000"
    bgpd_options="   -A 127.0.0.1"
    ospfd_options="  -A 127.0.0.1"
    ospf6d_options=" -A ::1"
    ripd_options="   -A 127.0.0.1"
    ripngd_options=" -A ::1"
    isisd_options="  -A 127.0.0.1"
    pimd_options="   -A 127.0.0.1"
    ldpd_options="   -A 127.0.0.1"
    nhrpd_options="  -A 127.0.0.1"
    eigrpd_options=" -A 127.0.0.1"
    babeld_options=" -A 127.0.0.1"
    sharpd_options=" -A 127.0.0.1"
    pbrd_options="   -A 127.0.0.1"
    staticd_options="-A 127.0.0.1"
    bfdd_options="   -A 127.0.0.1"
    fabricd_options="-A 127.0.0.1"
    vrrpd_options="  -A 127.0.0.1"
  frr.conf: |
    password zebra
    enable password zebra
    bfd
     peer 192.168.216.14 interface net1
       no shutdown
       debug bfd network
       debug bfd peer
       debug bfd zebra
     !
    !
    bfd
     peer 192.168.216.15 interface net1
       no shutdown
       debug bfd network
       debug bfd peer
       debug bfd zebra
     !
    !
    log file /var/log/frr/frr.log debugging
kind: ConfigMap
metadata:
  name: frr

Peer entries must point to the peer(s) BFD endpoint(s) (we assume they are already configured). I am using net1 as the pod will be created with two interfaces (the second one for establishing the L2 BFD connection).

Pod definition

Now let’s define the pod:

apiVersion: v1
kind: Pod
metadata:
  name: frr-bfd
  annotations:
    k8s.v1.cni.cncf.io/networks: '[{ "name": "sriov-net", "ips": [ "192.168.216.50/24" ]}]'
spec:
  containers:
  - name: bfd
    image: quay.io/fpaoline/ovnkbfdtest:0.2
    command: ["/bin/sh","-c"]
    args: ["/usr/lib/frr/frrinit.sh start && tail -f /var/log/frr/frr.log"]
    ports:
    - containerPort: 3784
      protocol: UDP
    securityContext:
      privileged: true
    volumeMounts:
    - name: config-volume
      mountPath: /etc/frr
  volumes:
    - name: config-volume
      configMap:
        name: frr

Once the pod gets created, let’s take a look to the logs:

$ oc logs frr-bfd
Started watchfrr
2021/07/04 21:14:17 ZEBRA: client 15 says hello and bids fair to announce only static routes vrf=0
2021/07/04 21:14:17 ZEBRA: client 32 says hello and bids fair to announce only bfd routes vrf=0
2021/07/04 21:14:18 BFD: zclient: add local address fe80::b0c5:acff:fe58:4c29/64
2021/07/04 21:14:19 BFD:  peer 192.168.216.15 found, but loc-addr 192.168.217.1 ignored
2021/07/04 21:14:20 BFD: state-change: [mhop:no peer:192.168.216.15 local:0.0.0.0 vrf:default ifname:net1] init -> up

Validation

From the last line, we can see the BFD session was properly established. Let’s do some tshooting from inside the container:

$ oc rsh frr-bfd
sh-5.0# sudo yum install -y telnet
sh-5.0# telnet localhost 2617
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Hello, this is FRRouting (version 7.5).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

User Access Verification
Password: zebra

frr-bfd> enable
Password: zebra

frr-bfd# show bfd peers
BFD Peers:
    peer 192.168.216.14 vrf default interface net1
        ID: 3380487267
        Remote ID: 0
        Active mode
        Status: down
        Downtime: 31 minute(s), 53 second(s)
        Diagnostics: ok
        Remote diagnostics: ok
        Peer Type: configured
        Local timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo transmission interval: 50ms
        Remote timers:
            Detect-multiplier: 3
            Receive interval: 1000ms
            Transmission interval: 1000ms
            Echo transmission interval: 0ms

    peer 192.168.216.15 vrf default interface net1
        ID: 1945828781
        Remote ID: 3129413095
        Active mode
        Status: up
        Uptime: 31 minute(s), 50 second(s)
        Diagnostics: ok
        Remote diagnostics: ok
        Peer Type: configured
        Local timers:
            Detect-multiplier: 3
            Receive interval: 300ms
            Transmission interval: 300ms
            Echo transmission interval: 50ms
        Remote timers:
            Detect-multiplier: 5
            Receive interval: 1000ms
            Transmission interval: 1000ms
            Echo transmission interval: 0ms

frr-bfd# show bfd peers brief
Session count: 2
SessionId  LocalAddress                             PeerAddress                             Status
=========  ============                             ===========                             ======
3380487267 unknown                                  192.168.216.14                          down
1945828781 192.168.216.50                           192.168.216.15                          up

frr-bfd# show bfd peers counters
BFD Peers:
    peer 192.168.216.14 vrf default interface net1
        Control packet input: 0 packets
        Control packet output: 2232 packets
        Echo packet input: 0 packets
        Echo packet output: 0 packets
        Session up events: 0
        Session down events: 0
        Zebra notifications: 0

    peer 192.168.216.15 vrf default interface net1
        Control packet input: 2229 packets
        Control packet output: 2234 packets
        Echo packet input: 0 packets
        Echo packet output: 0 packets
        Session up events: 1
        Session down events: 0
        Zebra notifications: 1