Linux for Network Engineers

This page is dedicated to some of the Linux resources I found useful especially for network administrators/engineers.

Basics

ifconfig

this command is now depracated and ip command is preferred.

To show the ip address

ip address

To see all the interfaces and the associated IPs in a glance.

ip -br a

Show routes

route

ip route
ip route get IP_ADDRESS/bits

frr:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.116.2   0.0.0.0         UG    203    0        0 eth1
10.72.100.0     *               255.255.255.0   U     0      0        0 eth1.100
10.72.200.0     *               255.255.255.0   U     0      0        0 eth1.200
192.168.116.0   *               255.255.255.0   U     0      0        0 eth1

FRR (Free Range Routing)

It is a network routing software suite for Linux/Unix platforms. It can be used to implement BGP, OSPF, RIP, RIPnG, EIGRP, ISIS and FHRPs.

Other supported protocols can be found on the official site.
https://frrouting.org/

After having FRR installed on the linux machine using either method, since I am using ubuntu I use the apt install frr. We switch to the vtysh shell.

Quagga

https://www.quagga.net/

FRRouting forked from Quagga aiming for a more open and faster development.

mdadm (Multiple Disk and Device Management)

mdadm is a Linux utility used to manage and monitor software RAID devices. It is used in modern Linux distributions in place of older software RAID utilities such as raidtools2 or raidtools.

Open Source Load Balancers

https://www.bizety.com/2020/06/24/open-source-load-balancers-neutrino-katran-maglev-seesaw-traefix-and-haproxy/amp/

Routing feature on Linux

To check whether the Linux machine is forwarding traffic and acting like a router.

cat /proc/sys/net/ipv4/ip_forward

if the value returned is 0 it means the routing feature is turned off. If the value returned is 1 it means routing is enabled.

To turn it on

echo "1" > /proc/sys/net/ipv4/ip_fowward

Editing the sysctl.conf file for persistent configuration

+ show block

/etc/sysctl.conf file is used to override default kernel parameter values, only a small number of parameters is predefined in the file.

cat /etc/sysctl.conf
#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#

#kernel.domainname = example.com

# Uncomment the following to stop low-level messages on console
#kernel.printk = 3 4 1 3

###################################################################
# Functions previously found in netbase
#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
#net.ipv4.conf.default.rp_filter=1
#net.ipv4.conf.all.rp_filter=1

# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
#net.ipv4.tcp_syncookies=1

# Uncomment the next line to enable packet forwarding for IPv4
#net.ipv4.ip_forward=1    <----------- remove # and save file

# Uncomment the next line to enable packet forwarding for IPv6
#  Enabling this option disables Stateless Address Autoconfiguration
#  based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1

###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
#net.ipv4.conf.all.log_martians = 1
#

###################################################################
# Magic system request Key
# 0=disable, 1=enable all, >1 bitmask of sysrq functions
# See https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html
# for what other values do
#kernel.sysrq=438

# set default and maximum socket buffer sizes to 12MB
net.core.rmem_default=12582912
net.core.wmem_default=12582912
net.core.rmem_max=12582912
net.core.wmem_max=12582912

# set minimum, default, and maximum tcp buffer sizes (10k, 87.38k (linux default), 12M resp)
net.ipv4.tcp_rmem=10240 87380 12582912
net.ipv4.tcp_wmem=10240 87380 12582912

# Enable TCP westwood for kernels greater than or equal to 2.6.13
net.ipv4.tcp_congestion_control=westwood

Updating & Upgrading Debian and FRR

+ show block

## Upgrade to debian 11
apt update -y && sudo apt upgrade -y
apt dist-upgrade
mv /etc/apt/sources.list /etc/apt/sources.list.bkp

printf  "deb http://deb.debian.org/debian bullseye main 
deb-src http://deb.debian.org/debian bullseye main 
deb https://deb.debian.org/debian-security bullseye-security main contrib 
deb-src https://deb.debian.org/debian-security bullseye-security main contrib 
deb http://deb.debian.org/debian bullseye-updates main 
deb-src http://deb.debian.org/debian bullseye-updates main 
deb http://deb.debian.org/debian bullseye-backports main 
deb-src http://deb.debian.org/debian bullseye-backports main" | tee -a /etc/apt/sources.list

apt update && sudo apt upgrade --without-new-pkgs
apt full-upgrade
apt autoremove
hostnamectl

##FRR installation
apt -y install gnupg2 gnupg
curl -s https://deb.frrouting.org/frr/keys.asc | sudo apt-key add - && \
FRRVER="frr-stable" && echo deb https://deb.frrouting.org/frr $(lsb_release -s -c) $FRRVER | sudo tee -a /etc/apt/sources.list.d/frr.list

apt -y update &&  apt -y install frr frr-pythontools
sed -i "s/^bgpd=no/bgpd=yes/" /etc/frr/daemons
service frr restart
systemctl enable frr.service

Reference: https://deb.frrouting.org/

To check bgp daemon is enabled for FRR

+ show block

If the bgpd daemon is no, then edit the /etc/frr/daemons file and enable it by adding yes.

# cat /etc/frr/daemons
# This file tells the frr package which daemons to start.
#
# Sample configurations for these daemons can be found in
# /usr/share/doc/frr/examples/.
#
# ATTENTION:
#
# When activating a daemon for the first time, a config file, even if it is
# empty, has to be present *and* be owned by the user and group "frr", else
# the daemon will not be started by /etc/init.d/frr. The permissions should
# be u=rw,g=r,o=.
# When using "vtysh" such a config file is also needed. It should be owned by
# group "frrvty" and set to ug=rw,o= though. Check /etc/pam.d/frr, too.
#
# The watchfrr, zebra and staticd daemons are always started.
#
bgpd=no   <--- change this to yes
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no
pimd=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
bfdd=no
fabricd=no
vrrpd=no
pathd=no

#
# If this option is set the /etc/init.d/frr script automatically loads
# the config via "vtysh -b" when the servers are started.
# Check /etc/pam.d/frr if you intend to use "vtysh"!
#
vtysh_enable=yes
zebra_options="  -A 127.0.0.1 -s 90000000"
bgpd_options="   -A 127.0.0.1"
ospfd_options="  -A 127.0.0.1"
ospf6d_options=" -A ::1"
ripd_options="   -A 127.0.0.1"
ripngd_options=" -A ::1"
isisd_options="  -A 127.0.0.1"
pimd_options="   -A 127.0.0.1"
ldpd_options="   -A 127.0.0.1"
nhrpd_options="  -A 127.0.0.1"
eigrpd_options=" -A 127.0.0.1"
babeld_options=" -A 127.0.0.1"
sharpd_options=" -A 127.0.0.1"
pbrd_options="   -A 127.0.0.1"
staticd_options="-A 127.0.0.1"
bfdd_options="   -A 127.0.0.1"
fabricd_options="-A 127.0.0.1"
vrrpd_options="  -A 127.0.0.1"
pathd_options="  -A 127.0.0.1"

# configuration profile
#
#frr_profile="traditional"
#frr_profile="datacenter"

#
# This is the maximum number of FD's that will be available.
# Upon startup this is read by the control files and ulimit
# is called.  Uncomment and use a reasonable value for your
# setup if you are expecting a large number of peers in
# say BGP.
#MAX_FDS=1024

# The list of daemons to watch is automatically generated by the init script.
#watchfrr_options=""

# To make watchfrr create/join the specified netns, use the following option:
#watchfrr_options="--netns"
# This only has an effect in /etc/frr/<somename>/daemons, and you need to
# start FRR with "/usr/lib/frr/frrinit.sh start <somename>".

# for debugging purposes, you can specify a "wrap" command to start instead
# of starting the daemon directly, e.g. to use valgrind on ospfd:
#   ospfd_wrap="/usr/bin/valgrind"
# or you can use "all_wrap" for all daemons, e.g. to use perf record:
#   all_wrap="/usr/bin/perf record --call-graph -"
# the normal daemon command is added to this at the end.

Reference: https://docs.frrouting.org/en/latest/setup.html

Don't forget to restart FRR after the changes are made

systemctl restart frr

or 

service frr restart

Configuring interfaces on Debian based distributions

The interface configuration can be found under;

/etc/network/interfaces

Here you will find the lines are commented out (#), removing the hash will enable the line. IP addresses can statically be configured or dhcp can be enabled. All physical interfaces will follow with an auto in the same line.

In some distributions the interface configuration can be found under /run/networking/interface.d/

root@km-server:/#  ls -l /run/network/interfaces.d/
total 8
-rw-r--r-- 1 root root   89 Sep 16 19:07 eth0
-rw-r--r-- 1 root root 1024 Oct 12 17:02 eth1

Sub-interfaces can also be configured in this file. An example output shown below.

+ show block

frr:~# cat /etc/network/interfaces 
#
# This is a sample network config, uncomment lines to configure the network
#
# Loopback interface
auto lo
iface lo inet loopback

# Static config for eth0
#auto eth0
#iface eth0 inet static
#       address 192.168.0.2
#       netmask 255.255.255.0
#       gateway 192.168.0.1
#       up echo nameserver 192.168.0.1 > /etc/resolv.conf
#  DHCP config for eth0
auto eth1
iface eth1 inet dhcp
        hostname frr

auto eth1.100
        address 10.72.10.2
        netmask 255.255.255.0
        gateway 10.72.10.1
auto eth1.200
        address 10.72.20.2 
        netmask 255.255.255.0
        gateway 10.72.20.1

Configuring interfaces on RedHat/Centos

+ show block

Edit the file with your preferred editor.

/etc/sysconfig/network-scripts/ifcfg-ethX

Configuring static routes on Redhat/Centos

/etc/sysconfig/network-scripts/route-ethX

Add the following to the file

DEVICE=enp3s0
ONBOOT=yes
IPADDR=192.168.10.10
NETMASK=255.255.255.0
GATEWAY=192.168.10.1

Let's see the interfaces configured according to this interface file.

frr:~# ip -br add
lo               UNKNOWN        127.0.0.1/8 ::1/128 
eth0             DOWN           
eth1             UP             192.168.116.173/24 fe80::e06:8bff:fe01:d301/64 
eth2             DOWN           
eth3             DOWN           
eth4             DOWN           
eth5             DOWN           
eth6             DOWN           
eth7             DOWN           
eth1.100@eth1    UP             10.72.10.2/24 fe80::e06:8bff:fe01:d301/64 
eth1.200@eth1    UP             10.72.20.2/24 fe80::e06:8bff:fe01:d301/64 
LINK_SYS         UP

To add or remove an IP address from an interface.

To add

root@linux:~# ip addr add 10.100.100.1/32 dev lo

root@linux:~# ip address show lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.100.100.1/32 scope global lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

To remove

root@linux:~# ip addr del 10.100.100.1/32 dev lo

root@linux:~# ip address show lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

Configuring a virtual interface

+ show block

First enable the module dummy

#sudo modprobe dummy

Create the dummy interface, in this example we will create eth5

#sudo ip link add eth5 type dummy

Since it is a virtual interface, it requires a hardware (mac) address.

#sudo ifconfig eth0 hw ether c0:01:ca:fe:ba:be

Configure the IP address. For this example I will configure 10.5.5.1/24 on interface eth5

#sudo ip addr add 10.5.5.1/24 dev eth5

Bring up the interface

sudo ip link set dev eth0 up
or
sudo ifup -v eth5

Configuring Tunnel interface

+ show block

# GRE Tunnel towards //device_name// 
allow-hotplug tun1
iface tun1 inet tunnel
        address 10.3.65.5
        netmask 30
        mode gre
        endpoint 10.65.10.11
        dstaddr 10.3.65.6
        local 10.3.200.74
        mtu 1500
        up ip route add 10.65.244.0/24 via 10.3.65.6 dev tun1
        up ip route add 10.65.64.0/24 via 10.3.65.6 dev tun1

To restart the networking

systemctl restart networking

service networking restart

/etc/init.d/networking start|stop|restart

To add SSH keys for a user

Edit /root/.ssh/authorized_keys and add to the list of users.

e.g

nano /root/.ssh/authorized_keys

To add a VRF in Linux created in FRR

frr#conf t

vrf LINK_SYS
 ip router-id 100.100.100.100
 exit-vrf

If we check the show status of this created VRF it will show inactive.

frr# show vrf 
vrf LINK_SYS inactive (configured)

To activate it we exit out of vtysh session and configure vrf table information.

km:~# ip link add LINK_SYS type vrf table 100
km:~# ip link add OLIVE_WL type vrf table 200

km:~# ip link set dev LINK_SYS up
km:~# ip link set dev OLIVE_WL up

km:~# ip vrf show
Name              Table
-----------------------
LINK_SYS           100
OLIVE_WL           200

Now the status of this VRF will show active in vtysh shell.

frr# show vrf 
vrf LINK_SYS id 10 table 100 (configured)
vrf OLIVE_WL id 11 table 200 (configured)

Adding an interface to a VRF

#ip link set dev INTERFACE_NAME master VRF_NAME

The ip link show command will show you the interface is added the configured vrf.

To see the IP addresses of the interfaces in a vrf.

#ip -br add show vrf VRF_NAME

sample output
ip -br add show vrf LINK_SYS
bond0.60@bond0 UP             10.10.60.4/27 fe80::63f:72ff:fed9:816e/64 
bond0.66@bond0 UP             10.10.60.36/27 fe80::63f:72ff:fed9:816e/64

Checking the routes in each VRF

frr:~# ip route show vrf LINK_SYS
unreachable default metric 4278198272 
10.202.100.0/24 dev eth2 proto kernel scope link src 10.202.100.1 

frr:~# ip route show vrf OLIVE_WL
unreachable default metric 4278198272 
10.202.200.0/24 dev eth3 proto kernel scope link src 10.202.200.1

Adding the vrf table number in the rt_table for persistent configuration.

+ show block

# cat /etc/iproute2/rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
20      INTERNET
100     LINK_SYS          <------Add to the file
200     OLIVE_WL

Policy Routing Rules

Linux supports up to 255 routing tables, each routing table has its own table name and table ID. IP rule action defines tables to lookup if the rule selector matches. IP rule also defines the priority parameter which indicates the priority of this rule. Higher number means lower priority, and rules get processed in order of increasing number. Each rule should have an explicitly set unique priority value. Source Pica8.com

+ show block

ip rule add priority 2002 from 10.200.192.0/18 lookup CUSTOMER-GI 

Validation#
root@km-server-4:~# ip rule
0:      from all lookup local
1000:   from all lookup [l3mdev-table]
2002:   from 10.200.192.0/18 lookup CUSTOMER-GI
2020:   from 10.199.0.0/16 lookup INTERNET
32766:  from all lookup main
32767:  from all lookup default

Connectivity test towards the dockers

In order to do a ping test to check connectivity with the docker container IPs, we need to specify the source interface which is in the respective VRF. There is no option to specify the source VRF as of of the time doing this test.

frr:~# ping -I eth2 10.202.100.2
PING 10.202.100.2 (10.202.100.2): 56 data bytes
64 bytes from 10.202.100.2: seq=0 ttl=64 time=1.632 ms
64 bytes from 10.202.100.2: seq=1 ttl=64 time=1.169 ms
64 bytes from 10.202.100.2: seq=2 ttl=64 time=1.672 ms
64 bytes from 10.202.100.2: seq=3 ttl=64 time=1.221 ms
64 bytes from 10.202.100.2: seq=4 ttl=64 time=1.163 ms
^C
--- 10.202.100.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 1.163/1.371/1.672 ms

# to validate I will ping the other docker container (10.202.200.2) by sourcing with the same source interface and you will see that it fails.

frr:~# ping -I eth2 10.202.200.2
PING 10.202.200.2 (10.202.200.2): 56 data bytes
ping: sendto: Host is unreachable

# Now I change the interface to the correct interface within OLIVE_WL. 

frr:~# ping -I eth3 10.202.200.2
PING 10.202.200.2 (10.202.200.2): 56 data bytes
64 bytes from 10.202.200.2: seq=0 ttl=64 time=1.028 ms
64 bytes from 10.202.200.2: seq=1 ttl=64 time=2.655 ms
64 bytes from 10.202.200.2: seq=2 ttl=64 time=1.740 ms
64 bytes from 10.202.200.2: seq=3 ttl=64 time=1.191 ms
64 bytes from 10.202.200.2: seq=4 ttl=64 time=1.639 ms
64 bytes from 10.202.200.2: seq=5 ttl=64 time=2.042 ms
^C64 bytes from 10.202.200.2: seq=5 ttl=64 time=2.042 ms
--- 10.202.200.2 ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 1.028/1.715/2.655 ms

Performing ping from within a vrf with source interface

ip vrf exec VRF_NAME ping -I10.10.60.1 8.8.8.8

Checking arp/cache

frr:~# ip neighbor show
10.100.72.2 dev eth2 lladdr 9a:f9:60:61:3c:d0 STALE
10.72.100.1 dev eth1.100 lladdr 0c:93:c2:98:f5:03 REACHABLE
10.72.200.1 dev eth1.200 lladdr 0c:93:c2:98:f5:03 REACHABLE
10.200.72.2 dev eth3 lladdr 8e:ef:5a:eb:32:e0 STALE
fe80::6089:2ff:fef6:d821 dev eth3 lladdr 62:89:02:f6:d8:21 STALE
fe80::8cef:5aff:feeb:32e0 dev eth3 lladdr 8e:ef:5a:eb:32:e0 STALE
fe80::98f9:60ff:fe61:3cd0 dev eth2 lladdr 9a:f9:60:61:3c:d0 STALE
fe80::5052:22ff:fe50:9f74 dev eth2 lladdr 52:52:22:50:9f:74 STALE
fe80::20c:29ff:fe88:c937 dev eth1 lladdr 00:0c:29:88:c9:37 STALE

Querying an NTP server

In Linux install ntpdate tool and run the query against the ntp pool as shown in the example.

+ show block

kaisar@master-node:/$ sudo ntpdate -q fr.pool.ntp.org
server 162.159.200.1, stratum 3, offset -0.034110, delay 0.09921
server 51.38.81.135, stratum 2, offset -0.045893, delay 0.21078
server 51.68.44.27, stratum 3, offset -0.102544, delay 0.32323
15 Feb 20:13:18 ntpdate[1081507]: adjust time server 51.38.81.135 offset -0.045893 sec

Additional useful resources:

Kernel.org Documentation
Useful IP commands
Working with VRF in Linux
VRF in Linux
Debian documentation
Dockers reference documentation