Multi Network Pod with Multus in Kubernetes
Table of Contents
Installation
Note: Check this website for more details on multus. (https://www.redhat.com/en/blog/using-the-multus-cni-in-openshift).
In this lab, we’ll be using K3S as it is easy to setup and is production ready. First we would need to disabled flannel-backend, disable-network-policy to install CNI; calico or CNI of your choice. Optional to disble servicelb, unless you’re implementing metallb.
1$ curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" INSTALL_K3S_EXEC="--flannel-backend=none --cluster-cidr=192.168.0.0/16 --disable-network-policy --disable=traefik --disable servicelb" sh -
Install calico and multus.
1$ kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.3/manifests/calico.yaml
2$ kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml
Check k8s status.
1$ mcbtaguiad@tags-kvm-ubuntu:~$ kubectl get pods -A
2NAMESPACE NAME READY STATUS RESTARTS AGE
3kube-system kube-multus-ds-s7kgz 1/1 Running 1 (50m ago) 6h32m
4kube-system calico-kube-controllers-787f445f84-kp274 1/1 Running 1 (50m ago) 6h35m
5kube-system coredns-6799fbcd5-hs547 1/1 Running 1 (50m ago) 6h36m
6kube-system local-path-provisioner-6c86858495-svqw7 1/1 Running 1 (50m ago) 6h36m
7kube-system calico-node-5nx6b 1/1 Running 1 (50m ago) 6h35m
8kube-system metrics-server-54fd9b65b-crfp7 1/1 Running 1 (50m ago) 6h36m
(Optional) Install CNI plugin. This might come in handy as you progress in exploring/implementing multus.
1$ curl -s -L https://github.com/containernetworking/plugins/releases/download/v1.4.1/cni-plugins-linux-amd64-v1.4.1.tgz | tar xvz - -C /opt/cni/bin
Multus Manifest
We’re going to use macvlan. Use NIC used by K8S cluster, in my case I’m using enp1s0. A dhcp server is also setup in KVM, with IP pool from 192.168.122.201 to 192.168.122.254, these will be the IPs that the pod can attach to.
1mcbtaguiad@tags-kvm-ubuntu:/opt/cni$ ip r
2default via 192.168.122.1 dev enp1s0 proto dhcp src 192.168.122.201 metric 100
3blackhole 172.16.58.128/26 proto bird
4172.16.58.133 dev cali205ed4120a1 scope link
5172.16.58.134 dev caliaeca16354bd scope link
6172.16.58.135 dev cali1a14292463c scope link
7172.16.58.136 dev cali93c699308e7 scope link
8192.168.122.0/24 dev enp1s0 proto kernel scope link src 192.168.122.201 metric 100
9192.168.122.1 dev enp1s0 proto dhcp scope link src 192.168.122.201 metric 100
Create network attachment definition (macvlan).
1cat <<EOF > macvlan-net.yaml
2apiVersion: "k8s.cni.cncf.io/v1"
3kind: NetworkAttachmentDefinition
4metadata:
5 name: macvlan-net
6spec:
7 config: |
8 {
9 "name": "macvlan-net",
10 "cniVersion": "0.3.1",
11 "plugins": [
12 {
13 "cniVersion": "0.3.1",
14 "type": "macvlan",
15 "master": "enp1s0",
16 "mode": "bridge",
17 "ipam": {
18 "type": "host-local",
19 "subnet": "192.168.122.0/24",
20 "rangeStart": "192.168.122.2",
21 "rangeEnd": "192.168.122.254",
22 "routes": [
23 {
24 "dst": "0.0.0.0/0",
25 "gw": "192.168.122.254"
26 }
27 ]
28 }
29 }
30 ]
31 }
32EOF
33kubectl create -f macvlan-net.yaml
Create test pod/deployment. Add custom defined resources or annotation k8s.v1.cni.cncf.io/networks: macvlan-net.
1cat <<EOF > test-multus.yaml
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5 name: test-multus
6spec:
7 selector:
8 matchLabels:
9 app: test-multus
10 replicas: 1
11 template:
12 metadata:
13 labels:
14 app: test-multus
15 annotations:
16 k8s.v1.cni.cncf.io/networks: macvlan-net
17 spec:
18 containers:
19 - name: test-multus
20 image: testcontainers/helloworld
21 ports:
22 - containerPort: 8080
23 imagePullPolicy: IfNotPresent
24 securityContext:
25 privileged: true
26EOF
27kubectl create -f test-multus.yaml
Get pod network status.
1$ kubectl get pods
2NAME READY STATUS RESTARTS AGE
3test-multus-868b8598b-t89g6 1/1 Running 0 2m36s
4
5$ kubectl describe pod test-multus-868b8598b-t89g6
Pod attached to 192.168.122.2.
1Name: test-multus-868b8598b-t89g6
2Namespace: default
3Priority: 0
4Service Account: default
5Node: tags-kvm-ubuntu/192.168.122.201
6Start Time: Wed, 24 Apr 2024 11:28:34 +0000
7Labels: app=test-multus
8 pod-template-hash=868b8598b
9Annotations: cni.projectcalico.org/containerID: d43ea4cbd08d963167a7b53f5dd7a59fe95acd3e73f5bafb69f7345dcb3e1f82
10 cni.projectcalico.org/podIP: 172.16.58.137/32
11 cni.projectcalico.org/podIPs: 172.16.58.137/32
12 k8s.v1.cni.cncf.io/network-status:
13 [{
14 "name": "k8s-pod-network",
15 "ips": [
16 "172.16.58.137"
17 ],
18 "default": true,
19 "dns": {}
20 },{
21 "name": "default/macvlan-net",
22 "interface": "net1",
23 "ips": [
24 "192.168.122.2"
25 ],
26 "mac": "26:39:5a:00:db:60",
27 "dns": {},
28 "gateway": [
29 "192.168.122.254"
30 ]
31 }]
32 k8s.v1.cni.cncf.io/networks: macvlan-net
33Status: Running
34IP: 172.16.58.137
35IPs:
36 IP: 172.16.58.137
37Controlled By: ReplicaSet/test-multus-868b8598b
38Containers:
39 test-multus:
40 Container ID: containerd://c84d601e64a094f8ae8a29c60440392054abe8c7f1ec491694bc08f8c4a2ada9
41 Image: testcontainers/helloworld
42 Image ID: docker.io/testcontainers/helloworld@sha256:4ee5a832ef6eee533df7224b80d4cceb9ab219599014f408d0b69690be94c396
43 Port: 8080/TCP
44 Host Port: 0/TCP
45 State: Running
46 Started: Wed, 24 Apr 2024 11:28:46 +0000
47 Ready: True
48 Restart Count: 0
49 Environment: <none>
50 Mounts:
51 /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-287fk (ro)
52Conditions:
53 Type Status
54 PodReadyToStartContainers True
55 Initialized True
56 Ready True
57 ContainersReady True
58 PodScheduled True
59Volumes:
60 kube-api-access-287fk:
61 Type: Projected (a volume that contains injected data from multiple sources)
62 TokenExpirationSeconds: 3607
63 ConfigMapName: kube-root-ca.crt
64 ConfigMapOptional: <nil>
65 DownwardAPI: true
66QoS Class: BestEffort
67Node-Selectors: <none>
68Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
69 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
70Events:
71 Type Reason Age From Message
72 ---- ------ ---- ---- -------
73 Normal Scheduled 26s default-scheduler Successfully assigned default/test-multus-868b8598b-t89g6 to tags-kvm-ubuntu
74 Normal AddedInterface 22s multus Add eth0 [172.16.58.137/32] from k8s-pod-network
75 Normal AddedInterface 22s multus Add net1 [192.168.122.2/24] from default/macvlan-net
76 Normal Pulling 22s kubelet Pulling image "testcontainers/helloworld"
77 Normal Pulled 14s kubelet Successfully pulled image "testcontainers/helloworld" in 7.796s (7.796s including waiting)
78 Normal Created 14s kubelet Created container test-multus
79 Normal Started 14s kubelet Started container test-multus
80
Testing
Attach to the pod and we can verify that another network is attached net1@tunl0 .
1$ kubectl exec -it test-multus-868b8598b-t89g6 -- ip a
21: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
3 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
4 inet 127.0.0.1/8 scope host lo
5 valid_lft forever preferred_lft forever
6 inet6 ::1/128 scope host
7 valid_lft forever preferred_lft forever
82: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
9 link/ipip 0.0.0.0 brd 0.0.0.0
103: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1480 qdisc noqueue state UP qlen 1000
11 link/ether 7e:15:53:bb:6b:46 brd ff:ff:ff:ff:ff:ff
12 inet 172.16.58.137/32 scope global eth0
13 valid_lft forever preferred_lft forever
14 inet6 fe80::7c15:53ff:febb:6b46/64 scope link
15 valid_lft forever preferred_lft forever
164: net1@tunl0: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
17 link/ether 26:39:5a:00:db:60 brd ff:ff:ff:ff:ff:ff
18 inet 192.168.122.2/24 brd 192.168.122.255 scope global net1
19 valid_lft forever preferred_lft forever
20 inet6 fe80::2439:5aff:fe00:db60/64 scope link
21 valid_lft forever preferred_lft forever
Curl and ping the pod.
1# test using the host network (kvm host)
2$ mcbtaguiad@pop-os:~/develop$ ip r
3default via 192.168.254.254 dev wlp4s0 proto dhcp metric 600
4169.254.0.0/16 dev virbr1 scope link metric 1000 linkdown
5172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
6172.18.0.0/16 dev br-f6e0458d2d6c proto kernel scope link src 172.18.0.1 linkdown
7192.168.100.0/24 dev virbr1 proto kernel scope link src 192.168.100.1 linkdown
8192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
9192.168.254.0/24 dev wlp4s0 proto kernel scope link src 192.168.254.191 metric 600
10
11# ping test
12mcbtaguiad@pop-os:~/develop$ ping 192.168.122.2 -c5
13PING 192.168.122.2 (192.168.122.2) 56(84) bytes of data.
1464 bytes from 192.168.122.2: icmp_seq=1 ttl=64 time=0.314 ms
1564 bytes from 192.168.122.2: icmp_seq=2 ttl=64 time=0.323 ms
1664 bytes from 192.168.122.2: icmp_seq=3 ttl=64 time=0.389 ms
1764 bytes from 192.168.122.2: icmp_seq=4 ttl=64 time=0.196 ms
1864 bytes from 192.168.122.2: icmp_seq=5 ttl=64 time=0.151 ms
19
20--- 192.168.122.2 ping statistics ---
215 packets transmitted, 5 received, 0% packet loss, time 4103ms
22rtt min/avg/max/mdev = 0.151/0.274/0.389/0.087 ms
23
24# curl test
25$ mcbtaguiad@pop-os:~/develop$ curl 192.168.122.2:8080
26<!doctype html>
27
28<html lang="en">
29<head>
30 <meta charset="utf-8">
31 <title>Hello world</title>
32 <style>
33 body {
34 font-family: sans-serif;
35 max-width: 38rem;
36 padding: 2rem;
37 margin: auto;
38 }
39
40 * {
41 max-width: 100%;
42 }
43 </style>
44</head>
45
46<body>
47<h1>Hello world</h1>
48<img src="logo.png" alt="Testcontainers logo"/>
49<p>
50 This is a test server used for Testcontainers' own self-tests. Find out more about this image on <a href="https://github.com/testcontainers/helloworld">GitHub</a>.
51</p>
52<p>
53 Find out more about Testcontainers at <a href="https://www.testcontainers.org">www.testcontainers.org</a>.
54</p>
55<p>
56 Hit <a href="/ping"><code>/ping</code></a> for a simple test response.
57</p>
58<p>
59 Hit <a href="/uuid"><code>/uuid</code></a> for a UUID that is unique to this running instance of the container.
60</p>
61</body>
62</html>