--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Arm Limited.
+
+
+
+===============================================
+Container4NFV Openwrt Demo Deployment on Arm Server
+===============================================
+
+Abstract
+========
+
+This document gives a brief introduction on how to deploy openwrt services with multiple networking interfaces on Arm platform.
+
+Introduction
+============
+.. _sriov_cni: https://github.com/hustcat/sriov-cni
+.. _Flannel: https://github.com/coreos/flannel
+.. _Multus: https://github.com/Intel-Corp/multus-cni
+.. _cni: https://github.com/containernetworking/cni
+.. _kubeadm: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
+.. _openwrt: https://github.com/openwrt/openwrt
+
+The OpenWrt Project is a Linux operating system targeting embedded devices.
+Also it is a famouse open source router project.
+
+We use it as a demo to show how to deploy an open source vCPE in Kubernetes.
+For Lan port, we configured flannel cni for it. And for Wan port, we configured sriov cni for it.
+
+For demo purpose, I suggest that we use Kubeadm to deploy a Kubernetes cluster firstly.
+
+Cluster
+=======
+
+Cluster Info
+
+In this case, we deploy master and slave as one node.
+Suppose it to be: 192.168.1.2
+
+In 192.168.1.2, 2 NIC as required.
+Suppose it to be: eth0, eth1. eth0 is used to be controle plane, and eth1 is used to be data plane.
+
+Deploy Kubernetes
+-----------------
+Please see link(https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) as reference.
+
+Creat CRD
+---------
+Please make sure that CRD was added for Kubernetes cluster.
+Here we name it as crdnetwork.yaml:
+
+::
+ apiVersion: apiextensions.k8s.io/v1beta1
+ kind: CustomResourceDefinition
+ metadata:
+ # name must match the spec fields below, and be in the form: <plural>.<group>
+ name: networks.kubernetes.com
+ spec:
+ # group name to use for REST API: /apis/<group>/<version>
+ group: kubernetes.com
+ # version name to use for REST API: /apis/<group>/<version>
+ version: v1
+ # either Namespaced or Cluster
+ scope: Namespaced
+ names:
+ # plural name to be used in the URL: /apis/<group>/<version>/<plural>
+ plural: networks
+ # singular name to be used as an alias on the CLI and for display
+ singular: network
+ # kind is normally the CamelCased singular type. Your resource manifests use this.
+ kind: Network
+ # shortNames allow shorter string to match your resource on the CLI
+ shortNames:
+ - net
+
+command:
+
+::
+ kubectl create -f crdnetwork.yaml
+
+Create Flannel-network for Control Plane
+----------------------------------------
+Create flannel network as control plane.
+Here we name it as flannel-network.yaml:
+
+::
+ apiVersion: "kubernetes.com/v1"
+ kind: Network
+ metadata:
+ name: flannel-conf
+ plugin: flannel
+ args: '[
+ {
+ "masterplugin": true,
+ "delegate": {
+ "isDefaultGateway": true
+ }
+ }
+ ]'
+
+command:
+
+::
+ kubectl create -f flannel-network.yaml
+
+Create Sriov-network for Data Plane
+-----------------------------------
+Create sriov network with PF mode as data plane.
+Here we name it as sriov-network.yaml:
+
+::
+ apiVersion: "kubernetes.com/v1"
+ kind: Network
+ metadata:
+ name: sriov-conf
+ plugin: sriov
+ args: '[
+ {
+ "master": "eth1",
+ "pfOnly": true,
+ "ipam": {
+ "type": "dhcp",
+ }
+ }
+ ]'
+
+command:
+
+::
+ kubectl create -f sriov-network.yaml
+
+CNI Installation
+================
+.. _CNI: https://github.com/containernetworking/plugins
+Firstly, we should deploy all CNI plugins. The build process is following:
+
+
+::
+ git clone https://github.com/containernetworking/plugins.git
+ cd plugins
+ ./build.sh
+ cp bin/* /opt/cni/bin
+
+.. _Multus: https://github.com/Intel-Corp/multus-cni
+
+To deploy control plane and data plane interfaces, besides the Flannel CNI and SRIOV CNI,
+we need to deploy the Multus_. The build process of it is as:
+
+::
+ git clone https://github.com/Intel-Corp/multus-cni.git
+ cd multus-cni
+ ./build
+ cp bin/multus /opt/cni/bin
+
+To use the Multus_ CNI,
+we should put the Multus CNI binary to /opt/cni/bin/ where the Flannel CNI and SRIOV
+CNIs are put.
+
+.. _SRIOV: https://github.com/hustcat/sriov-cni
+The build process of it is as:
+
+::
+ git clone https://github.com/hustcat/sriov-cni.git
+ cd sriov-cni
+ ./build
+ cp bin/* /opt/cni/bin
+
+We also need to enable DHCP client for Wan port.
+So we should enable dhcp cni for it.
+
+::
+ /opt/cni/bin/dhcp daemon &
+
+CNI Configuration
+=================
+The following multus CNI configuration is located in /etc/cni/net.d/, here we name it
+as multus-cni.conf:
+
+::
+ {
+ "name": "minion-cni-network",
+ "type": "multus",
+ "kubeconfig": "/etc/kubernetes/admin.conf",
+ "delegates": [{
+ "type": "flannel",
+ "masterplugin": true,
+ "delegate": {
+ "isDefaultGateway": true
+ }
+ }]
+ }
+
+command:
+
+::
+ step1, remove all files in /etc/cni/net.d/
+ rm /etc/cni/net.d/* -rf
+
+ step2, copy /etc/kubernetes/admin.conf into each nodes.
+
+ step3, copy multus-cni.conf into /etc/cni/net.d/
+
+ step4, restart kubelet
+ systemctl restart kubelet
+
+
+Configuring Pod with Control Plane and Data Plane
+=================================================
+
+1, Save the below following YAML to openwrt-vpn-multus.yaml.
+In this case flannle-conf network object act as the primary network.
+
+::
+ apiVersion: v1
+ kind: ReplicationController
+ metadata:
+ name: openwrtvpn1
+ spec:
+ replicas: 1
+ template:
+ metadata:
+ name: openwrtvpn1
+ labels:
+ app: openwrtvpn1
+ annotations:
+ networks: '[
+ { "name": "flannel-conf" },
+ { "name": "sriov-conf" }
+ ]'
+ spec:
+ containers:
+ - name: openwrtvpn1
+ image: "younglook/openwrt-demo:arm64"
+ imagePullPolicy: "IfNotPresent"
+ command: ["/sbin/init"]
+ securityContext:
+ capabilities:
+ add:
+ - NET_ADMIN
+ stdin: true
+ tty: true
+ ports:
+ - containerPort: 80
+ - containerPort: 4500
+ - containerPort: 500
+ ---
+ apiVersion: v1
+ kind: Service
+ metadata:
+ name: openwrtvpn1
+ spec: # specification of the pod's contents
+ type: NodePort
+ selector:
+ app: openwrtvpn1
+ ports: [
+ {
+ "name": "floatingu",
+ "protocol": "UDP",
+ "port": 4500,
+ "targetPort": 4500
+ },
+ {
+ "name": "actualu",
+ "protocol": "UDP",
+ "port": 500,
+ "targetPort": 500
+ },
+ {
+ "name": "web",
+ "protocol": "TCP",
+ "port": 80,
+ "targetPort": 80
+ },
+ ]
+
+2, Create Pod
+
+::
+ command:
+ kubectl create -f openwrt-vpn-multus.yaml
+
+3, Get the details of the running pod from the master
+
+::
+ # kubectl get pods
+ NAME READY STATUS RESTARTS AGE
+ openwrtvpn1 1/1 Running 0 30s
+
+Verifying Pod Network
+=====================
+
+::
+ # kubectl exec openwrtvpn1 -- ip a
+ 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
+ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+ inet 127.0.0.1/8 scope host lo
+ valid_lft forever preferred_lft forever
+ inet6 ::1/128 scope host
+ valid_lft forever preferred_lft forever
+ 3: eth0@if124: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
+ link/ether 0a:58:0a:e9:40:2a brd ff:ff:ff:ff:ff:ff
+ inet 10.233.64.42/24 scope global eth0
+ valid_lft forever preferred_lft forever
+ inet6 fe80::8e6:32ff:fed3:7645/64 scope link
+ valid_lft forever preferred_lft forever
+ 4: net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
+ link/ether 52:54:00:d4:d2:e5 brd ff:ff:ff:ff:ff:ff
+ inet 192.168.123.2/24 scope global net0
+ valid_lft forever preferred_lft forever
+ inet6 fe80::5054:ff:fed4:d2e5/64 scope link
+ valid_lft forever preferred_lft forever
+
+Contacts
+========
+
+Bin Lu: bin.lu@arm.com
Code Review / container4nfv.git / commitdiff