--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Arm Limited.
+
+====================
+Container4NFV on Arm
+====================
+
+Abstract
+========
+
+This document gives a brief introduction on Arm's work status, strategy and possible roadmap for Container4NFV
+on arm64 server platform.
+The target of the description is to let you get to know what are Arm's current capability and future
+direction for Container4NFV project.
+
+Introduction
+============
+
+.. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
+.. _Yardsticktst: https://wiki.opnfv.org/download/attachments/2925202/opnfv_summit_-_bridging_opnfv_and_etsi.pdf?version=1&modificationDate=1458848320000&api=v2
+
+As you know, Arm is a silver member of OPNFV. Arm actively takes part in the work
+of Container4NFV project which aims to enable building a container-based NFV
+infrastructure in edge node or core-network on Arm server platform.
+We would introduce the work of Arm containerized NFV-I from the following points:
+1. Architecture
+2. Container Networking
+3. Related Projects
+4. Current Status and Future Plan
+5. Contacts
+
+Architecture
+============
+
+.. _architecture: https://wiki.opnfv.org/pages/viewpage.action?spaceKey=OpenRetriever&title=Container%27s+Architecture+for+Cloud+Native+NFV
+
+Basically, Arm's containerized NFV infrastructure aligns with the architecture_ of Container4NFV which now usually uses
+the Kubernetes as the Container Orchestration Engine(COE), and the CNI as the network framework.
+Currently a typical architecture of containerized NFV-I on Arm is composed of installer, Kubernetes, related OPNFV projects, such as
+Functest, Yardstick, and possible Arm Node Feature Discovery(A-NFD) which would enable finding certain resources and their usage status
+of Arm servers and is still to be developed. In the future, other high-level VNF orchestration engines, such as Tacker or ONAP would also
+be brought in to facilitate the deployment of actual VNFs.
+
+.. image:: images/Arm_Container4NFV_Architecture.PNG
+ :width: 800px
+ :alt: Containerized NFV Infrastruture on Arm
+
+A typical VNF networking service deployment is given as the following graph:
+
+.. image:: images/networking_service_deployment_on_Arm_server.PNG
+ :width: 800px
+ :alt: Networking Service Deployment on Arm Server
+
+
+Container Networking
+====================
+
+Basic Networking Model
+----------------------
+
+
+Since Arm's containerized NFV infrastructure uses Kubernetes as the COE, so the CNI plug-ins are used to orchestrate networking.
+Every time a POD is initialized or removed, the default CNI plug-in is called with the default configuration. This CNI plug-in
+creates a pseudo interface, attaches it to the relevant underlay network, sets the IP and routes and maps it to the POD namespace.
+
+.. _model: https://kubernetes.io/docs/concepts/cluster-administration/networking/
+.. _Flannel: https://github.com/coreos/flannel
+.. _here: https://github.com/coreos/flannel/releases
+.. _Calico: http://docs.projectcalico.org/
+
+The Kubernetes networking model_ satisfies the following fundamental requirements:
+* *1* all containers can communicate with all other containers without NAT
+* *2* all nodes can communicate with all containers (and vice-versa) without NAT
+* *3* the IP that a container sees itself as is the same IP that others see it as
+
+On the Arm platform, the most common Kubernetes networking solution is Flannel_ which uses overlay technique to resolve the pod
+communication across hosts. The arm64 version of Flannel release can be found here_. Project Calico_ is also a high performance,
+highly scalable networking solution which provides network policy for connecting Kubernetes pods based on the same IP networking
+principles as the internet. But Calico for Arm is still under development and it's one of our task to enable it for container
+networking on Arm Container4NFV.
+
+.. _guide: https://thenewstack.io/hackers-guide-kubernetes-networking/
+
+Refer to guide_, Kubernetes uses CNI plug-ins to orchestrate networking. Every time a POD is initialized or removed, the default
+CNI plug-in is called with the default configuration. This CNI plug-in creates a pseudo interface, attaches it to the relevant
+underlay network, sets the IP and routes and maps it to the POD namespace.
+
+.. _CNI: https://github.com/containernetworking/cni
+.. _release: https://github.com/containernetworking/cni/releases
+
+Most of the ordinary Kubernetes CNI_ plugins for arm64, including bridge, flannel, loopback, host-local, portmap, macvlan, ipvlan, ptp,
+noop could be found in the release_ of containernetworking CNI. Current CNI plugins stable version for arm64 is v0.6.0.
+
+
+Multiple Interfaces Support in a Pod
+------------------------------------
+
+.. _Multus: https://github.com/Intel-Corp/multus-cni
+.. _SR-IOV: https://github.com/hustcat/sriov-cni
+.. _DPDK: https://github.com/Intel-Corp/sriov-cni
+.. _PF: https://github.com/hustcat/sriov-cni/issues/14
+
+Kubernetes initially supports only one CNI interface per POD with one cluster-wide configuration. But some VNFs with data plane
+acceleration, there would be one or two interfaces used for high performance data access besides the normal interfaces, such as
+Flannel, Calico, Weave, PTP, which are still kept for control or configuration purpose.
+
+The SR-IOV_ CNI or DPDK_ CNI could be chosen to add data plane acceleration interfaces for Kubernetes Pods. Arm is doing some
+improvement on SR-IOV_ CNI to assign PF_ directly if VF is not needed or available.
+
+With the help of Multus_ CNI plugin, multiple interfaces can be added at the same time when deploying a pod. The Multus CNI has
+the following features:
+
+
+- It is a contact between the container runtime and other plugins, and it doesn't have any of its own net configuration, it calls
+ other plugins like flannel/calico to do the real net conf job.
+
+- Multus reuses the concept of invoking the delegates in flannel, it groups the multi plugins into delegates and invoke each
+ other in sequential order, according to the JSON scheme in the cni configuration.
+
+- No. of plugins supported is dependent upon the number of delegates in the conf file.
+
+- Master plugin invokes "eth0" interface in the pod, rest of plugins(Mininon plugins eg: sriov,ipam) invoke interfaces as "net0",
+ "net1".. "netn"
+
+- The "masterplugin" is the only net conf option of Multus cni, it identifies the primary network. The default route will point
+ to the primary network.
+
+
+A typical Multus CNI configuration with DPDK passthrough(SR-IOV PF) enabled is given below:
+
+::
+
+ {
+ "name": "multus-k8s-network",
+ "type": "multus",
+ "delegates": [
+ {
+ "type": "flannel",
+ "masterplugin": true,
+ "delegate": {
+ "isDefaultGateway": true
+ }
+ },
+ {
+ "type": "sriov",
+ "master": "eth1",
+ "dpdk": {
+ "ethernet_driver": "ixgbe",
+ "io_driver": "vfio-pci",
+ "dpdk_devbind": "/root/dpdk/usertools/dpdk-devbind.py"
+ }
+ },
+ {
+ "type": "sriov",
+ "master": "eth2",
+ "dpdk": {
+ "ethernet_driver": "ixgbe",
+ "io_driver": "vfio-pci",
+ "dpdk_devbind": "/root/dpdk/usertools/dpdk-devbind.py"
+ }
+ }
+ ]
+ }
+
+
+Related Projects
+================
+
+For aligning with Container4NFV 'E' release requirement, there are 3 related projects on Arm Container4NFV platform, which are
+Installer(Joid or others), Yardstick for performance evaluation and Functest for basic function verification.
+
+Installer
+---------
+
+Installer is responsible for the deployment of a typical Container4NFV environment, and for 'E' release, it is considered to have
+a Kubernetes cluster, a simple networking solution(Flannel), Yardstick, and some VNFs. The possible installer for arm64 platform
+would be based on one of Compass, Joid and Apex.
+
+Yardstick
+---------
+
+.. _Yardstick: https://wiki.opnfv.org/yardstick
+.. _contribution: https://gerrit.opnfv.org/gerrit/#/c/43721/
+.. _feature: https://gerrit.opnfv.org/gerrit/#/c/43719/
+.. _enabling: https://gerrit.opnfv.org/gerrit/#/c/43723/
+
+Yardstick_ is an OPNFV Project. The project's goal is to verify infrastructure compliance, from the perspective
+of a Virtual Network Function(VNF). The latest Yardstick implementation had been enhanced to support Kubernetes context and we have
+enabled it on arm64 platform.
+
+A typical sample arm64 config file corresponding to yardstick/tests/opnfv/test_cases/opnfv_yardstick_tc080.yaml which is for x86 is
+given below:
+
+::
+
+ schema: "yardstick:task:0.1"
+
+ scenarios:
+ -
+ type: Ping
+ options:
+ packetsize: 200
+
+ host: host-k8s
+ target: target-k8s
+
+ runner:
+ type: Duration
+ duration: 60
+ interval: 1
+
+ sla:
+ max_rtt: 10
+ action: monitor
+
+ context:
+ type: Kubernetes
+ name: k8s
+
+ servers:
+ host:
+ image: openretriever/yardstick_aarch64
+ command: /bin/bash
+ args: ['-c', 'chmod 700 ~/.ssh; chmod 600 ~/.ssh/*; service ssh restart;while true ; do sleep 10000; done']
+ target:
+ image: openretriever/yardstick_aarch64
+ command: /bin/bash
+ args: ['-c', 'chmod 700 ~/.ssh; chmod 600 ~/.ssh/*; service ssh restart;while true ; do sleep 10000; done']
+
+.. _openretriever: https://hub.docker.com/u/openretriever/
+.. _container4nfv: https://hub.docker.com/u/container4nfv/
+
+Some other useful testing images for arm64, such as images for iperf3, netperf could also be found in the docker hub under openretriever_,
+which would be changed to container4nfv_ in the future because of the project renaming.
+
+Besides enabling_ some tests on Arm platform, Arm has made contributions to add more support for Kubernetes context
+and enable a special feature_ for the netperf test scenario.
+
+Functest
+--------
+
+.. _functest: https://wiki.opnfv.org/display/functest/Opnfv+Functional+Testing
+.. _Danube: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/userguide/index.html
+
+The Functest project provides comprehensive testing methodology, test suites and test cases to test and verify OPNFV Platform functionality
+that covers the VIM and NFVI components.
+
+Functest for Container4NFV could used to verify the basic VIM functionality to support VNF operations(create, modify, shrink, destroy). For
+the Danube_ release, there are 4 domains(VIM, Controllers, Features, VNF) and 5 tiers(healthcheck, smoke, features, components, vnf) and more
+than 20 test cases.
+
+But now the Functest has not been extended to support Kubernetes, which is still under developing.
+
+
+Current Status and Future Plan
+==============================
+
+Now for Arm containerized NFV-I, we have enabled Multus CNI with Flannel CNI, SR-IOV/DPDK CNI. Data plane acceleration with DPDK on SR-IOV or NIC
+passthrough in containers has also been enabled and tested.
+
+
+.. image:: images/Container_Networking_Acceleration_with_DPDK.PNG
+ :width: 800px
+ :alt: Container Networking Acceleration with DPDK
+
+
+We have also enabled Yardstick to verify the compliance of the Pod communication in the Kubernetes context.
+
+.. image:: images/yardstick_container_test_environment_on_arm_nfv-i.PNG
+ :width: 800px
+ :alt: Yardstick Container Test Environment on Arm NFV-I
+
+For the future plan, we would continue to align with the development roadmap of Container4NFV. And the following work would be also be preferred
+ for Arm Contaier4NFV of the next 'F' release:
+
+* *1* Project Calico enablement for arm64
+* *2* VPP DPDK/ODP for container networking
+* *3* OPNFV installer enablement on Arm for Container4NFV
+* *4* Possible enhancement to Yardstick, Functest
+* *5* Typical VNFs w/o data plane accelerations
+* *6* CI work with Yardstick, Functest
+
+
+Contacts
+========
+
+Trevor Tao(Zijin Tao), Bin Lu, Song Zhu, Kaly Xin and Yibo Cai from Arm have made contributions to this document.
+
+Trevor Tao: trevor.tao@arm.com
+Bin Lu: bin.lu@arm.com
+Song Zhu: song.zhu@arm.com
+Kaly xin: kaly.xin@arm.com
+Yibo Cai: yibo.cai@arm.com
Code Review / container4nfv.git / commitdiff
