1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
5 NFVbench Xtesting test cases and Xtesting CI integration
6 --------------------------------------------------------
8 NFVbench can leverages on `Xtesting CI <https://galaxy.ansible.com/collivier/xtesting>`_ and the common test case execution proposed by `Xtesting <https://xtesting.readthedocs.io/en/latest/>`_.
9 Thanks to a simple test case list, this tool deploys anywhere plug-and-play `CI/CD toolchains in a few commands <https://wiki.opnfv.org/pages/viewpage.action?pageId=32015004>`_.
10 In addition, it supports multiple components such as Jenkins and Gitlab CI (test schedulers) and multiple deployment models such as all-in-one or centralized services.
12 NFVbench using Xtesting and Xtesting CI will permit:
14 - smoothly assemble multiple heterogeneous test cases
15 - generate the Jenkins jobs
16 - deploy local CI/CD toolchains everywhere
17 - dump all test case results and logs for third-party conformance review
19 Xtesting CI only requires GNU/Linux as Operating System and asks for a few dependencies as described in Deploy your own Xtesting CI/CD toolchains:
25 Please note the next two points depending on the GNU/Linux distributions and the network settings:
27 - SELinux: you may have to add --system-site-packages when creating the virtualenv ("Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!")
28 - Proxy: you may set your proxy in env for Ansible and in systemd for Docker https://docs.docker.com/config/daemon/systemd/#httphttps-proxy
30 Here is the default NFVbench tree on the host running nfvbench container as
31 proposed in ``xtesting/ansible/host_vars/127.0.0.1`` file:
33 - ``/home/opnfv/nfvbench/config``: contains nfvbench config files including
34 ``nfvbench.cfg``, the default config file used by xtesting test cases
35 - ``/home/opnfv/nfvbench/results``: top directory to write nfvbench results and log files
47 - /lib/modules/$(uname -r):/lib/modules/$(uname -r)
48 - /usr/src/kernels:/usr/src/kernels -v /dev:/dev
49 - /home/opnfv/nfvbench/config:/etc/nfvbench
50 - /home/opnfv/nfvbench/results:/var/lib/xtesting/results
52 Please note: if you want to use a different directory structure on the host,
53 replace ``/home/opnfv/nfvbench/*`` paths with appropriate paths to permit
54 NFVbench container to access config file and results directory. The config and
55 results paths do not need to share the same root directory.
57 To deploy your own CI toolchain running NFVbench:
62 . nfvbench/bin/activate
64 ansible-galaxy install collivier.xtesting
65 git clone https://gerrit.opnfv.org/gerrit/nfvbench nfvbench
66 ansible-playbook nfvbench/xtesting/ansible/site.yml
69 NFVbench Xtesting test cases and existing CI chain
70 --------------------------------------------------
72 For test automation purpose, Xtesting framework can be used as an executor of NFVbench test cases and called by a CI chain (Jenkins, Gitlab CI ...).
73 Xtesting use a testcases.yaml file to list and run test case. One basic testcases.yaml is provided by NFVbench natively but can be override.
75 Example of CI scenario:
77 .. image:: images/nfvbench-xtesting.png
79 1. Run NFVbench container using Xtesting python library
81 The NFVbench container can be started using docker run command.
83 To run NFVbench using docker run:
88 -e S3_ENDPOINT_URL=http://127.0.0.1:9000 \
89 -e S3_DST_URL=s3://xtesting/$BUILD_TAG/$JOB_NAME-$BUILD_ID \
90 -e HTTP_DST_URL=http://127.0.0.1:8181/$BUILD_TAG/$JOB_NAME-$BUILD_ID \
91 -e AWS_ACCESS_KEY_ID=xtesting \
92 -e AWS_SECRET_ACCESS_KEY=xtesting \
93 -e TEST_DB_URL=http://127.0.0.1:8000/api/v1/results \
94 -e NODE_NAME=nfvbench \
95 -e BUILD_TAG=$BUILD_TAG \
97 -v /lib/modules/$(uname -r):/lib/modules/$(uname -r) \
98 -v /usr/src/kernels:/usr/src/kernels -v /dev:/dev \
99 -v $HOME/nfvbench/config:/etc/nfvbench \
100 -v $HOME/workspace/$JOB_NAME/results:/var/lib/xtesting/results \
101 opnfv/nfvbench run_tests -t 10kpps-pvp-run -r -p
103 +---------------------------------------------------------------+----------------------------------------------------------------------------+
104 | Docker options | Description |
105 +===============================================================+============================================================================+
106 | --rm | clean up container after execution |
107 +---------------------------------------------------------------+----------------------------------------------------------------------------+
108 | -e S3_ENDPOINT_URL | (Xtesting) Environnement variable used to store NFVbench artifacts to Minio|
109 +---------------------------------------------------------------+----------------------------------------------------------------------------+
110 | -e S3_DST_URL | (Xtesting) Environnement variable used for S3 storage destination |
111 +---------------------------------------------------------------+----------------------------------------------------------------------------+
112 | -e HTTP_DST_URL | (Xtesting) Environnement variable used for S3www service |
113 +---------------------------------------------------------------+----------------------------------------------------------------------------+
114 | -e AWS_ACCESS_KEY_ID | (Xtesting) Environnement variable used for S3 access key |
115 +---------------------------------------------------------------+----------------------------------------------------------------------------+
116 | -e AWS_SECRET_ACCESS_KEY | (Xtesting) Environnement variable used for S3 access secret |
117 +---------------------------------------------------------------+----------------------------------------------------------------------------+
118 | -e TEST_DB_URL | (Xtesting) Environnement variable used to export NFVbench results in DB |
119 +---------------------------------------------------------------+----------------------------------------------------------------------------+
120 | -e NODE_NAME | (Xtesting) Environnement variable used as result key identifier in DB |
121 +---------------------------------------------------------------+----------------------------------------------------------------------------+
122 | -e BUILD_TAG | (Xtesting) Environnement variable used as result key identifier in DB |
123 +---------------------------------------------------------------+----------------------------------------------------------------------------+
124 | --privileged | (optional) required if SELinux is enabled on the host |
125 +---------------------------------------------------------------+----------------------------------------------------------------------------+
126 | -v /lib/modules:/lib/modules | needed by kernel modules in the container |
127 +---------------------------------------------------------------+----------------------------------------------------------------------------+
128 | -v /usr/src/kernels:/usr/src/kernels | needed by TRex to build kernel modules when needed |
129 +---------------------------------------------------------------+----------------------------------------------------------------------------+
130 | -v /dev:/dev | needed by kernel modules in the container |
131 +---------------------------------------------------------------+----------------------------------------------------------------------------+
132 | -v $HOME/nfvbench/config:/etc/nfvbench | folder mapping to pass config files between the |
133 | | host and the docker space (see examples below) |
134 | | Here we map the $HOME/nfvbench/config directory on the host |
135 | | to the /etc/nfvbench directory in the container. |
136 | | Any other mapping can work as well |
137 +---------------------------------------------------------------+----------------------------------------------------------------------------+
138 | -v $HOME/workspace/$JOB_NAME/results:/var/lib/xtesting/results| (Xtesting) folder mapping to pass files between the |
139 | | CI chain workspace and the docker space to store Xtesting result files |
140 | | in orchestrator (Jenkins, Gitlab ...) |
141 +---------------------------------------------------------------+----------------------------------------------------------------------------+
142 | opnfv/nfvbench | container image name |
143 +---------------------------------------------------------------+----------------------------------------------------------------------------+
144 | run_tests | (Xtesting) Xtesting command to run test cases |
145 +---------------------------------------------------------------+----------------------------------------------------------------------------+
146 | -t 10kpps-pvp-run | (Xtesting) Xtesting parameter: Test case or tier (group of tests) |
147 | | to be executed. It will run all the test if not specified. |
148 +---------------------------------------------------------------+----------------------------------------------------------------------------+
149 | -r | (Xtesting) Xtesting parameter: publish result to database |
150 +---------------------------------------------------------------+----------------------------------------------------------------------------+
151 | -p | (Xtesting) Xtesting parameter: publish artifacts to a S3 service |
152 +---------------------------------------------------------------+----------------------------------------------------------------------------+
154 2. Run Xtesting test cases
156 Executed directly by NFVbench docker entrypoint after docker start.
158 3. Perform NFVbench test
160 Xtesting call NFVbench python script to execute test case scenario and wait for run to be terminated.
162 4. Export NFVbench result
164 If ``-r`` option is used, results are pushed to a DB through Xtesting.
165 If ``-p`` option is used, results are pushed to a S3 service through Xtesting.
168 Override testcases.yaml file
169 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
171 To replace existing testcases.yaml file, using Xtesting CI add the volume mapping in ``xtesting/ansible/host_vars/127.0.0.1`` file:
178 - /lib/modules/$(uname -r):/lib/modules/$(uname -r)
179 - /usr/src/kernels:/usr/src/kernels -v /dev:/dev
180 - /home/opnfv/nfvbench/config:/etc/nfvbench
181 - /home/opnfv/nfvbench/results:/var/lib/xtesting/results
182 - /home/opnfv/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml
184 * ``/home/opnfv/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml`` : volume mapping to pass testcases.yaml file between the host and the docker space. Host path required testcases.yaml file inside.
187 To replace existing testcases.yaml file, using NFVbench container:
191 docker run --name nfvbench --detach --privileged \
192 -v /lib/modules/$(uname -r):/lib/modules/$(uname -r) \
193 -v /usr/src/kernels:/usr/src/kernels \
195 -v $HOME/nfvbench/config:/etc/nfvbench \
196 -v $HOME/nfvbench/results:/var/lib/xtesting/results \
197 -v $HOME/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.8/dist-packages/xtesting/ci/testcases.yaml \
200 * ``$HOME/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.8/dist-packages/xtesting/ci/testcases.yaml`` : volume mapping to pass testcases.yaml file between the host and the docker space. Host path required testcases.yaml file inside.
203 Example of Xtesting test case
204 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
213 description: 'Data Plane Performance Testing'
216 case_name: 10kpps-pvp-run
217 project_name: nfvbench
226 - nfvbench -c /etc/nfvbench/nfvbench.cfg --rate 10kpps
229 Examples of manual run
230 ~~~~~~~~~~~~~~~~~~~~~~
232 If NFVbench container is already started in CLI mode (see Starting NFVbench in CLI mode dedicated chapter).
233 To do a single run at 10,000pps bi-directional (or 5kpps in each direction) using the PVP packet path:
237 docker exec -it nfvbench run_tests -t 10kpps-pvp-run
239 Xtesting option used:
241 * ``-t 10kpps-pvp-run`` : specify the test case to run
243 To pass all test cases:
247 docker exec -it nfvbench run_tests -t all
249 Xtesting option used:
251 * ``-t all`` : select all test cases existing in testcases.yaml file