docs/testing/user/userguide/xtesting.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. SPDX-License-Identifier: CC-BY-4.0
   3
   4
   5 NFVbench Xtesting test cases and Xtesting CI integration
   6 --------------------------------------------------------
   7
   8 NFVbench can leverages on `Xtesting CI <https://galaxy.ansible.com/collivier/xtesting>`_ and the common test case execution proposed by `Xtesting <https://xtesting.readthedocs.io/en/latest/>`_.
   9 Thanks to a simple test case list, this tool deploys anywhere plug-and-play `CI/CD toolchains in a few commands <https://wiki.opnfv.org/pages/viewpage.action?pageId=32015004>`_.
  10 In addition, it supports multiple components such as Jenkins and Gitlab CI (test schedulers) and multiple deployment models such as all-in-one or centralized services.
  11
  12 NFVbench using Xtesting and Xtesting CI will permit:
  13
  14 - smoothly assemble multiple heterogeneous test cases
  15 - generate the Jenkins jobs
  16 - deploy local CI/CD toolchains everywhere
  17 - dump all test case results and logs for third-party conformance review
  18
  19 Xtesting CI only requires GNU/Linux as Operating System and asks for a few dependencies as described in Deploy your own Xtesting CI/CD toolchains:
  20
  21 - python-virtualenv
  22 - docker.io
  23 - git
  24
  25 Please note the next two points depending on the GNU/Linux distributions and the network settings:
  26
  27 - SELinux: you may have to add --system-site-packages when creating the virtualenv ("Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!")
  28 - Proxy: you may set your proxy in env for Ansible and in systemd for Docker https://docs.docker.com/config/daemon/systemd/#httphttps-proxy
  29
  30 Here is the default NFVbench tree as proposed in ``xtesting/ansible/host_vars/127.0.0.1`` file:
  31
  32 - ``/home/opnfv/nfvbench``
  33
  34 File content:
  35
  36 .. code-block:: bash
  37
  38     docker_args:
  39     env: {}
  40     volumes:
  41       - /lib/modules/$(uname -r):/lib/modules/$(uname -r)
  42       - /usr/src/kernels:/usr/src/kernels -v /dev:/dev
  43       - /home/opnfv/nfvbench:/tmp/nfvbench
  44
  45 Please note: replace ``/home/opnfv/nfvbench`` with appropriate path to permit NFVbench container to access config file
  46
  47 To deploy your own CI toolchain running NFVbench:
  48
  49 .. code-block:: bash
  50
  51     virtualenv nfvbench
  52     . nfvbench/bin/activate
  53     pip install ansible
  54     ansible-galaxy install collivier.xtesting
  55     git clone https://gerrit.opnfv.org/gerrit/nfvbench nfvbench
  56     ansible-playbook nfvbench/xtesting/ansible/site.yml
  57
  58
  59 NFVbench Xtesting test cases and existing CI chain
  60 --------------------------------------------------
  61
  62 For test automation purpose, Xtesting framework can be used as an executor of NFVbench test cases and called by a CI chain (Jenkins, Gitlab CI ...).
  63 Xtesting use a testcases.yaml file to list and run test case. One basic testcases.yaml is provided by NFVbench natively but can be override.
  64
  65 Example of CI scenario:
  66
  67 .. image:: images/nfvbench-xtesting.png
  68
  69 1. Run NFVbench container using Xtesting python library
  70
  71 The NFVbench container can be started using docker run command.
  72
  73 To run NFVbench using docker run:
  74
  75 .. code-block:: bash
  76
  77     docker run --rm \
  78         -e S3_ENDPOINT_URL=http://127.0.0.1:9000 \
  79         -e S3_DST_URL=s3://xtesting/$BUILD_TAG/$JOB_NAME-$BUILD_ID \
  80         -e HTTP_DST_URL=http://127.0.0.1:8181/$BUILD_TAG/$JOB_NAME-$BUILD_ID \
  81         -e AWS_ACCESS_KEY_ID=xtesting \
  82         -e AWS_SECRET_ACCESS_KEY=xtesting \
  83         -e TEST_DB_URL=http://127.0.0.1:8000/api/v1/results \
  84         -e NODE_NAME=nfvbench \
  85         -e BUILD_TAG=$BUILD_TAG \
  86         --privileged \
  87         -v /lib/modules/$(uname -r):/lib/modules/$(uname -r) \
  88         -v /usr/src/kernels:/usr/src/kernels -v /dev:/dev \
  89         -v $HOME/nfvbench:/tmp/nfvbench \
  90         -v $HOME/workspace/$JOB_NAME/results:/var/lib/xtesting/results \
  91         opnfv/nfvbench run_tests -t 10kpps-pvp-run -r -p
  92
  93 +---------------------------------------------------------------+----------------------------------------------------------------------------+
  94 | Docker options                                                | Description                                                                |
  95 +===============================================================+============================================================================+
  96 | --rm                                                          | clean up container after execution                                         |
  97 +---------------------------------------------------------------+----------------------------------------------------------------------------+
  98 | -e S3_ENDPOINT_URL                                            | (Xtesting) Environnement variable used to store NFVbench artifacts to Minio|
  99 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 100 | -e S3_DST_URL                                                 | (Xtesting) Environnement variable used for S3 storage destination          |
 101 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 102 | -e HTTP_DST_URL                                               | (Xtesting) Environnement variable used for S3www service                   |
 103 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 104 | -e AWS_ACCESS_KEY_ID                                          | (Xtesting) Environnement variable used for S3 access key                   |
 105 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 106 | -e AWS_SECRET_ACCESS_KEY                                      | (Xtesting) Environnement variable used for S3 access secret                |
 107 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 108 | -e TEST_DB_URL                                                | (Xtesting) Environnement variable used to export NFVbench results in DB    |
 109 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 110 | -e NODE_NAME                                                  | (Xtesting) Environnement variable used as result key identifier in DB      |
 111 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 112 | -e BUILD_TAG                                                  | (Xtesting) Environnement variable used as result key identifier in DB      |
 113 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 114 | --privileged                                                  | (optional) required if SELinux is enabled on the host                      |
 115 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 116 | -v /lib/modules:/lib/modules                                  | needed by kernel modules in the container                                  |
 117 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 118 | -v /usr/src/kernels:/usr/src/kernels                          | needed by TRex to build kernel modules when needed                         |
 119 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 120 | -v /dev:/dev                                                  | needed by kernel modules in the container                                  |
 121 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 122 | -v $HOME/nfvbench:/tmp/nfvbench                               | folder mapping to pass files between the                                   |
 123 |                                                               | host and the docker space (see examples below)                             |
 124 |                                                               | Here we map the $HOME/nfvbench directory on the host                       |
 125 |                                                               | to the /tmp/nfvbench director in the container.                            |
 126 |                                                               | Any other mapping can work as well                                         |
 127 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 128 | -v $HOME/workspace/$JOB_NAME/results:/var/lib/xtesting/results| (Xtesting) folder mapping to pass files between the                        |
 129 |                                                               | CI chain workspace and the docker space to store Xtesting result files     |
 130 |                                                               | in orchestrator (Jenkins, Gitlab ...)                                      |
 131 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 132 | opnfv/nfvbench                                                | container image name                                                       |
 133 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 134 | run_tests                                                     | (Xtesting) Xtesting command to run test cases                              |
 135 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 136 | -t 10kpps-pvp-run                                             | (Xtesting) Xtesting parameter: Test case or tier (group of tests)          |
 137 |                                                               | to be executed. It will run all the test if not specified.                 |
 138 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 139 | -r                                                            | (Xtesting) Xtesting parameter: publish result to database                  |
 140 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 141 | -p                                                            | (Xtesting) Xtesting parameter: publish artifacts to a S3 service           |
 142 +---------------------------------------------------------------+----------------------------------------------------------------------------+
 143
 144 2. Run Xtesting test cases
 145
 146 Executed directly by NFVbench docker entrypoint after docker start.
 147
 148 3. Perform NFVbench test
 149
 150 Xtesting call NFVbench python script to execute test case scenario and wait for run to be terminated.
 151
 152 4. Export NFVbench result
 153
 154 If ``-r`` option is used, results are pushed to a DB through Xtesting.
 155 If ``-p`` option is used, results are pushed to a S3 service through Xtesting.
 156
 157
 158 Override testcases.yaml file
 159 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 160
 161 To replace existing testcases.yaml file, using Xtesting CI add the volume mapping in ``xtesting/ansible/host_vars/127.0.0.1`` file:
 162
 163 .. code-block:: bash
 164
 165     docker_args:
 166     env: {}
 167     volumes:
 168       - /lib/modules/$(uname -r):/lib/modules/$(uname -r)
 169       - /usr/src/kernels:/usr/src/kernels -v /dev:/dev
 170       - /home/opnfv/nfvbench:/tmp/nfvbench
 171       - /home/opnfv/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml
 172
 173 * ``/home/opnfv/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml`` : volume mapping to pass testcases.yaml file between the host and the docker space. Host path required testcases.yaml file inside.
 174
 175
 176 To replace existing testcases.yaml file, using NFVbench container:
 177
 178 .. code-block:: bash
 179
 180     docker run --name nfvbench --detach --privileged -v /lib/modules/$(uname -r):/lib/modules/$(uname -r) -v /usr/src/kernels:/usr/src/kernels -v /dev:/dev -v $HOME/nfvbench:/tmp/nfvbench \
 181     -v $HOME/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml \
 182     opnfv/nfvbench
 183
 184
 185 * ``$HOME/nfvbench/xtesting/testcases.yaml:/usr/local/lib/python3.6/dist-packages/xtesting/ci/testcases.yaml`` : volume mapping to pass testcases.yaml file between the host and the docker space. Host path required testcases.yaml file inside.
 186
 187
 188 Example of Xtesting test case
 189 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 190
 191 .. code-block:: bash
 192
 193     ---
 194     tiers:
 195         -
 196             name: nfvbench
 197             order: 1
 198             description: 'Data Plane Performance Testing'
 199             testcases:
 200                 -
 201                     case_name: 10kpps-pvp-run
 202                     project_name: nfvbench
 203                     criteria: 100
 204                     blocking: true
 205                     clean_flag: false
 206                     description: ''
 207                     run:
 208                         name: 'bashfeature'
 209                         args:
 210                             cmd:
 211                                 - nfvbench -c /tmp/nfvbench/nfvbench.cfg --rate 10kpps
 212
 213
 214 Examples of manual run
 215 ~~~~~~~~~~~~~~~~~~~~~~
 216
 217 If NFVbench container is already started in CLI mode (see Starting NFVbench in CLI mode dedicated chapter).
 218 To do a single run at 10,000pps bi-directional (or 5kpps in each direction) using the PVP packet path:
 219
 220 .. code-block:: bash
 221
 222    docker exec -it nfvbench run_tests -t 10kpps-pvp-run
 223
 224 Xtesting option used:
 225
 226 * ``-t 10kpps-pvp-run`` : specify the test case to run
 227
 228 To pass all test cases:
 229
 230 .. code-block:: bash
 231
 232    docker exec -it nfvbench run_tests -t all
 233
 234 Xtesting option used:
 235
 236 * ``-t all`` : select all test cases existing in testcases.yaml file
 237