README.md

   1 # Functest
   2
   3 Network virtualization has dramatically modified our architectures which asks
   4 for more automation and powerful testing tools like Functest, a collection of
   5 state-of-the-art virtual infrastructure test suites, including automatic VNF
   6 testing (cf.
   7 [[1]](https://www.linuxfoundation.org/press-release/2019/05/opnfv-hunter-delivers-test-tools-ci-cd-framework-to-enable-common-nfvi-for-verifying-vnfs/)).
   8
   9 In context of OPNFV, Functest verifies any kind of OpenStack and Kubernetes
  10 deployments including production environments. It conforms to upstream rules
  11 and integrates smoothly lots of the test cases available in the opensource
  12 market. It includes about 3000+ functional tests and 3 hours upstream API and
  13 dataplane benchmarks. It’s completed by Virtual Network Function deployments
  14 and testing (vIMS, vRouter and vEPC) to ensure that the platforms meet Network
  15 Functions Virtualization requirements. Raspberry PI is also supported to verify
  16 datacenters as the lowest cost (50 euros hardware and software included).
  17
  18 | Functest releases | OpenStack releases |
  19 |-------------------|--------------------|
  20 | Hunter                  | Rocky              |
  21 | Iruya             | Stein              |
  22 | Jerma             | Train              |
  23 | Kali              | Ussuri             |
  24 | Leguer            | Victoria           |
  25 | **Master**          | **next Wallaby**   |
  26
  27 ## Prepare your environment
  28
  29 cat env
  30 ```
  31 DEPLOY_SCENARIO=XXX  # if not os-nosdn-nofeature-noha scenario
  32 NAMESERVER=XXX  # if not 8.8.8.8
  33 EXTERNAL_NETWORK=XXX  # if not first network with router:external=True
  34 DASHBOARD_URL=XXX  # else tempest_horizon will be skipped
  35 NEW_USER_ROLE=XXX  # if not member
  36 SDN_CONTROLLER_IP=XXX  # if odl scenario
  37 VOLUME_DEVICE_NAME=XXX  # if not vdb
  38 FLAVOR_EXTRA_SPECS=hw:mem_page_size:large  # if fdio scenarios
  39 ```
  40
  41 cat openstack.creds
  42 ```
  43 export OS_AUTH_URL=XXX
  44 export OS_USER_DOMAIN_NAME=XXX
  45 export OS_PROJECT_DOMAIN_NAME=XXX
  46 export OS_USERNAME=XXX
  47 export OS_PROJECT_NAME=XXX
  48 export OS_PASSWORD=XXX
  49 export OS_IDENTITY_API_VERSION=3
  50 export OS_REGION_NAME=XXX
  51 ```
  52
  53 mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh | bash -s -- images && ls -1 images/*
  54 ```
  55 images/cirros-0.5.1-aarch64-disk.img
  56 images/cirros-0.5.1-x86_64-disk.img
  57 images/cloudify-docker-manager-community-19.01.24.tar
  58 images/Fedora-Cloud-Base-30-1.2.x86_64.qcow2
  59 images/shaker-image-1.3.0+stretch.qcow2
  60 images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
  61 images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
  62 images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
  63 images/vyos-1.1.8-amd64.qcow2
  64 ```
  65
  66 ## Run healthcheck suite
  67
  68 ```bash
  69 sudo docker run --env-file env \
  70     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
  71     -v $(pwd)/images:/home/opnfv/functest/images \
  72     opnfv/functest-healthcheck
  73 ```
  74
  75 ```
  76 +--------------------------+------------------+---------------------+------------------+----------------+
  77 |        TEST CASE         |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
  78 +--------------------------+------------------+---------------------+------------------+----------------+
  79 |     connection_check     |     functest     |     healthcheck     |      00:03       |      PASS      |
  80 |      tenantnetwork1      |     functest     |     healthcheck     |      00:05       |      PASS      |
  81 |      tenantnetwork2      |     functest     |     healthcheck     |      00:06       |      PASS      |
  82 |         vmready1         |     functest     |     healthcheck     |      00:06       |      PASS      |
  83 |         vmready2         |     functest     |     healthcheck     |      00:08       |      PASS      |
  84 |        singlevm1         |     functest     |     healthcheck     |      00:32       |      PASS      |
  85 |        singlevm2         |     functest     |     healthcheck     |      00:37       |      PASS      |
  86 |        vping_ssh         |     functest     |     healthcheck     |      00:46       |      PASS      |
  87 |      vping_userdata      |     functest     |     healthcheck     |      00:39       |      PASS      |
  88 |       cinder_test        |     functest     |     healthcheck     |      01:05       |      PASS      |
  89 |      tempest_smoke       |     functest     |     healthcheck     |      05:39       |      PASS      |
  90 |     tempest_horizon      |     functest     |     healthcheck     |      01:05       |      PASS      |
  91 |           odl            |     functest     |     healthcheck     |      00:00       |      SKIP      |
  92 +--------------------------+------------------+---------------------+------------------+----------------+
  93 ```
  94
  95 ## Run smoke suite
  96
  97 ```bash
  98 sudo docker run --env-file env \
  99     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
 100     -v $(pwd)/images:/home/opnfv/functest/images \
 101     opnfv/functest-smoke
 102 ```
 103
 104 ```
 105 +---------------------------+------------------+---------------+------------------+----------------+
 106 |         TEST CASE         |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
 107 +---------------------------+------------------+---------------+------------------+----------------+
 108 |      tempest_neutron      |     functest     |     smoke     |      15:30       |      PASS      |
 109 |       tempest_cinder      |     functest     |     smoke     |      02:01       |      PASS      |
 110 |      tempest_keystone     |     functest     |     smoke     |      01:17       |      PASS      |
 111 |        tempest_heat       |     functest     |     smoke     |      22:14       |      PASS      |
 112 |     tempest_telemetry     |     functest     |     smoke     |      00:00       |      SKIP      |
 113 |        rally_sanity       |     functest     |     smoke     |      17:24       |      PASS      |
 114 |      refstack_compute     |     functest     |     smoke     |      07:03       |      PASS      |
 115 |      refstack_object      |     functest     |     smoke     |      02:09       |      PASS      |
 116 |     refstack_platform     |     functest     |     smoke     |      07:31       |      PASS      |
 117 |        tempest_full       |     functest     |     smoke     |      41:52       |      PASS      |
 118 |      tempest_scenario     |     functest     |     smoke     |      08:42       |      PASS      |
 119 |        tempest_slow       |     functest     |     smoke     |      43:42       |      PASS      |
 120 |       patrole_admin       |     functest     |     smoke     |      21:06       |      PASS      |
 121 |       patrole_member      |     functest     |     smoke     |      21:23       |      PASS      |
 122 |       patrole_reader      |     functest     |     smoke     |      21:56       |      PASS      |
 123 |      tempest_barbican     |     functest     |     smoke     |      02:30       |      PASS      |
 124 |      tempest_octavia      |     functest     |     smoke     |      00:00       |      SKIP      |
 125 |       tempest_cyborg      |     functest     |     smoke     |      00:00       |      SKIP      |
 126 +---------------------------+------------------+---------------+------------------+----------------+
 127 ```
 128
 129 ## Run smoke CNTT suite
 130
 131 ```bash
 132 sudo docker run --env-file env \
 133     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
 134     -v $(pwd)/images:/home/opnfv/functest/images \
 135     opnfv/functest-smoke-cntt
 136 ```
 137
 138 ```
 139 +-------------------------------+------------------+---------------+------------------+----------------+
 140 |           TEST CASE           |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
 141 +-------------------------------+------------------+---------------+------------------+----------------+
 142 |      tempest_neutron_cntt     |     functest     |     smoke     |      11:35       |      PASS      |
 143 |      tempest_cinder_cntt      |     functest     |     smoke     |      01:58       |      PASS      |
 144 |     tempest_keystone_cntt     |     functest     |     smoke     |      01:13       |      PASS      |
 145 |       tempest_heat_cntt       |     functest     |     smoke     |      22:32       |      PASS      |
 146 |       rally_sanity_cntt       |     functest     |     smoke     |      17:16       |      PASS      |
 147 |       tempest_full_cntt       |     functest     |     smoke     |      41:13       |      PASS      |
 148 |     tempest_scenario_cntt     |     functest     |     smoke     |      08:57       |      PASS      |
 149 |       tempest_slow_cntt       |     functest     |     smoke     |      35:58       |      PASS      |
 150 +-------------------------------+------------------+---------------+------------------+----------------+
 151 ```
 152
 153 ## Run benchmarking suite
 154
 155 ```bash
 156 sudo docker run --env-file env \
 157     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
 158     -v $(pwd)/images:/home/opnfv/functest/images \
 159     opnfv/functest-benchmarking
 160 ```
 161
 162 ```
 163 +--------------------+------------------+----------------------+------------------+----------------+
 164 |     TEST CASE      |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
 165 +--------------------+------------------+----------------------+------------------+----------------+
 166 |     rally_full     |     functest     |     benchmarking     |      93:03       |      PASS      |
 167 |     rally_jobs     |     functest     |     benchmarking     |      27:05       |      PASS      |
 168 |        vmtp        |     functest     |     benchmarking     |      17:56       |      PASS      |
 169 |       shaker       |     functest     |     benchmarking     |      24:02       |      PASS      |
 170 +--------------------+------------------+----------------------+------------------+----------------+
 171 ```
 172
 173 ## Run benchmarking CNTT suite
 174
 175 ```bash
 176 sudo docker run --env-file env \
 177     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
 178     -v $(pwd)/images:/home/opnfv/functest/images \
 179     opnfv/functest-benchmarking-cntt
 180 ```
 181
 182 ```
 183 +-------------------------+------------------+----------------------+------------------+----------------+
 184 |        TEST CASE        |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
 185 +-------------------------+------------------+----------------------+------------------+----------------+
 186 |     rally_full_cntt     |     functest     |     benchmarking     |      89:52       |      PASS      |
 187 |     rally_jobs_cntt     |     functest     |     benchmarking     |      19:39       |      PASS      |
 188 |           vmtp          |     functest     |     benchmarking     |      16:59       |      PASS      |
 189 |          shaker         |     functest     |     benchmarking     |      23:43       |      PASS      |
 190 +-------------------------+------------------+----------------------+------------------+----------------+
 191 ```
 192
 193 ## Run vnf suite
 194
 195 ```bash
 196 sudo docker run --env-file env \
 197     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
 198     -v $(pwd)/images:/home/opnfv/functest/images \
 199     opnfv/functest-vnf
 200 ```
 201
 202 ```
 203 +----------------------+------------------+--------------+------------------+----------------+
 204 |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
 205 +----------------------+------------------+--------------+------------------+----------------+
 206 |       cloudify       |     functest     |     vnf      |      05:08       |      PASS      |
 207 |     cloudify_ims     |     functest     |     vnf      |      24:46       |      PASS      |
 208 |       heat_ims       |     functest     |     vnf      |      33:12       |      PASS      |
 209 |     vyos_vrouter     |     functest     |     vnf      |      15:53       |      PASS      |
 210 |       juju_epc       |     functest     |     vnf      |      27:52       |      PASS      |
 211 +----------------------+------------------+--------------+------------------+----------------+
 212 ```