README.md

   1 # Functest
   2
   3 Network virtualization has dramatically modified our architectures which asks
   4 for more automation and powerful testing tools like Functest, a collection of
   5 state-of-the-art virtual infrastructure test suites, including automatic VNF
   6 testing (cf.
   7 [[1]](https://www.linuxfoundation.org/press-release/2019/05/opnfv-hunter-delivers-test-tools-ci-cd-framework-to-enable-common-nfvi-for-verifying-vnfs/)).
   8
   9 In context of OPNFV, Functest verifies any kind of OpenStack and Kubernetes
  10 deployments including production environments. It conforms to upstream rules
  11 and integrates smoothly lots of the test cases available in the opensource
  12 market. It includes about 3000+ functional tests and 3 hours upstream API and
  13 dataplane benchmarks. It’s completed by Virtual Network Function deployments
  14 and testing (vIMS, vRouter and vEPC) to ensure that the platforms meet Network
  15 Functions Virtualization requirements. Raspberry PI is also supported to verify
  16 datacenters as the lowest cost (50 euros hardware and software included).
  17
  18 | Functest releases | OpenStack releases |
  19 |-------------------|--------------------|
  20 | Jerma             | Train              |
  21 | Kali              | Ussuri             |
  22 | Leguer            | Victoria           |
  23 | Wallaby           | Wallaby            |
  24 | **Master**        | **next Xena**      |
  25
  26 ## Prepare your environment
  27
  28 cat env
  29 ```
  30 DEPLOY_SCENARIO=XXX  # if not os-nosdn-nofeature-noha scenario
  31 NAMESERVER=XXX  # if not 8.8.8.8
  32 EXTERNAL_NETWORK=XXX  # if not first network with router:external=True
  33 DASHBOARD_URL=XXX  # else tempest_horizon will be skipped
  34 NEW_USER_ROLE=XXX  # if not member
  35 SDN_CONTROLLER_IP=XXX  # if odl scenario
  36 VOLUME_DEVICE_NAME=XXX  # if not vdb
  37 FLAVOR_EXTRA_SPECS=hw:mem_page_size:large  # if fdio scenarios
  38 ```
  39
  40 cat openstack.creds
  41 ```
  42 export OS_AUTH_URL=XXX
  43 export OS_USER_DOMAIN_NAME=XXX
  44 export OS_PROJECT_DOMAIN_NAME=XXX
  45 export OS_USERNAME=XXX
  46 export OS_PROJECT_NAME=XXX
  47 export OS_PASSWORD=XXX
  48 export OS_IDENTITY_API_VERSION=3
  49 export OS_REGION_NAME=XXX
  50 ```
  51
  52 mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh | sh -s -- images && ls -1 images/*
  53 ```
  54 images/cirros-0.6.1-aarch64-disk.img
  55 images/cirros-0.6.1-x86_64-disk.img
  56 images/cloudify-docker-manager-community-19.01.24.tar
  57 images/Fedora-Cloud-Base-30-1.2.x86_64.qcow2
  58 images/shaker-image-1.3.0+stretch.qcow2
  59 images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
  60 images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
  61 images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
  62 images/vyos-1.1.8-amd64.qcow2
  63 ```
  64
  65 ## Run healthcheck suite
  66
  67 ```shell
  68 sudo docker run --env-file env \
  69     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
  70     -v $(pwd)/images:/home/opnfv/functest/images \
  71     opnfv/functest-healthcheck
  72 ```
  73
  74 ```
  75 +--------------------------+------------------+---------------------+------------------+----------------+
  76 |        TEST CASE         |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
  77 +--------------------------+------------------+---------------------+------------------+----------------+
  78 |     connection_check     |     functest     |     healthcheck     |      00:03       |      PASS      |
  79 |      tenantnetwork1      |     functest     |     healthcheck     |      00:05       |      PASS      |
  80 |      tenantnetwork2      |     functest     |     healthcheck     |      00:06       |      PASS      |
  81 |         vmready1         |     functest     |     healthcheck     |      00:06       |      PASS      |
  82 |         vmready2         |     functest     |     healthcheck     |      00:08       |      PASS      |
  83 |        singlevm1         |     functest     |     healthcheck     |      00:32       |      PASS      |
  84 |        singlevm2         |     functest     |     healthcheck     |      00:37       |      PASS      |
  85 |        vping_ssh         |     functest     |     healthcheck     |      00:46       |      PASS      |
  86 |      vping_userdata      |     functest     |     healthcheck     |      00:39       |      PASS      |
  87 |       cinder_test        |     functest     |     healthcheck     |      01:05       |      PASS      |
  88 |      tempest_smoke       |     functest     |     healthcheck     |      05:39       |      PASS      |
  89 |     tempest_horizon      |     functest     |     healthcheck     |      01:05       |      PASS      |
  90 |           odl            |     functest     |     healthcheck     |      00:00       |      SKIP      |
  91 +--------------------------+------------------+---------------------+------------------+----------------+
  92 ```
  93
  94 ## Run smoke suite
  95
  96 ```shell
  97 sudo docker run --env-file env \
  98     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
  99     -v $(pwd)/images:/home/opnfv/functest/images \
 100     opnfv/functest-smoke
 101 ```
 102
 103 ```
 104 +---------------------------+------------------+---------------+------------------+----------------+
 105 |         TEST CASE         |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
 106 +---------------------------+------------------+---------------+------------------+----------------+
 107 |      tempest_neutron      |     functest     |     smoke     |      15:30       |      PASS      |
 108 |       tempest_cinder      |     functest     |     smoke     |      02:01       |      PASS      |
 109 |      tempest_keystone     |     functest     |     smoke     |      01:17       |      PASS      |
 110 |        tempest_heat       |     functest     |     smoke     |      22:14       |      PASS      |
 111 |     tempest_telemetry     |     functest     |     smoke     |      00:00       |      SKIP      |
 112 |        rally_sanity       |     functest     |     smoke     |      17:24       |      PASS      |
 113 |      refstack_compute     |     functest     |     smoke     |      07:03       |      PASS      |
 114 |      refstack_object      |     functest     |     smoke     |      02:09       |      PASS      |
 115 |     refstack_platform     |     functest     |     smoke     |      07:31       |      PASS      |
 116 |        tempest_full       |     functest     |     smoke     |      41:52       |      PASS      |
 117 |      tempest_scenario     |     functest     |     smoke     |      08:42       |      PASS      |
 118 |        tempest_slow       |     functest     |     smoke     |      43:42       |      PASS      |
 119 |       patrole_admin       |     functest     |     smoke     |      21:06       |      PASS      |
 120 |       patrole_member      |     functest     |     smoke     |      21:23       |      PASS      |
 121 |       patrole_reader      |     functest     |     smoke     |      21:56       |      PASS      |
 122 |      tempest_barbican     |     functest     |     smoke     |      02:30       |      PASS      |
 123 |      tempest_octavia      |     functest     |     smoke     |      00:00       |      SKIP      |
 124 |       tempest_cyborg      |     functest     |     smoke     |      00:00       |      SKIP      |
 125 +---------------------------+------------------+---------------+------------------+----------------+
 126 ```
 127
 128 ## Run smoke CNTT suite
 129
 130 ```shell
 131 sudo docker run --env-file env \
 132     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
 133     -v $(pwd)/images:/home/opnfv/functest/images \
 134     opnfv/functest-smoke-cntt
 135 ```
 136
 137 ```
 138 +-------------------------------+------------------+---------------+------------------+----------------+
 139 |           TEST CASE           |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
 140 +-------------------------------+------------------+---------------+------------------+----------------+
 141 |      tempest_neutron_cntt     |     functest     |     smoke     |      11:35       |      PASS      |
 142 |      tempest_cinder_cntt      |     functest     |     smoke     |      01:58       |      PASS      |
 143 |     tempest_keystone_cntt     |     functest     |     smoke     |      01:13       |      PASS      |
 144 |       tempest_heat_cntt       |     functest     |     smoke     |      22:32       |      PASS      |
 145 |       rally_sanity_cntt       |     functest     |     smoke     |      17:16       |      PASS      |
 146 |       tempest_full_cntt       |     functest     |     smoke     |      41:13       |      PASS      |
 147 |     tempest_scenario_cntt     |     functest     |     smoke     |      08:57       |      PASS      |
 148 |       tempest_slow_cntt       |     functest     |     smoke     |      35:58       |      PASS      |
 149 +-------------------------------+------------------+---------------+------------------+----------------+
 150 ```
 151
 152 ## Run benchmarking suite
 153
 154 ```shell
 155 sudo docker run --env-file env \
 156     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
 157     -v $(pwd)/images:/home/opnfv/functest/images \
 158     opnfv/functest-benchmarking
 159 ```
 160
 161 ```
 162 +--------------------+------------------+----------------------+------------------+----------------+
 163 |     TEST CASE      |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
 164 +--------------------+------------------+----------------------+------------------+----------------+
 165 |     rally_full     |     functest     |     benchmarking     |      93:03       |      PASS      |
 166 |     rally_jobs     |     functest     |     benchmarking     |      27:05       |      PASS      |
 167 |        vmtp        |     functest     |     benchmarking     |      17:56       |      PASS      |
 168 |       shaker       |     functest     |     benchmarking     |      24:02       |      PASS      |
 169 +--------------------+------------------+----------------------+------------------+----------------+
 170 ```
 171
 172 ## Run benchmarking CNTT suite
 173
 174 ```shell
 175 sudo docker run --env-file env \
 176     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
 177     -v $(pwd)/images:/home/opnfv/functest/images \
 178     opnfv/functest-benchmarking-cntt
 179 ```
 180
 181 ```
 182 +-------------------------+------------------+----------------------+------------------+----------------+
 183 |        TEST CASE        |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
 184 +-------------------------+------------------+----------------------+------------------+----------------+
 185 |     rally_full_cntt     |     functest     |     benchmarking     |      89:52       |      PASS      |
 186 |     rally_jobs_cntt     |     functest     |     benchmarking     |      19:39       |      PASS      |
 187 |           vmtp          |     functest     |     benchmarking     |      16:59       |      PASS      |
 188 |          shaker         |     functest     |     benchmarking     |      23:43       |      PASS      |
 189 +-------------------------+------------------+----------------------+------------------+----------------+
 190 ```
 191
 192 ## Run vnf suite
 193
 194 ```shell
 195 sudo docker run --env-file env \
 196     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
 197     -v $(pwd)/images:/home/opnfv/functest/images \
 198     opnfv/functest-vnf
 199 ```
 200
 201 ```
 202 +----------------------+------------------+--------------+------------------+----------------+
 203 |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
 204 +----------------------+------------------+--------------+------------------+----------------+
 205 |       cloudify       |     functest     |     vnf      |      05:08       |      PASS      |
 206 |     cloudify_ims     |     functest     |     vnf      |      24:46       |      PASS      |
 207 |       heat_ims       |     functest     |     vnf      |      33:12       |      PASS      |
 208 |     vyos_vrouter     |     functest     |     vnf      |      15:53       |      PASS      |
 209 |       juju_epc       |     functest     |     vnf      |      27:52       |      PASS      |
 210 +----------------------+------------------+--------------+------------------+----------------+
 211 ```