Code Review / functest.git / blobdiff
? search:
summary | shortlog | log | commit | commitdiff | review | tree
raw | inline | side by side
Updates all testcases and their results
[functest.git] / docs / testing / user / configguide / configguide.rst
diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst
index 378aa2e..25aa7cd 100644 (file)
--- a/docs/testing/user/configguide/configguide.rst
+++ b/docs/testing/user/configguide/configguide.rst
@@ -16,19 +16,8 @@ Docker images are available on the dockerhub:
   * opnfv/functest-core
   * opnfv/functest-healthcheck
   * opnfv/functest-smoke
   * opnfv/functest-core
   * opnfv/functest-healthcheck
   * opnfv/functest-smoke
-  * opnfv/functest-features
-  * opnfv/functest-components
+  * opnfv/functest-benchmarking
   * opnfv/functest-vnf
   * opnfv/functest-vnf
-  * opnfv/functest-parser
-
-The tag "opnfv-6.0.0" is the official release image in Fraser, but you can
-also pull "fraser" tag as it is being maintained by Functest team and might
-include bugfixes.
-
-The Functest docker container environment can -in principle- be also
-used with non-OPNFV official installers (e.g. 'devstack'), with the
-**disclaimer** that support for such environments is outside of the
-scope and responsibility of the OPNFV project.
 
 
 Preparing your environment
 
 
 Preparing your environment
@@ -36,9 +25,13 @@ Preparing your environment
 
 cat env::
 
 
 cat env::
 
-  EXTERNAL_NETWORK=XXX
   DEPLOY_SCENARIO=XXX  # if not os-nosdn-nofeature-noha scenario
   NAMESERVER=XXX  # if not 8.8.8.8
   DEPLOY_SCENARIO=XXX  # if not os-nosdn-nofeature-noha scenario
   NAMESERVER=XXX  # if not 8.8.8.8
+  EXTERNAL_NETWORK=XXX # if not first network with router:external=True
+  NEW_USER_ROLE=XXX # if not member
+  SDN_CONTROLLER_IP=XXX  # if odl scenario
+  VOLUME_DEVICE_NAME=XXX  # if not vdb
+  FLAVOR_EXTRA_SPECS=hw:mem_page_size:large    # if fdio scenarios
 
 See section on environment variables for details.
 
 
 See section on environment variables for details.
 
@@ -51,26 +44,22 @@ cat env_file::
   export OS_PROJECT_NAME=XXX
   export OS_PASSWORD=XXX
   export OS_IDENTITY_API_VERSION=3
   export OS_PROJECT_NAME=XXX
   export OS_PASSWORD=XXX
   export OS_IDENTITY_API_VERSION=3
+  export OS_REGION_NAME=XXX
 
 See section on OpenStack credentials for details.
 
 Create a directory for the different images (attached as a Docker volume)::
 
 
 See section on OpenStack credentials for details.
 
 Create a directory for the different images (attached as a Docker volume)::
 
-  mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh?h=stable/fraser | bash -s -- images && ls -1 images/*
+  mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh | bash -s -- images && ls -1 images/*
 
 
-  images/CentOS-7-aarch64-GenericCloud.qcow2
-  images/CentOS-7-aarch64-GenericCloud.qcow2.xz
-  images/CentOS-7-x86_64-GenericCloud.qcow2
+  images/cirros-0.4.0-aarch64-disk.img
   images/cirros-0.4.0-x86_64-disk.img
   images/cirros-0.4.0-x86_64-disk.img
-  images/cirros-0.4.0-x86_64-lxc.tar.gz
-  images/cirros-d161201-aarch64-disk.img
-  images/cirros-d161201-aarch64-initramfs
-  images/cirros-d161201-aarch64-kernel
-  images/cloudify-manager-premium-4.0.1.qcow2
+  images/cloudify-docker-manager-community-19.01.24.tar
+  images/shaker-image-1.3.0+stretch.qcow2
   images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
   images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
   images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
   images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
   images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
   images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
-  images/vyos-1.1.7.img
+  images/vyos-1.1.8-amd64.qcow2
 
 Testing healthcheck suite
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Testing healthcheck suite
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -84,14 +73,24 @@ Run healthcheck suite::
 
 Results shall be displayed as follows::
 
 
 Results shall be displayed as follows::
 
-  +----------------------------+------------------+---------------------+------------------+----------------+
-  |         TEST CASE          |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
-  +----------------------------+------------------+---------------------+------------------+----------------+
-  |      connection_check      |     functest     |     healthcheck     |      00:07       |      PASS      |
-  |         api_check          |     functest     |     healthcheck     |      07:46       |      PASS      |
-  |     snaps_health_check     |     functest     |     healthcheck     |      00:36       |      PASS      |
-  +----------------------------+------------------+---------------------+------------------+----------------+
-  NOTE: the duration is a reference and it might vary depending on your SUT.
+  +--------------------------+------------------+---------------------+------------------+----------------+
+  |        TEST CASE         |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
+  +--------------------------+------------------+---------------------+------------------+----------------+
+  |     connection_check     |     functest     |     healthcheck     |      00:02       |      PASS      |
+  |      tenantnetwork1      |     functest     |     healthcheck     |      00:06       |      PASS      |
+  |      tenantnetwork2      |     functest     |     healthcheck     |      00:06       |      PASS      |
+  |         vmready1         |     functest     |     healthcheck     |      00:06       |      PASS      |
+  |         vmready2         |     functest     |     healthcheck     |      00:08       |      PASS      |
+  |        singlevm1         |     functest     |     healthcheck     |      00:28       |      PASS      |
+  |        singlevm2         |     functest     |     healthcheck     |      00:25       |      PASS      |
+  |        vping_ssh         |     functest     |     healthcheck     |      00:36       |      PASS      |
+  |      vping_userdata      |     functest     |     healthcheck     |      00:34       |      PASS      |
+  |       cinder_test        |     functest     |     healthcheck     |      01:03       |      PASS      |
+  |      tempest_smoke       |     functest     |     healthcheck     |      05:13       |      PASS      |
+  |           odl            |     functest     |     healthcheck     |      00:00       |      SKIP      |
+  +--------------------------+------------------+---------------------+------------------+----------------+
+
+NOTE: the duration is a reference and it might vary depending on your SUT.
 
 Testing smoke suite
 ^^^^^^^^^^^^^^^^^^^
 
 Testing smoke suite
 ^^^^^^^^^^^^^^^^^^^
@@ -105,64 +104,51 @@ Run smoke suite::
 
 Results shall be displayed as follows::
 
 
 Results shall be displayed as follows::
 
-  +------------------------------+------------------+---------------+------------------+----------------+
-  |          TEST CASE           |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
-  +------------------------------+------------------+---------------+------------------+----------------+
-  |          vping_ssh           |     functest     |     smoke     |      00:57       |      PASS      |
-  |        vping_userdata        |     functest     |     smoke     |      00:33       |      PASS      |
-  |     tempest_smoke_serial     |     functest     |     smoke     |      13:22       |      PASS      |
-  |         rally_sanity         |     functest     |     smoke     |      24:07       |      PASS      |
-  |       refstack_defcore       |     functest     |     smoke     |      05:21       |      PASS      |
-  |           patrole            |     functest     |     smoke     |      04:29       |      PASS      |
-  |         snaps_smoke          |     functest     |     smoke     |      46:54       |      PASS      |
-  |             odl              |     functest     |     smoke     |      00:00       |      SKIP      |
-  |        neutron_trunk         |     functest     |     smoke     |      00:00       |      SKIP      |
-  +------------------------------+------------------+---------------+------------------+----------------+
-  Note: if the scenario does not support some tests, they are indicated as SKIP.
-  See User guide for details.
-
-Testing features suite
-^^^^^^^^^^^^^^^^^^^^^^
-
-Run features suite::
-
-  sudo docker run --env-file env \
-      -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
-      -v $(pwd)/images:/home/opnfv/functest/images \
-      opnfv/functest-features
-
-Results shall be displayed as follows::
-
-  +-----------------------------+------------------------+------------------+------------------+----------------+
-  |          TEST CASE          |        PROJECT         |       TIER       |     DURATION     |     RESULT     |
-  +-----------------------------+------------------------+------------------+------------------+----------------+
-  |     doctor-notification     |         doctor         |     features     |      00:00       |      SKIP      |
-  |            bgpvpn           |         sdnvpn         |     features     |      00:00       |      SKIP      |
-  |       functest-odl-sfc      |          sfc           |     features     |      00:00       |      SKIP      |
-  |      barometercollectd      |       barometer        |     features     |      00:00       |      SKIP      |
-  |             fds             |     fastdatastacks     |     features     |      00:00       |      SKIP      |
-  +-----------------------------+------------------------+------------------+------------------+----------------+
-  Note: if the scenario does not support some tests, they are indicated as SKIP.
-  See User guide for details.
-
-Testing components suite
-^^^^^^^^^^^^^^^^^^^^^^^^
+  +------------------------------------+------------------+---------------+------------------+----------------+
+  |             TEST CASE              |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
+  +------------------------------------+------------------+---------------+------------------+----------------+
+  |     neutron-tempest-plugin-api     |     functest     |     smoke     |      09:12       |      PASS      |
+  |            rally_sanity            |     functest     |     smoke     |      16:29       |      PASS      |
+  |          refstack_compute          |     functest     |     smoke     |      06:25       |      PASS      |
+  |          refstack_object           |     functest     |     smoke     |      01:54       |      PASS      |
+  |         refstack_platform          |     functest     |     smoke     |      06:52       |      PASS      |
+  |            tempest_full            |     functest     |     smoke     |      30:26       |      PASS      |
+  |          tempest_scenario          |     functest     |     smoke     |      09:23       |      PASS      |
+  |            tempest_slow            |     functest     |     smoke     |      24:42       |      PASS      |
+  |              patrole               |     functest     |     smoke     |      02:36       |      PASS      |
+  |              barbican              |     functest     |     smoke     |      02:13       |      PASS      |
+  |           neutron_trunk            |     functest     |     smoke     |      00:00       |      SKIP      |
+  |         networking-bgpvpn          |     functest     |     smoke     |      00:00       |      SKIP      |
+  |           networking-sfc           |     functest     |     smoke     |      00:00       |      SKIP      |
+  |              octavia               |     functest     |     smoke     |      00:00       |      SKIP      |
+  +------------------------------------+------------------+---------------+------------------+----------------+
+
+Note: if the scenario does not support some tests, they are indicated as SKIP.
+See User guide for details.
+
+Testing benchmarking suite
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 
-Run components suite::
+Run benchmarking suite::
 
   sudo docker run --env-file env \
       -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
       -v $(pwd)/images:/home/opnfv/functest/images \
 
   sudo docker run --env-file env \
       -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
       -v $(pwd)/images:/home/opnfv/functest/images \
-      opnfv/functest-components
+      opnfv/functest-benchmarking
 
 Results shall be displayed as follows::
 
 
 Results shall be displayed as follows::
 
-  +-------------------------------+------------------+--------------------+------------------+----------------+
-  |           TEST CASE           |     PROJECT      |        TIER        |     DURATION     |     RESULT     |
-  +-------------------------------+------------------+--------------------+------------------+----------------+
-  |     tempest_full_parallel     |     functest     |     components     |      48:28       |      PASS      |
-  |           rally_full          |     functest     |     components     |      126:02      |      PASS      |
-  +-------------------------------+------------------+--------------------+------------------+----------------+
+  +--------------------+------------------+----------------------+------------------+----------------+
+  |     TEST CASE      |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
+  +--------------------+------------------+----------------------+------------------+----------------+
+  |     rally_full     |     functest     |     benchmarking     |      92:16       |      PASS      |
+  |     rally_jobs     |     functest     |     benchmarking     |      18:49       |      PASS      |
+  |        vmtp        |     functest     |     benchmarking     |      15:28       |      PASS      |
+  |       shaker       |     functest     |     benchmarking     |      24:04       |      PASS      |
+  +--------------------+------------------+----------------------+------------------+----------------+
+
+Note: if the scenario does not support some tests, they are indicated as SKIP.
+See User guide for details.
 
 Testing vnf suite
 ^^^^^^^^^^^^^^^^^
 
 Testing vnf suite
 ^^^^^^^^^^^^^^^^^
@@ -179,12 +165,13 @@ Results shall be displayed as follows::
   +----------------------+------------------+--------------+------------------+----------------+
   |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
   +----------------------+------------------+--------------+------------------+----------------+
   +----------------------+------------------+--------------+------------------+----------------+
   |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
   +----------------------+------------------+--------------+------------------+----------------+
-  |     cloudify_ims     |     functest     |     vnf      |      28:15       |      PASS      |
-  |     vyos_vrouter     |     functest     |     vnf      |      17:59       |      PASS      |
-  |       juju_epc       |     functest     |     vnf      |      46:44       |      PASS      |
+  |       cloudify       |     functest     |     vnf      |      03:49       |      PASS      |
+  |     cloudify_ims     |     functest     |     vnf      |      24:20       |      PASS      |
+  |       heat_ims       |     functest     |     vnf      |      32:13       |      PASS      |
+  |     vyos_vrouter     |     functest     |     vnf      |      14:55       |      PASS      |
+  |       juju_epc       |     functest     |     vnf      |      41:24       |      PASS      |
   +----------------------+------------------+--------------+------------------+----------------+
 
   +----------------------+------------------+--------------+------------------+----------------+
 
-
 Functest Dockers for Kubernetes deployment
 ------------------------------------------
 Docker images are available on the dockerhub:
 Functest Dockers for Kubernetes deployment
 ------------------------------------------
 Docker images are available on the dockerhub:
@@ -192,21 +179,13 @@ Docker images are available on the dockerhub:
   * opnfv/functest-kubernetes-core
   * opnfv/functest-kubernetest-healthcheck
   * opnfv/functest-kubernetest-smoke
   * opnfv/functest-kubernetes-core
   * opnfv/functest-kubernetest-healthcheck
   * opnfv/functest-kubernetest-smoke
-  * opnfv/functest-kubernetest-features
 
 Preparing your environment
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 cat env::
 
 
 Preparing your environment
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 cat env::
 
-  INSTALLER_TYPE=XXX
-  DEPLOY_SCENARIO=XXX
-
-cat k8s.creds::
-
-  export KUBERNETES_PROVIDER=local
-  export KUBE_MASTER_URL=XXX
-  export KUBE_MASTER_IP=XXX
+  DEPLOY_SCENARIO=k8s-XXX
 
 Testing healthcheck suite
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Testing healthcheck suite
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -214,7 +193,6 @@ Testing healthcheck suite
 Run healthcheck suite::
 
   sudo docker run -it --env-file env \
 Run healthcheck suite::
 
   sudo docker run -it --env-file env \
-      -v $(pwd)/k8s.creds:/home/opnfv/functest/conf/env_file \
       -v $(pwd)/config:/root/.kube/config \
       opnfv/functest-kubernetes-healthcheck
 
       -v $(pwd)/config:/root/.kube/config \
       opnfv/functest-kubernetes-healthcheck
 
@@ -223,11 +201,11 @@ volume mapped to ~/.kube/config inside kubernetes container.
 
 Results shall be displayed as follows::
 
 
 Results shall be displayed as follows::
 
-  +-------------------------+------------------+-----------------+------------------+----------------+
-  |        TEST CASE        |     PROJECT      |       TIER      |     DURATION     |     RESULT     |
-  +-------------------------+------------------+-----------------+------------------+----------------+
-  |        k8s_smoke        |     functest     |   healthcheck   |      01:54       |      PASS      |
-  +-------------------------+------------------+-----------------+------------------+----------------+
+  +-------------------+------------------+---------------------+------------------+----------------+
+  |     TEST CASE     |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
+  +-------------------+------------------+---------------------+------------------+----------------+
+  |     k8s_smoke     |     functest     |     healthcheck     |      01:09       |      PASS      |
+  +-------------------+------------------+---------------------+------------------+----------------+
 
 Testing smoke suite
 ^^^^^^^^^^^^^^^^^^^
 
 Testing smoke suite
 ^^^^^^^^^^^^^^^^^^^
@@ -235,45 +213,29 @@ Testing smoke suite
 Run smoke suite::
 
   sudo docker run -it --env-file env \
 Run smoke suite::
 
   sudo docker run -it --env-file env \
-      -v $(pwd)/k8s.creds:/home/opnfv/functest/conf/env_file \
       -v $(pwd)/config:/root/.kube/config \
       opnfv/functest-kubernetes-smoke
 
 Results shall be displayed as follows::
 
       -v $(pwd)/config:/root/.kube/config \
       opnfv/functest-kubernetes-smoke
 
 Results shall be displayed as follows::
 
-  +-------------------------+------------------+---------------+------------------+----------------+
-  |        TEST CASE        |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
-  +-------------------------+------------------+---------------+------------------+----------------+
-  |     k8s_conformance     |     functest     |     smoke     |      57:47       |      PASS      |
-  +-------------------------+------------------+---------------+------------------+----------------+
-
-Testing features suite
-^^^^^^^^^^^^^^^^^^^^^^
-
-Run features suite::
-
-  sudo docker run -it --env-file env \
-      -v $(pwd)/k8s.creds:/home/opnfv/functest/conf/env_file \
-      -v $(pwd)/config:/root/.kube/config \
-      opnfv/functest-kubernetes-features
-
-Results shall be displayed as follows::
-
-  +----------------------+------------------+------------------+------------------+----------------+
-  |      TEST CASE       |     PROJECT      |       TIER       |     DURATION     |     RESULT     |
-  +----------------------+------------------+------------------+------------------+----------------+
-  |     stor4nfv_k8s     |     stor4nfv     |     stor4nfv     |      00:00       |      SKIP      |
-  |      clover_k8s      |      clover      |      clover      |      00:00       |      SKIP      |
-  +----------------------+------------------+------------------+------------------+----------------+
+  +---------------------------+------------------+---------------+------------------+----------------+
+  |         TEST CASE         |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
+  +---------------------------+------------------+---------------+------------------+----------------+
+  |     xrally_kubernetes     |     functest     |     smoke     |      22:04       |      PASS      |
+  |      k8s_conformance      |     functest     |     smoke     |      173:48      |      PASS      |
+  +---------------------------+------------------+---------------+------------------+----------------+
 
 Environment variables
 =====================
 
 Several environement variables may be specified:
 
 Environment variables
 =====================
 
 Several environement variables may be specified:
-  * INSTALLER_TYPE=(apex|compass|daisy|fuel|joid)
+
   * INSTALLER_IP=<Specific IP Address>
   * DEPLOY_SCENARIO=<vim>-<controller>-<nfv_feature>-<ha_mode>
   * INSTALLER_IP=<Specific IP Address>
   * DEPLOY_SCENARIO=<vim>-<controller>-<nfv_feature>-<ha_mode>
-
+  * NAMESERVER=XXX  # if not 8.8.8.8
+  * VOLUME_DEVICE_NAME=XXX  # if not vdb
+  * EXTERNAL_NETWORK=XXX # if not first network with router:external=True
+  * NEW_USER_ROLE=XXX # if not member
 
 INSTALLER_IP is required by Barometer in order to access the installer node and
 the deployment.
 
 INSTALLER_IP is required by Barometer in order to access the installer node and
 the deployment.
@@ -306,8 +268,9 @@ to the Functest Docker Container, using the -e
 only relevant to Jenkins CI invoked testing and **should not be used**
 when performing manual test scenarios:
 
 only relevant to Jenkins CI invoked testing and **should not be used**
 when performing manual test scenarios:
 
-  * NODE_NAME = <Test POD Name>
-  * BUILD_TAG = <Jenkins Build Tag>
+  * INSTALLER_TYPE=(apex|compass|daisy|fuel)
+  * NODE_NAME=<Test POD Name>
+  * BUILD_TAG=<Jenkins Build Tag>
 
 where:
 
 
 where:
 
@@ -390,7 +353,7 @@ By default all the logs are put un /home/opnfv/functest/results/functest.log.
 If you want to have more logs in console, you may edit the logging.ini file
 manually.
 Connect on the docker then edit the file located in
 If you want to have more logs in console, you may edit the logging.ini file
 manually.
 Connect on the docker then edit the file located in
-/usr/lib/python2.7/site-packages/xtesting/ci/logging.ini
+/usr/lib/python3.6/site-packages/xtesting/ci/logging.ini
 
 Change wconsole to console in the desired module to get more traces.
 
 
 Change wconsole to console in the desired module to get more traces.
 
@@ -400,8 +363,8 @@ Configuration
 
 You may also directly modify the python code or the configuration file (e.g.
 testcases.yaml used to declare test constraints) under
 
 You may also directly modify the python code or the configuration file (e.g.
 testcases.yaml used to declare test constraints) under
-/usr/lib/python2.7/site-packages/xtesting and
-/usr/lib/python2.7/site-packages/functest
+/usr/lib/python3.6/site-packages/xtesting and
+/usr/lib/python3.6/site-packages/functest
 
 
 Tips
 
 
 Tips
@@ -550,7 +513,7 @@ docker container::
 Note: In a Jumphost node based on the CentOS family OS, the **nc**
 commands might not work. You can use the **curl** command instead.
 
 Note: In a Jumphost node based on the CentOS family OS, the **nc**
 commands might not work. You can use the **curl** command instead.
 
-  curl http://www.opnfv.org:80
+  curl https://www.opnfv.org/
 
   <HTML><HEAD><meta http-equiv="content-type"
   .
 
   <HTML><HEAD><meta http-equiv="content-type"
   .
@@ -608,7 +571,6 @@ and install the **docker-engine**. The instructions conclude with a
 work with the above pre-requisite actions.
 
 
 work with the above pre-requisite actions.
 
 
-.. _`[4]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
 .. _`dockerdocs`: https://docs.docker.com/
 .. _`Proxy`: https://docs.docker.com/engine/admin/systemd/#http-proxy
 .. _`Install Docker on CentOS`: https://docs.docker.com/engine/installation/linux/centos/
 .. _`dockerdocs`: https://docs.docker.com/
 .. _`Proxy`: https://docs.docker.com/engine/admin/systemd/#http-proxy
 .. _`Install Docker on CentOS`: https://docs.docker.com/engine/installation/linux/centos/
Test suites and cases to verify OPNFV Platform functionality