Update tab results

It also removes all refs to components and to the former binaries.

Change-Id: I4f7e3308673ba6ba4dd9c28a7a053e3d558c289d
Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
(cherry picked from commit e108f1ad405e788e0516a55875269adafb338c28)

diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst
index d0c86f9..4938c57 100644 (file)
--- a/docs/release/release-notes/functest-release.rst
+++ b/docs/release/release-notes/functest-release.rst
@@ -95,7 +95,6 @@ Software
  * https://hub.docker.com/r/opnfv/functest-smoke
  * https://hub.docker.com/r/opnfv/functest-benchmarking
  * https://hub.docker.com/r/opnfv/functest-features
- * https://hub.docker.com/r/opnfv/functest-components
  * https://hub.docker.com/r/opnfv/functest-vnf
 
  Functest Docker images (Kubernetes):

diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst
index 9ca010b..07191dd 100644 (file)
--- a/docs/testing/developer/devguide/index.rst
+++ b/docs/testing/developer/devguide/index.rst
@@ -53,7 +53,6 @@ to the different tiers:
   * functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/
   * functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/
   * functest-features: https://hub.docker.com/r/opnfv/functest-features/
-  * functest-components: https://hub.docker.com/r/opnfv/functest-components/
   * functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/
   * functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/
 
@@ -163,8 +162,8 @@ The definition of the tiers has been agreed by the testing working group.
 The tiers are:
   * healthcheck
   * smoke
+  * benchmarking
   * features
-  * components
   * vnf
 
 Functest abstraction classes

diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst
index d3e770a..ba0b4bf 100644 (file)
--- a/docs/testing/user/configguide/configguide.rst
+++ b/docs/testing/user/configguide/configguide.rst
@@ -18,7 +18,6 @@ Docker images are available on the dockerhub:
   * opnfv/functest-smoke
   * opnfv/functest-benchmarking
   * opnfv/functest-features
-  * opnfv/functest-components
   * opnfv/functest-vnf
 
 
@@ -27,9 +26,10 @@ Preparing your environment
 
 cat env::
 
-  EXTERNAL_NETWORK=XXX
   DEPLOY_SCENARIO=XXX  # if not os-nosdn-nofeature-noha scenario
   NAMESERVER=XXX  # if not 8.8.8.8
+  EXTERNAL_NETWORK=XXX # if not first network with router:external=True
+  NEW_USER_ROLE=XXX # if not member
 
 See section on environment variables for details.
 
@@ -47,20 +47,17 @@ See section on OpenStack credentials for details.
 
 Create a directory for the different images (attached as a Docker volume)::
 
-  mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh?h=stable/fraser | bash -s -- images && ls -1 images/*
+  mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh?h=stable/iruya | bash -s -- images && ls -1 images/*
 
-  images/CentOS-7-aarch64-GenericCloud.qcow2
-  images/CentOS-7-aarch64-GenericCloud.qcow2.xz
-  images/CentOS-7-x86_64-GenericCloud.qcow2
+  images/cirros-0.4.0-aarch64-disk.img
   images/cirros-0.4.0-x86_64-disk.img
-  images/cirros-0.4.0-x86_64-lxc.tar.gz
-  images/cloudify-manager-premium-4.0.1.qcow2
+  images/cloudify-docker-manager-community-19.01.24.tar
   images/shaker-image-arm64.qcow2
-  images/shaker-image.qcow
+  images/shaker-image.qcow2
   images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
   images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
   images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
-  images/vyos-1.1.7.img
+  images/vyos-1.1.8-amd64.qcow2
 
 Testing healthcheck suite
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -74,23 +71,22 @@ Run healthcheck suite::
 
 Results shall be displayed as follows::
 
-  +----------------------------+------------------+---------------------+------------------+----------------+
-  |         TEST CASE          |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
-  +----------------------------+------------------+---------------------+------------------+----------------+
-  |      connection_check      |     functest     |     healthcheck     |      00:09       |      PASS      |
-  |       tenantnetwork1       |     functest     |     healthcheck     |      00:14       |      PASS      |
-  |       tenantnetwork2       |     functest     |     healthcheck     |      00:11       |      PASS      |
-  |          vmready1          |     functest     |     healthcheck     |      00:19       |      PASS      |
-  |          vmready2          |     functest     |     healthcheck     |      00:16       |      PASS      |
-  |         singlevm1          |     functest     |     healthcheck     |      00:41       |      PASS      |
-  |         singlevm2          |     functest     |     healthcheck     |      00:36       |      PASS      |
-  |         vping_ssh          |     functest     |     healthcheck     |      00:46       |      PASS      |
-  |       vping_userdata       |     functest     |     healthcheck     |      00:41       |      PASS      |
-  |        cinder_test         |     functest     |     healthcheck     |      01:18       |      PASS      |
-  |         api_check          |     functest     |     healthcheck     |      10:33       |      PASS      |
-  |     snaps_health_check     |     functest     |     healthcheck     |      00:44       |      PASS      |
-  |            odl             |     functest     |     healthcheck     |      00:00       |      SKIP      |
-  +----------------------------+------------------+---------------------+------------------+----------------+
+  +--------------------------+------------------+---------------------+------------------+----------------+
+  |        TEST CASE         |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
+  +--------------------------+------------------+---------------------+------------------+----------------+
+  |     connection_check     |     functest     |     healthcheck     |      00:01       |      PASS      |
+  |      tenantnetwork1      |     functest     |     healthcheck     |      00:06       |      PASS      |
+  |      tenantnetwork2      |     functest     |     healthcheck     |      00:06       |      PASS      |
+  |         vmready1         |     functest     |     healthcheck     |      00:08       |      PASS      |
+  |         vmready2         |     functest     |     healthcheck     |      00:08       |      PASS      |
+  |        singlevm1         |     functest     |     healthcheck     |      00:35       |      PASS      |
+  |        singlevm2         |     functest     |     healthcheck     |      00:38       |      PASS      |
+  |        vping_ssh         |     functest     |     healthcheck     |      00:47       |      PASS      |
+  |      vping_userdata      |     functest     |     healthcheck     |      00:33       |      PASS      |
+  |       cinder_test        |     functest     |     healthcheck     |      01:16       |      PASS      |
+  |      tempest_smoke       |     functest     |     healthcheck     |      07:05       |      PASS      |
+  |           odl            |     functest     |     healthcheck     |      00:00       |      SKIP      |
+  +--------------------------+------------------+---------------------+------------------+----------------+
 
 NOTE: the duration is a reference and it might vary depending on your SUT.
 
@@ -109,17 +105,16 @@ Results shall be displayed as follows::
   +------------------------------------+------------------+---------------+------------------+----------------+
   |             TEST CASE              |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
   +------------------------------------+------------------+---------------+------------------+----------------+
-  |           tempest_smoke            |     functest     |     smoke     |      06:13       |      PASS      |
-  |     neutron-tempest-plugin-api     |     functest     |     smoke     |      09:32       |      PASS      |
-  |            rally_sanity            |     functest     |     smoke     |      29:34       |      PASS      |
-  |             rally_jobs             |     functest     |     smoke     |      24:02       |      PASS      |
-  |          refstack_defcore          |     functest     |     smoke     |      13:07       |      PASS      |
-  |              patrole               |     functest     |     smoke     |      05:17       |      PASS      |
-  |            snaps_smoke             |     functest     |     smoke     |      90:13       |      PASS      |
+  |     neutron-tempest-plugin-api     |     functest     |     smoke     |      13:50       |      PASS      |
+  |            rally_sanity            |     functest     |     smoke     |      14:13       |      PASS      |
+  |          refstack_defcore          |     functest     |     smoke     |      09:60       |      PASS      |
+  |            tempest_full            |     functest     |     smoke     |      58:40       |      PASS      |
+  |          tempest_scenario          |     functest     |     smoke     |      09:47       |      PASS      |
+  |              patrole               |     functest     |     smoke     |      05:34       |      PASS      |
   |           neutron_trunk            |     functest     |     smoke     |      00:00       |      SKIP      |
-  |         networking-bgpvpn          |     functest     |     smoke     |      00:00       |      SKIP      |
-  |           networking-sfc           |     functest     |     smoke     |      00:00       |      SKIP      |
-  |              barbican              |     functest     |     smoke     |      05:01       |      PASS      |
+  |         networking-bgpvpn          |     functest     |     smoke     |      01:43       |      PASS      |
+  |           networking-sfc           |     functest     |     smoke     |      04:49       |      PASS      |
+  |              barbican              |     functest     |     smoke     |      05:09       |      PASS      |
   +------------------------------------+------------------+---------------+------------------+----------------+
 
 Note: if the scenario does not support some tests, they are indicated as SKIP.
@@ -137,12 +132,14 @@ Run benchmarking suite::
 
 Results shall be displayed as follows::
 
-  +-------------------+------------------+----------------------+------------------+----------------+
-  |     TEST CASE     |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
-  +-------------------+------------------+----------------------+------------------+----------------+
-  |        vmtp       |     functest     |     benchmarking     |      18:43       |      PASS      |
-  |       shaker      |     functest     |     benchmarking     |      29:45       |      PASS      |
-  +-------------------+------------------+----------------------+------------------+----------------+
+  +--------------------+------------------+----------------------+------------------+----------------+
+  |     TEST CASE      |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
+  +--------------------+------------------+----------------------+------------------+----------------+
+  |     rally_full     |     functest     |     benchmarking     |      121:55      |      PASS      |
+  |     rally_jobs     |     functest     |     benchmarking     |      24:21       |      PASS      |
+  |        vmtp        |     functest     |     benchmarking     |      00:00       |      SKIP      |
+  |       shaker       |     functest     |     benchmarking     |      00:00       |      SKIP      |
+  +--------------------+------------------+----------------------+------------------+----------------+
 
 Note: if the scenario does not support some tests, they are indicated as SKIP.
 See User guide for details.
@@ -159,41 +156,19 @@ Run features suite::
 
 Results shall be displayed as follows::
 
-  +-----------------------------+------------------------+------------------+------------------+----------------+
-  |          TEST CASE          |        PROJECT         |       TIER       |     DURATION     |     RESULT     |
-  +-----------------------------+------------------------+------------------+------------------+----------------+
-  |     doctor-notification     |         doctor         |     features     |      00:00       |      SKIP      |
-  |            bgpvpn           |         sdnvpn         |     features     |      00:00       |      SKIP      |
-  |       functest-odl-sfc      |          sfc           |     features     |      00:00       |      SKIP      |
-  |      barometercollectd      |       barometer        |     features     |      00:00       |      SKIP      |
-  |             fds             |     fastdatastacks     |     features     |      00:00       |      SKIP      |
-  |             vgpu            |        functest        |     features     |      00:00       |      SKIP      |
-  |         stor4nfv_os         |        stor4nfv        |     features     |      00:00       |      SKIP      |
-  +-----------------------------+------------------------+------------------+------------------+----------------+
+  +-----------------------------+-------------------+------------------+------------------+----------------+
+  |          TEST CASE          |      PROJECT      |       TIER       |     DURATION     |     RESULT     |
+  +-----------------------------+-------------------+------------------+------------------+----------------+
+  |     doctor-notification     |       doctor      |     features     |      00:00       |      SKIP      |
+  |       functest-odl-sfc      |        sfc        |     features     |      00:00       |      SKIP      |
+  |      barometercollectd      |     barometer     |     features     |      00:00       |      SKIP      |
+  |             vgpu            |      functest     |     features     |      00:00       |      SKIP      |
+  |         stor4nfv_os         |      stor4nfv     |     features     |      00:00       |      SKIP      |
+  +-----------------------------+-------------------+------------------+------------------+----------------+
 
 Note: if the scenario does not support some tests, they are indicated as SKIP.
 See User guide for details.
 
-Testing components suite
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Run components suite::
-
-  sudo docker run --env-file env \
-      -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
-      -v $(pwd)/images:/home/opnfv/functest/images \
-      opnfv/functest-components
-
-Results shall be displayed as follows::
-
-  +--------------------------+------------------+--------------------+------------------+----------------+
-  |        TEST CASE         |     PROJECT      |        TIER        |     DURATION     |     RESULT     |
-  +--------------------------+------------------+--------------------+------------------+----------------+
-  |       tempest_full       |     functest     |     components     |      49:51       |      PASS      |
-  |     tempest_scenario     |     functest     |     components     |      18:50       |      PASS      |
-  |        rally_full        |     functest     |     components     |      167:13      |      PASS      |
-  +--------------------------+------------------+--------------------+------------------+----------------+
-
 Testing vnf suite
 ^^^^^^^^^^^^^^^^^
 
@@ -209,11 +184,11 @@ Results shall be displayed as follows::
   +----------------------+------------------+--------------+------------------+----------------+
   |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
   +----------------------+------------------+--------------+------------------+----------------+
-  |       cloudify       |     functest     |     vnf      |      04:05       |      PASS      |
-  |     cloudify_ims     |     functest     |     vnf      |      24:07       |      PASS      |
-  |       heat_ims       |     functest     |     vnf      |      18:15       |      PASS      |
-  |     vyos_vrouter     |     functest     |     vnf      |      15:48       |      PASS      |
-  |       juju_epc       |     functest     |     vnf      |      29:38       |      PASS      |
+  |       cloudify       |     functest     |     vnf      |      06:42       |      PASS      |
+  |     cloudify_ims     |     functest     |     vnf      |      39:21       |      PASS      |
+  |       heat_ims       |     functest     |     vnf      |      34:33       |      PASS      |
+  |     vyos_vrouter     |     functest     |     vnf      |      20:20       |      PASS      |
+  |       juju_epc       |     functest     |     vnf      |      56:55       |      PASS      |
   +----------------------+------------------+--------------+------------------+----------------+
 
 Functest Dockers for Kubernetes deployment

diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst
index bfe8c95..ecbc7f5 100644 (file)
--- a/docs/testing/user/userguide/index.rst
+++ b/docs/testing/user/userguide/index.rst
@@ -12,7 +12,6 @@ Functest User Guide
    intro.rst
    test_overview.rst
    test_details.rst
-   runfunctest.rst
    test_results.rst
    reporting.rst
    troubleshooting.rst

diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst
deleted file mode 100644 (file)
index aea8c02..0000000
--- a/docs/testing/user/userguide/runfunctest.rst
+++ /dev/null
@@ -1,114 +0,0 @@
-.. SPDX-License-Identifier: CC-BY-4.0
-
-Executing Functest suites
-=========================
-
-As mentioned in the :ref:`functest-install-guide`, Alpine docker containers
-have been introduced in Euphrates.
-Tier containers have been created.
-Assuming that you pulled the container and your environement is ready, you can
-simply run the tiers by typing (e.g. with functest-healthcheck)::
-
-  sudo docker run --env-file env \
-      -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file  \
-      -v $(pwd)/images:/home/opnfv/functest/images  \
-      opnfv/functest-healthcheck
-
-You should get::
-
-  +----------------------------+------------------+---------------------+------------------+----------------+
-  |         TEST CASE          |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
-  +----------------------------+------------------+---------------------+------------------+----------------+
-  |      connection_check      |     functest     |     healthcheck     |      00:02       |      PASS      |
-  |         api_check          |     functest     |     healthcheck     |      03:19       |      PASS      |
-  |     snaps_health_check     |     functest     |     healthcheck     |      00:46       |      PASS      |
-  +----------------------------+------------------+---------------------+------------------+----------------+
-
-You can run functest-healcheck, functest-smoke, functest-features,
-functest-components and functest-vnf.
-
-The result tables show the results by test case, it can be::
-
-  * PASS
-  * FAIL
-  * SKIP: if the scenario/installer does not support the test case
-
-
-Manual run
-==========
-If you want to run the test step by step, you may add docker option then run
-the different commands within the docker.
-
-Considering the healthcheck example, running functest manaully means::
-
-  sudo docker run -ti --env-file env \
-    -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file  \
-    -v $(pwd)/images:/home/opnfv/functest/images  \
-    opnfv/functest-healthcheck /bin/bash
-
-The docker prompt shall be returned. Then within the docker run the following
-commands::
-
-  $ source /home/opnfv/functest/conf/env_file
-
-Tier
-----
-Each Alpine container provided on the docker hub matches with a tier.
-The following commands are available::
-
-  # functest tier list
-    - 0. healthcheck:
-    ['connection_check', 'api_check', 'snaps_health_check']
-  # functest tier show healthcheck
-  +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+
-  |        TIERS        |     ORDER     |         CI LOOP          |                   DESCRIPTION                   |             TESTCASES              |
-  +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+
-  |     healthcheck     |       0       |     (daily)|(weekly)     |     First tier to be executed to verify the     |     connection_check api_check     |
-  |                     |               |                          |           basic operations in the VIM.          |         snaps_health_check         |
-  +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+
-
-To run all the cases of the tier, type::
-
-  # functest tier run healthcheck
-
-Testcase
---------
-Testcases can be listed, shown and run though the CLI::
-
-  # functest testcase list
-   connection_check
-   api_check
-   snaps_health_check
-  # functest testcase show api_check
-  +-------------------+--------------------------------------------------+------------------+---------------------------+
-  |     TEST CASE     |                   DESCRIPTION                    |     CRITERIA     |         DEPENDENCY        |
-  +-------------------+--------------------------------------------------+------------------+---------------------------+
-  |     api_check     |     This test case verifies the retrieval of     |       100        |       ^((?!lxd).)*$       |
-  |                   |       OpenStack clients: Keystone, Glance,       |                  |                           |
-  |                   |      Neutron and Nova and may perform some       |                  |                           |
-  |                   |     simple queries. When the config value of     |                  |                           |
-  |                   |       snaps.use_keystone is True, functest       |                  |                           |
-  |                   |     must have access to the cloud's private      |                  |                           |
-  |                   |                     network.                     |                  |                           |
-  +-------------------+--------------------------------------------------+------------------+---------------------------+
-  # functest testcase run connection_check
-  ...
-  # functest run all
-
-You can also type run_tests -t all to run all the tests.
-
-Note the list of test cases depend on the installer and the scenario.
-
-Note that the flavors for the SNAPS test cases are able to be configured giving
-new metadata values as well as new values for the basic elements of flavor
-(i.e. ram, vcpu, disk, ephemeral, swap etc).
-The snaps.flavor_extra_specs dict in the config_functest.yaml file could be
-used for this purpose.
-
-Reporting results to the test Database
-======================================
-In OPNFV CI we collect all the results from CI. A test API shall be available
-as well as a test database `[16]`_.
-
-
-.. _`[16]`: http://testresults.opnfv.org/test/

diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst
index 5755544..b0711be 100644 (file)
--- a/docs/testing/user/userguide/troubleshooting.rst
+++ b/docs/testing/user/userguide/troubleshooting.rst
@@ -258,8 +258,8 @@ debug information can be found from tempest.log file stored into related Rally
 deployment folder.
 
 Functest offers a possibility to test a customized list of Tempest test cases.
-To enable that, add a new entry in docker/components/testcases.yaml on the
-"components" container with the following content::
+To enable that, add a new entry in docker/smoke/testcases.yaml on the
+"smoke" container with the following content::
 
     -
         case_name: tempest_custom
@@ -292,7 +292,7 @@ This is an example of running a customized list of Tempest tests in Functest::
       -v $(pwd)/images:/home/opnfv/functest/images \
       -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python2.7/site-packages/functest/ci/testcases.yaml \
       -v $(pwd)/my-custom-tempest-tests.txt:/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt \
-      opnfv/functest-components run_tests -t tempest_custom
+      opnfv/functest-smoke run_tests -t tempest_custom
 
 
 Rally