Including minor fixes and corrections.
Change-Id: I7c4bd9ca4f5fb35aeb2350096520c66b11aecbe8
Signed-off-by: ChristopherPrice <christopher.price@ericsson.com>
--- /dev/null
+Functional testing Installation
+===============================
+
+Pull the Functest Docker image from the Docker hub::
+
+ $ docker pull opnfv/functest:brahmaputra.1.0
+
+
+Check that the image is available::
+
+ $ docker images
+
+Run the docker container giving the environment variables::
+
+ - INSTALLER_TYPE. Possible values are "apex", "compass", "fuel" or "joid".
+ - INSTALLER_IP. each installer has its installation strategy.
+
+Functest may need to know the IP of the installer to retrieve the credentials
+(e.g. usually "10.20.0.2" for fuel, not neede for joid...).
+
+The minimum command to create the Functest docker file can be described as
+follows::
+
+ docker run -it -e "INSTALLER_IP=10.20.0.2" -e "INSTALLER_TYPE=fuel" opnfv/functest:brahmaputra.1.0 /bin/bash
+
+Optionally, it is possible to precise the container name through the option
+--name::
+
+ docker run --name "CONTAINER_NAME" -it -e "INSTALLER_IP=10.20.0.2" -e "INSTALLER_TYPE=fuel" opnfv/functest:brahmaputra.1.0 /bin/bash
+
+It is also possible to to indicate the path of the OpenStack creds using -v::
+
+ docker run -it -e "INSTALLER_IP=10.20.0.2" -e "INSTALLER_TYPE=fuel" -v <path_to_your_local_creds_file>:/home/opnfv/functest/conf/openstack.creds opnfv/functest:brahmaputra.1.0 /bin/bash
+
+The local file will be mounted in the container under
+/home/opnfv/functest/conf/openstack.creds
+
+After the run command the prompt appears which means that we are inside the
+container and ready to run Functest.
+
+Inside the container, the following directory structure should be in place::
+
+ `-- home
+ `-- opnfv
+ |-- functest
+ | |-- conf
+ | |-- data
+ | `-- results
+ `-- repos
+ |-- bgpvpn
+ |-- functest
+ |-- odl_integration
+ |-- rally
+ |-- releng
+ `-- vims-test
+
+
+Basically the container includes:
+
+ * Functest directory to store the configuration (the OpenStack creds are paste
+ in /home/opngb/functest/conf), the data (images neede for test for offline
+ testing), results (some temporary artifacts may be stored here)
+ * Repositories: the functest repository will be used to prepare the
+ environment, run the tests. Other repositories are used for the installation
+ of the tooling (e.g. rally) and/or the retrieval of feature projects
+ scenarios (e.g. bgpvpn)
+
+The arborescence under the functest repo can be described as follow::
+
+ .
+ |-- INFO
+ |-- LICENSE
+ |-- commons
+ | |-- ims
+ | |-- mobile
+ | `-- traffic-profile-guidelines.rst
+ |-- docker
+ | |-- Dockerfile
+ | |-- common.sh
+ | |-- prepare_env.sh
+ | |-- requirements.pip
+ | `-- run_tests.sh
+ |-- docs
+ | |-- configguide
+ | |-- functest.rst
+ | |-- images
+ | `-- userguide
+ `-- testcases
+ |-- Controllers
+ |-- VIM
+ |-- __init__.py
+ |-- config_functest.py
+ |-- config_functest.yaml
+ |-- functest_utils.py
+ |-- functest_utils.pyc
+ |-- vIMS
+ `-- vPing
+
+We may distinguish 4 different folders:
+
+ * commons: it is a folder dedicated to store traffic profile or any test
+ inputs that could be reused by any test project
+ * docker: this folder includes the scripts that will be used to setup the
+ environment and run the tests
+ * docs: this folder includes the user and installation/configuration guide
+ * testcases: this folder includes the scripts required by Functest internal
+ test cases
+
+
+Firstly run the script to install functest environment::
+
+ $ ${repos_dir}/functest/docker/prepare_env.sh
+
+NOTE: ${repos_dir} is a default environment variable inside the docker
+container, which points to /home/opnfv/repos
+
+Run the script to start the tests::
+
+ $ ${repos_dir}/functest/docker/run_tests.sh
would be possible to adapt it to any VIM+controller environment as most of the
testcases are upstream testcases.
-Manual Installation
-===================
-
-Pull the Functest Docker image from the Docker hub::
-
- $ docker pull opnfv/functest:brahmaputra.1.0
-
-
-Check that the image is available::
-
- $ docker images
-
-Run the docker container giving the environment variables::
-
- - INSTALLER_TYPE. Possible values are "apex", "compass", "fuel" or "joid".
- - INSTALLER_IP. each installer has its installation strategy.
-
-Functest may need to know the IP of the installer to retrieve the credentials
-(e.g. usually "10.20.0.2" for fuel, not neede for joid...).
-
-The minimum command to create the Functest docker file can be described as
-follows::
-
- docker run -it -e "INSTALLER_IP=10.20.0.2" -e "INSTALLER_TYPE=fuel" opnfv/functest:brahmaputra.1.0 /bin/bash
-
-Optionally, it is possible to precise the container name through the option
---name::
-
- docker run --name "CONTAINER_NAME" -it -e "INSTALLER_IP=10.20.0.2" -e "INSTALLER_TYPE=fuel" opnfv/functest:brahmaputra.1.0 /bin/bash
-
-It is also possible to to indicate the path of the OpenStack creds using -v::
-
- docker run -it -e "INSTALLER_IP=10.20.0.2" -e "INSTALLER_TYPE=fuel" -v <path_to_your_local_creds_file>:/home/opnfv/functest/conf/openstack.creds opnfv/functest:brahmaputra.1.0 /bin/bash
-
-The local file will be mounted in the container under
-/home/opnfv/functest/conf/openstack.creds
-
-After the run command the prompt appears which means that we are inside the
-container and ready to run Functest.
-
-Inside the container, you must have the following directory structure::
-
- `-- home
- `-- opnfv
- |-- functest
- | |-- conf
- | |-- data
- | `-- results
- `-- repos
- |-- bgpvpn
- |-- functest
- |-- odl_integration
- |-- rally
- |-- releng
- `-- vims-test
-
-
-Basically the container includes:
-
- * Functest directory to store the configuration (the OpenStack creds are paste
- in /home/opngb/functest/conf), the data (images neede for test for offline
- testing), results (some temporary artifacts may be stored here)
- * Repositories: the functest repository will be used to prepare the
- environment, run the tests. Other repositories are used for the installation
- of the tooling (e.g. rally) and/or the retrieval of feature projects
- scenarios (e.g. bgpvpn)
-
-The arborescence under the functest repo can be described as follow::
-
- .
- |-- INFO
- |-- LICENSE
- |-- commons
- | |-- ims
- | |-- mobile
- | `-- traffic-profile-guidelines.rst
- |-- docker
- | |-- Dockerfile
- | |-- common.sh
- | |-- prepare_env.sh
- | |-- requirements.pip
- | `-- run_tests.sh
- |-- docs
- | |-- configguide
- | |-- functest.rst
- | |-- images
- | `-- userguide
- `-- testcases
- |-- Controllers
- |-- VIM
- |-- __init__.py
- |-- config_functest.py
- |-- config_functest.yaml
- |-- functest_utils.py
- |-- functest_utils.pyc
- |-- vIMS
- `-- vPing
-
-We may distinguish 4 different folders:
-
- * commons: it is a folder dedicated to store traffic profile or any test
- inputs that could be reused by any test project
- * docker: this folder includes the scripts that will be used to setup the
- environment and run the tests
- * docs: this folder includes the user and installation/configuration guide
- * testcases: this folder includes the scripts required by Functest internal
- test cases
-
-
-Firstly run the script to install functest environment::
-
- $ ${repos_dir}/functest/docker/prepare_env.sh
-
-NOTE: ${repos_dir} is a default environment variable inside the docker
-container, which points to /home/opnfv/repos
-
-Run the script to start the tests::
-
- $ ${repos_dir}/functest/docker/run_tests.sh
+.. include:: ./configguide.rst
NOTE: This will run ALL the tests by default, see `[2]`_ for details
--- /dev/null
+Description of the test cases
+=============================
+
+Functest is an OPNFV project dedicated to functional testing.
+In the continuous integration, it is launched after an OPNFV fresh installation.
+The Functest target is to verify the basic functions of the infrastructure.
+
+Functest includes different test suites which several test cases within.
+Test cases are developed in Functest and in feature projects.
+
+The current list of test suites can be distributed in 3 main domains::
+
+ +----------------+----------------+--------------------------------------------+
+ | Domain | Test suite | Comments |
+ +================+================+============================================+
+ | | vPing | NFV "Hello World" |
+ | +----------------+--------------------------------------------+
+ | VIM | vPing_userdata | Ping using userdata and cloud-init |
+ | | | mechanism |
+ | +----------------+--------------------------------------------+
+ |(Virtualised | Tempest | OpenStack reference test suite `[2]`_ |
+ | Infrastructure +----------------+--------------------------------------------+
+ | Manager) | Rally scenario | OpenStack testing tool testing OpenStack |
+ | | | modules `[3]`_ |
+ +----------------+----------------+--------------------------------------------+
+ | | OpenDaylight | Opendaylight Test suite |
+ | +----------------+--------------------------------------------+
+ | Controllers | ONOS | Test suite of ONOS L2 and L3 functions |
+ | +----------------+--------------------------------------------+
+ | | OpenContrail | |
+ +----------------+----------------+--------------------------------------------+
+ | Features | vIMS | Show the capability to deploy a real NFV |
+ | | | test cases. |
+ | | | The IP Multimedia Subsytem is a typical |
+ | | | Telco test case, referenced by ETSI. |
+ | | | It provides a fully functional VoIP System.|
+ | +----------------+--------------------------------------------+
+ | | Promise | Resource reservation and management project|
+ | | | to identify NFV related requirements and |
+ | | | realize resource reservation for future |
+ | | | usage by capacity management of resource |
+ | | | pools regarding compute, network and |
+ | | | storage. |
+ | +----------------+--------------------------------------------+
+ | | SDNVPN | |
+ +----------------+----------------+--------------------------------------------+
+
+
+Most of the test suites are developed upstream.
+For example, `Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
+OpenStack integration test suite.
+Functest is in charge of the integration of different functional test suites.
+
+The Tempest suite has been customized but no new test cases have been created.
+Some OPNFV feature projects (e.g. SDNVPN) have created Tempest tests cases and
+pushed to upstream.
+
+The tests run from CI are pushed into a database.
+The goal is to populate the database with results and to show them on a Test
+Dashboard.
+
+There is no real notion of Test domain or Test coverage yet.
+Basic components (VIM, controllers) are tested through their own suites.
+Feature projects also provide their own test suites.
+
+vIMS test case was integrated to demonstrate the capability to deploy a
+relatively complex NFV scenario on top of the OPNFV infrastructure.
+
+Functest considers OPNFV as a black box.
+OPNFV, since Brahmaputra, offers lots of possible combinations:
+
+ * 3 controllers (OpenDayligh, ONOS, OpenContrail)
+ * 4 installers (Apex, Compass, Fuel, Joid)
+
+However most of the tests shall be runnable on any configuration.
It is assumed that Functest container has been properly installed `[1]`_.
-
-Description of the test cases
-=============================
-
-Functest is an OPNFV project dedicated to functional testing.
-In the continuous integration, it is launched after an OPNFV fresh installation.
-The Functest target is to verify the basic functions of the infrastructure.
-
-Functest includes different test suites which several test cases within.
-Test cases are developed in Functest and in feature projects.
-
-The current list of test suites can be distributed in 3 main domains::
-
- +----------------+----------------+--------------------------------------------+
- | Method | Test suite | Comments |
- +================+================+============================================+
- | | vPing | NFV "Hello World" |
- | +----------------+--------------------------------------------+
- | VIM | vPing_userdata | Ping using userdata and cloud-init |
- | | | mechanism |
- | +----------------+--------------------------------------------+
- |(Virtualised | Tempest | OpenStack reference test suite `[2]`_ |
- | Infrastructure +----------------+--------------------------------------------+
- | Manager) | Rally scenario | OpenStack testing tool testing OpenStack |
- | | | modules `[3]`_ |
- +----------------+----------------+--------------------------------------------+
- | | OpenDaylight | Opendaylight Test suite |
- | +----------------+--------------------------------------------+
- | Controllers | ONOS | Test suite of ONOS L2 and L3 functions |
- | +----------------+--------------------------------------------+
- | | OpenContrail | |
- +----------------+----------------+--------------------------------------------+
- | Features | vIMS | Show the capability to deploy a real NFV |
- | | | test cases. |
- | | | The IP Multimedia Subsytem is a typical |
- | | | Telco test case, referenced by ETSI. |
- | | | It provides a fully functional VoIP System.|
- | +----------------+--------------------------------------------+
- | | Promise | Resource reservation and management project|
- | | | to identify NFV related requirements and |
- | | | realize resource reservation for future |
- | | | usage by capacity management of resource |
- | | | pools regarding compute, network and |
- | | | storage. |
- | +----------------+--------------------------------------------+
- | | SDNVPN | |
- +----------------+----------------+--------------------------------------------+
-
-
-Most of the test suites are developed upstream.
-For example, Tempest `[2]`_ is the OpenStack integration test suite.
-Functest is in charge of the integration of different functional test suites.
-
-The Tempest suite has been customized but no new test cases have been created.
-Some OPNFV feature projects (.e.g. SDNVPN) have created Tempest tests cases and
-pushed to upstream.
-
-The tests run from CI are pushed into a database.
-The goal is to populate the database with results and to show them on a Test
-Dashboard.
-
-There is no real notion of Test domain or Test coverage yet.
-Basic components (VIM, controllers) are tested through their own suites.
-Feature projects also provide their own test suites.
-
-vIMS test case was integrated to demonstrate the capability to deploy a
-relatively complex NFV scenario on top of the OPNFV infrastructure.
-
-Functest considers OPNFV as a black box.
-OPNFV, since Brahmaputra, offers lots of possible combinations:
-
- * 3 controllers (OpenDayligh, ONOS, OpenContrail)
- * 4 installers (Apex, Compass, Fuel, Joid)
-
-However most of the tests shall be runnable on any configuration.
+.. include:: ./description.rst
The different scenarios are described in the section hereafter.
* OpenStack API validation
* Scenarios
- * other specific tests useful in validating an OpenStack deployment
+ * Other specific tests useful in validating an OpenStack deployment
We use Rally `[3]`_ to run Tempest suite.
Rally generates automatically tempest.conf configuration file.
The full console output is stored in the tempest.log file.
As an addition of Arno, Brahmaputra runs a customized set of Tempest test cases.
-The list is specificed through --tests-file when running Rally.
+The list is specificed through *--tests-file* when running Rally.
This option has been introduced in Rally in version 0.1.2.
-The customized test list is available in the Functest repo `[4]`_
+The customized test list is available in the Functest repo `[4]`_.
This list contains more than 200 Tempest test cases.
The list can be divied into two main parts:
flavor_ram: 512
flavor_disk: 0
-
-Manual testing
-==============
-
-Once the Functest docker container is running and Functest environment ready
-(through /home/opnfv/repos/functest/docker/prepare_env.sh script), the system is
-ready to run the tests.
-
-The script run_tests.sh is located in $repos_dir/functest/docker and it has
-several options::
-
- ./run_tests.sh -h
- Script to trigger the tests automatically.
-
- usage:
- bash run_tests.sh [--offline] [-h|--help] [-t <test_name>]
-
- where:
- -h|--help show this help text
- -r|--report push results to database (false by default)
- -n|--no-clean do not clean up OpenStack resources after test run
- -t|--test run specific set of tests
- <test_name> one or more of the following: vping,vping_userdata,odl,rally,tempest,vims,onos,promise. Separated by comma.
-
- examples:
- run_tests.sh
- run_tests.sh --test vping,odl
- run_tests.sh -t tempest,rally --no-clean
-
-The -r option is used by the Continuous Integration in order to push the test
-results into a test collection database, see in next section for details.
-In manual mode, you must not use it, your try will be anyway probably rejected
-as your POD must be declared in the database to collect the data.
-
-The -n option is used for preserving all the existing OpenStack resources after
-execution test cases.
-
-The -t option can be used to specify the list of test you want to launch, by
-default Functest will try to launch all its test suites in the following order
-vPing, odl, Tempest, vIMS, Rally.
-You may launch only one single test by using -t <the test you want to launch>
-
-Within Tempest test suite you can define which test cases you want to execute in
-your environment by editing test_list.txt file before executing run_tests.sh
-script.
-
-Please note that Functest includes cleaning mechanism in order to remove
-everything except what was present after a fresh install.
-If you create your own VMs, tenants, networks etc. and then launch Functest,
-they all will be deleted after executing the tests. Use --no-clean option with
-run_test.sh in order to preserve all the existing resources.
-However, be aware that Tempest and Rally create of lot of resources (users,
-tenants, networks, volumes etc.) that are not always properly cleaned, so this
-cleaning function has been set to keep the system as clean as possible after a
-full Functest run.
-
-You may also add you own test by adding a section into the function run_test()
-
-
-Automated testing
-=================
-
-As mentioned in `[1]`, the prepare-env.sh and run_test.sh can be executed within
-the container from jenkins.
-2 jobs have been created, one to run all the test and one that allows testing
-test suite by test suite.
-You thus just have to launch the acurate jenkins job on the target lab, all the
-tests shall be automatically run.
-
-When the tests are automatically started from CI, a basic algorithm has been
-created in order to detect whether the test is runnable or not on the given
-scenario.
-In fact, one of the most challenging task in Brahmaputra consists in dealing
-with lots of scenario and installers.
-Functest test suites cannot be systematically run (e.g. run the ODL suite on an
-ONOS scenario).
-
-CI provides several information:
-
- * The installer (apex|compass|fuel|joid)
- * The scenario [controller]-[feature]-[mode] with
-
- * controller = (odl|onos|ocl|nosdn)
- * feature = (ovs(dpdk)|kvm)
- * mode = (ha|noha)
-
-Constraints per test case are defined in the Functest configuration file
-/home/opnfv/functest/config/config_functest.yaml::
-
- test-dependencies:
- functest:
- vims:
- scenario: '(ocl)|(odl)|(nosdn)'
- vping:
- vping_userdata:
- scenario: '(ocl)|(odl)|(nosdn)'
- tempest:
- rally:
- odl:
- scenario: 'odl'
- onos:
- scenario: 'onos'
- ....
-
-At the end of the Functest environment creation (prepare_env.sh see `[1]`_), a
-file (/home/opnfv/functest/conf/testcase-list.txt) is created with the list of
-all the runnable tests.
-We consider the static constraints as regex and compare them with the scenario.
-For instance, odl can be run only on scenario including odl in its name.
-
-The order of execution is also described in the Functest configuration file::
-
- test_exec_priority:
-
- 1: vping
- 2: vping_userdata
- 3: tempest
- 4: odl
- 5: onos
- 6: ovno
- #7: doctor
- 8: promise
- 9: odl-vpnservice
- 10: bgpvpn
- #11: openstack-neutron-bgpvpn-api-extension-tests
- 12: vims
- 13: rally
-
-The tests are executed as follow:
-
- * Basic scenario (vPing, vPing_userdata, Tempest)
- * Controller suites: ODL or ONOS or OpenContrail
- * Feature projects
- * vIMS
- * Rally (benchmark scenario)
-
-At the end of an automated execution, everything is cleaned.
-We keep only the users/networks that have been statically declared in '[9]'_
-
+.. include:: ./runfunctest.rst
Test results
============
* log.html
* report.html
- ODL result page
+**ODL result page**
.. figure:: ../images/functestODL.png
:width: 170mm
ONOS
^^^^
-The ONOS test logs can be found in OnosSystemTest/TestON/logs
-(ONOSCI_PATH to be added),and also can be seen in the console::
+The ONOS test logs can be found in OnosSystemTest/, and TestON/, and logs/
+(ONOSCI_PATH to be added), and can also be seen in the console::
******************************
Result summary for Testcase4
* INFO: environment prepared successfully => environment OK
* INFO - Cloudify-manager server is UP ! => orchestrator deployed
- * INFO The deployment of clearwater-opnfv is ended => VNF deployed
+ * INFO - The deployment of clearwater-opnfv is ended => VNF deployed
* Multiple Identities (UDP) - (6505550771, 6505550675) Passed => tests run
* DEBUG - Pushing results to DB.... => tests saved
The API can described as follow:
-**Version:**::
+Version:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
+--------+--------------------------+------------------------------------------+
-**Pods:**::
+Pods:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
| | | } |
+--------+--------------------------+------------------------------------------+
-**Projects:**::
+Projects:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
+--------+--------------------------+------------------------------------------+
-**Test cases:**::
+Test cases:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
| | | } |
+--------+--------------------------+------------------------------------------+
| PUT | /test_projects/{project}?| Modify a test case of {project} |
- | | case_name={case} | |
+ | | case_name={case} | |
| | | Content-Type: application/json |
| | | { |
| | | <the field(s) you want to modify> |
+--------+--------------------------+------------------------------------------+
| DELETE | /test_projects/{project}/| Delete a test case |
| | case_name={case} | |
- +----------------+----------------+--------------------------------------------+
+ +--------+--------------------------+------------------------------------------+
-**Test Results:**::
+Test Results:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
+--------+--------------------------+------------------------------------------+
-**Dashboard:**::
+Dashboard:
+--------+--------------------------+------------------------------------------+
| Method | Path | Description |
.. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
.. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py
.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml
-
--- /dev/null
+Executing the functest suites
+=============================
+
+Manual testing
+--------------
+
+Once the Functest docker container is running and Functest environment ready
+(through /home/opnfv/repos/functest/docker/prepare_env.sh script), the system is
+ready to run the tests.
+
+The script *run_tests.sh* is located in $repos_dir/functest/docker and it has
+several options::
+
+ ./run_tests.sh -h
+ Script to trigger the tests automatically.
+
+ usage:
+ bash run_tests.sh [--offline] [-h|--help] [-t <test_name>]
+
+ where:
+ -h|--help show this help text
+ -r|--report push results to database (false by default)
+ -n|--no-clean do not clean up OpenStack resources after test run
+ -t|--test run specific set of tests
+ <test_name> one or more of the following: vping,vping_userdata,odl,rally,tempest,vims,onos,promise. Separated by comma.
+
+ examples:
+ run_tests.sh
+ run_tests.sh --test vping,odl
+ run_tests.sh -t tempest,rally --no-clean
+
+The *-r* option is used by the Continuous Integration in order to push the test
+results into a test collection database, see in next section for details.
+In manual mode, you must not use it, your try will be anyway probably rejected
+as your POD must be declared in the database to collect the data.
+
+The *-n* option is used for preserving all the existing OpenStack resources after
+execution test cases.
+
+The *-t* option can be used to specify the list of test you want to launch, by
+default Functest will try to launch all its test suites in the following order
+vPing, odl, Tempest, vIMS, Rally.
+You may launch only one single test by using *-t <the test you want to launch>*.
+
+Within Tempest test suite you can define which test cases you want to execute in
+your environment by editing test_list.txt file before executing *run_tests.sh*
+script.
+
+Please note that Functest includes cleaning mechanism in order to remove
+everything except what was present after a fresh install.
+If you create your own VMs, tenants, networks etc. and then launch Functest,
+they all will be deleted after executing the tests. Use the *--no-clean* option with
+run_test.sh in order to preserve all the existing resources.
+However, be aware that Tempest and Rally create of lot of resources (users,
+tenants, networks, volumes etc.) that are not always properly cleaned, so this
+cleaning function has been set to keep the system as clean as possible after a
+full Functest run.
+
+You may also add you own test by adding a section into the function run_test().
+
+
+Automated testing
+-----------------
+
+As mentioned in `[1]`, the *prepare-env.sh* and *run_test.sh* can be executed within
+the container from jenkins.
+2 jobs have been created, one to run all the test and one that allows testing
+test suite by test suite.
+You thus just have to launch the acurate jenkins job on the target lab, all the
+tests shall be automatically run.
+
+When the tests are automatically started from CI, a basic algorithm has been
+created in order to detect whether the test is runnable or not on the given
+scenario.
+In fact, one of the most challenging task in Brahmaputra consists in dealing
+with lots of scenario and installers.
+Functest test suites cannot be systematically run (e.g. run the ODL suite on an
+ONOS scenario).
+
+CI provides several information:
+
+ * The installer (apex|compass|fuel|joid)
+ * The scenario [controller]-[feature]-[mode] with
+
+ * controller = (odl|onos|ocl|nosdn)
+ * feature = (ovs(dpdk)|kvm)
+ * mode = (ha|noha)
+
+Constraints per test case are defined in the Functest configuration file
+/home/opnfv/functest/config/config_functest.yaml::
+
+ test-dependencies:
+ functest:
+ vims:
+ scenario: '(ocl)|(odl)|(nosdn)'
+ vping:
+ vping_userdata:
+ scenario: '(ocl)|(odl)|(nosdn)'
+ tempest:
+ rally:
+ odl:
+ scenario: 'odl'
+ onos:
+ scenario: 'onos'
+ ....
+
+At the end of the Functest environment creation (prepare_env.sh see `[1]`_), a
+file (/home/opnfv/functest/conf/testcase-list.txt) is created with the list of
+all the runnable tests.
+We consider the static constraints as regex and compare them with the scenario.
+For instance, odl can be run only on scenario including odl in its name.
+
+The order of execution is also described in the Functest configuration file::
+
+ test_exec_priority:
+
+ 1: vping
+ 2: vping_userdata
+ 3: tempest
+ 4: odl
+ 5: onos
+ 6: ovno
+ 7: doctor
+ 8: promise
+ 9: odl-vpnservice
+ 10: bgpvpn
+ 11: openstack-neutron-bgpvpn-api-extension-tests
+ 12: vims
+ 13: rally
+
+The tests are executed as follow:
+
+ * Basic scenario (vPing, vPing_userdata, Tempest)
+ * Controller suites: ODL or ONOS or OpenContrail
+ * Feature projects
+ * vIMS
+ * Rally (benchmark scenario)
+
+At the end of an automated execution, everything is cleaned.
+We keep only the users/networks that have been statically declared in 'https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml'_
Code Review / functest-xtesting.git / commitdiff