.. toctree::
:maxdepth: 2
+Version history
+===============
++------------+----------+------------------+----------------------------------+
+| **Date** | **Ver.** | **Author** | **Comment** |
+| | | | |
++------------+----------+------------------+----------------------------------+
+| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release |
+| | | Column Gaynor | |
++------------+----------+------------------+----------------------------------+
+| 2017-01-23 | 1.0.1 | Morgan Richomme | Adaptations for Danube |
+| | | | |
+| | | | |
++------------+----------+------------------+----------------------------------+
+
Introduction
============
The goal of this document is to describe the OPNFV Functest test cases and to
-provide a procedure to execute them. In the OPNFV Colorado system release,
+provide a procedure to execute them. In the OPNFV Danube system release,
a Functest CLI utility is introduced for easier execution of test procedures.
-An overview presentation has been created for the first OPNFV Summit `[4]`_.
-
-This document is a continuation of the OPNFV Functest Configuration Guide `[1]`_.
-
**IMPORTANT**: It is assumed here that the Functest Docker container is already
properly deployed and that all instructions described in this guide are to be
performed from *inside* the deployed Functest Docker container.
Healthcheck
^^^^^^^^^^^
In Colorado release a new Tier 'healthcheck' with one testcase 'healthcheck'
-is introduced. The healthcheck testcase verifies that some basic IP connectivity
+was introduced. The healthcheck testcase verifies that some basic IP connectivity
and essential operations of OpenStack functionality over the command line are
working correctly.
Tiers:
* Smoke Tier - Test Case 'tempest_smoke_serial'
- * Openstack Tier - Test case 'tempest_full_parallel'
+ * Components Tier - Test case 'tempest_full_parallel'
NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke
tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel'
The Rally testcases are distributed accross two Tiers:
* Smoke Tier - Test Case 'rally_sanity'
- * Openstack Tier - Test case 'rally_full'
+ * Components Tier - Test case 'rally_full'
NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test
cases. Test case 'rally_full' executes the full defined set of Rally tests.
+SNAPS
+-----
+
+SNAPS stands for "SNA/NFV Application development Platform and Stack".
+This project seeks to develop baseline OpenStack NFV installations. It has been
+developed by Steven Pisarski and provided an object oriented library to perform
+functional and performance tests. It has been declined in several test suites in
+Functest.
+
+connection check
+^^^^^^^^^^^^^^^^
+Connection_check consists in 9 test cases (test duration < 5s) checking the
+connectivity with Glance, Keystone, Neutron, Nova and the external network.
+
+api_check
+^^^^^^^^^
+This test case verifies the retrieval of OpenStack clients: Keystone, Glance,
+Neutron and Nova and may perform some simple queries. When the config value of
+snaps.use_keystone is True, functest must have access to the cloud's private
+network.
+This suite consists in 49 tests (test duration< 2 minutes)
+
+Snaps_smoke
+^^^^^^^^^^^
+This test case contains tests that setup and destroy environments with VMs with
+and without Floating IPs with a newly created user and project. Set the config
+value snaps.use_floating_ips (True|False) to toggle this functionality. When
+the config value of snaps.use_keystone is True, functest must have access
+the cloud's private network.
+This suite consists in 38 tests (test duration < 10 minutes)
+
+More information on SNAPS can be found in `[13]`_
+
SDN Controllers
---------------
Features
--------
-Most of the features have been developped by feature projects.
-Security_scan has been initiated in Functest repository but should soon
-be declared in its own repository as well.
-
Please refer to the dedicated feature user guides for details:
- * bgpvpn: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html
- * copper: http://artifacts.opnfv.org/copper/colorado/docs/userguide/index.html
- * doctor: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html
+ * bgpvpn: http://artifacts.opnfv.org/sdnvpn/danube/docs/userguide/index.html
+ * copper: http://artifacts.opnfv.org/copper/danube/docs/userguide/index.html
+ * doctor: http://artifacts.opnfv.org/doctor/danube/userguide/index.html
* domino: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html
* moon: http://artifacts.opnfv.org/moon/docs/userguide/index.html
* multisites: http://artifacts.opnfv.org/multisite/docs/userguide/index.html
- * onos-sfc: http://artifacts.opnfv.org/onosfw/colorado/userguide/index.html
- * odl-sfc: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html
- * promise: http://artifacts.opnfv.org/promise/colorado/docs/userguide/index.html
+ * onos-sfc: http://artifacts.opnfv.org/onosfw/danube/userguide/index.html
+ * odl-sfc: http://artifacts.opnfv.org/sfc/danube/userguide/index.html
+ * promise: http://artifacts.opnfv.org/danube/colorado/docs/userguide/index.html
+ * security_scan: http://artifacts.opnfv.org/security_scan/colorado/docs/userguide/index.html
+ * TODO
security_scan
^^^^^^^^^^^^^
Security Scanning, is a project to insure security compliance and vulnerability
checks, as part of an automated CI / CD platform delivery process.
-The project makes use of the existing SCAP format[6] to perform deep scanning of
-NFVi nodes, to insure they are hardened and free of known CVE reported vulnerabilities.
+The project makes use of the existing SCAP format `[6]`_ to perform deep
+scanning of NFVI nodes, to insure they are hardened and free of known CVE
+reported vulnerabilities.
The SCAP content itself, is then consumed and run using an upstream opensource tool
-known as OpenSCAP[7].
+known as OpenSCAP `[7]`_.
The OPNFV Security Group have developed the code that will called by the OPNFV Jenkins
build platform, to perform a complete scan. Resulting reports are then copied to the
* If the config file value 'clean' is set to 'True' then the application installed in
step 5 is removed, and all reports created at step 6 are deleted.
-At present, only the Apex installer is supported, with support for other installers due
-within D-release.
+Security scan is supported by Apex, TODO....
vIMS has been integrated in Functest to demonstrate the capability to deploy a
relatively complex NFV scenario on the OPNFV platform. The deployment of a complete
-functional VNF allows the test of most of the
-essential functions needed for a NFV platform.
+functional VNF allows the test of most of the essential functions needed for a
+NFV platform.
The goal of this test suite consists of:
.. _`[9]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml
.. _`[11]`: http://robotframework.org/
.. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
+.. _`[13]`: TODO URL doc SNAPS
OPNFV main site: opnfvmain_.
installation to validate and verify the basic functions of the
infrastructure.
-The current list of test suites can be distributed over 4 main domains: VIM
+The current list of test suites can be distributed over 5 main domains: VIM
(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers),
-Features and VNF (Virtual Network Functions).
+Features, VNF (Virtual Network Functions) and MANO stacks.
+
+Functest test suites are also distributed in the OPNFV testing categories:
+healthcheck, smoke, features, components, performance, VNF, Stress tests.
+
+All the Healthcheck and smoke tests of a given scenario must be succesful to
+validate the scenario for the release.
+-------------+---------------+----------------+----------------------------------+
| Domain | Tier | Test case | Comments |
+=============+===============+================+==================================+
| VIM | healthcheck | healthcheck | Verify basic operation in VIM |
+| | +----------------+----------------------------------+
+| | | connection | Check OpenStack connectivity |
+| | | _check | through SNAPS framework |
+| | +----------------+----------------------------------+
+| | | api_check | Check OpenStack API through |
+| | | | SNAPS framework |
| +---------------+----------------+----------------------------------+
| | smoke | vPing_SSH | NFV "Hello World" using an SSH |
| | | | connection to a destination VM |
| | +----------------+----------------------------------+
| | | rally_sanity | Run a subset of the OpenStack |
| | | | Rally Test Suite in smoke mode |
+| | +----------------+----------------------------------+
+| | | snaps_smoke | Run a subset of the OpenStack |
+| | | | Rally Test Suite in smoke mode |
| +---------------+----------------+----------------------------------+
-| | openstack | tempest_full | Generate and run a full set of |
+| | components | tempest_full | Generate and run a full set of |
| | | \_parallel | the OpenStack Tempest Test Suite.|
| | | | See the OpenStack reference test |
| | | | suite `[2]`_. The generated |
| | | | benchmarking OpenStack modules |
| | | | See the Rally documents `[3]`_. |
+-------------+---------------+----------------+----------------------------------+
-| Controllers | sdn_suites | odl | Opendaylight Test suite |
+| Controllers | smoke | odl | Opendaylight Test suite |
| | | | Limited test suite to check the |
| | | | basic neutron (Layer 2) |
| | | | operations mainly based on |
| | | | See `ONOSFW User Guide`_ for |
| | | | details. |
+-------------+---------------+----------------+----------------------------------+
-| Features | features | Promise | Resource reservation and |
+| Features | features | promise | Resource reservation and |
| | | | management project to identify |
| | | | NFV related requirements and |
| | | | realize resource reservation for |
| | | | See `Promise User Guide`_ for |
| | | | details. |
| | +----------------+----------------------------------+
-| | | Doctor | Doctor platform, as of Colorado |
+| | | doctor | Doctor platform, as of Colorado |
| | | | release, provides the three |
| | | | features: |
| | | | * Immediate Notification |
| | | | See `Domino User Guide`_ for |
| | | | details |
| | +----------------+----------------------------------+
-| | | Copper | Copper develops OPNFV platform |
+| | | copper | Copper develops OPNFV platform |
| | | | support for policy management, |
| | | | using open source projects such |
| | | | as OpenStack Congress, focused |
| | | | See `Moon User Guide`_ for |
| | | | details |
+-------------+---------------+----------------+----------------------------------+
-| VNF | vnf | vims | Example of a real VNF deployment |
+| VNF | vnf | cloudify_ims | Example of a real VNF deployment |
| | | | to show the NFV capabilities of |
| | | | the platform. The IP Multimedia |
| | | | Subsytem is a typical Telco test |
| | | | case, referenced by ETSI. |
| | | | It provides a fully functional |
| | | | VoIP System |
+| | +----------------+----------------------------------+
+| | | opera_ims | vIMS deployment using openBaton |
+| | +----------------+----------------------------------+
+| | | orchestra_ims | vIMS deployment using open-O |
+ +---------------+----------------+----------------------------------+
| | | parser | Parser is an integration project |
| | | | which aims to provide |
'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
Tier testcases, there is little point to launch a full testcase execution round.
+In Danube, we merged smoke and sdn controller tiers in smoke tier.
+
An overview of the Functest Structural Concept is depicted graphically below:
.. figure:: ../images/concepts_mapping_final.png
sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot
of a live Functest Dashboard is shown below:
-.. figure:: ../images/FunctestDashboardColorado.png
+** TODO **
+.. figure:: ../images/FunctestDashboardDanube.png
:align: center
:alt: Functest Dashboard
-There is no real notion of Test domain or Test coverage. Basic components
-(VIM, SDN controllers) are tested through their own suites. Feature projects
-also provide their own test suites with different ways of running their tests.
+Basic components (VIM, SDN controllers) are tested through their own suites.
+Feature projects also provide their own test suites with different ways of
+running their tests.
+
+The notion of domain has been introduced in the description of the test cases
+stored in the Database.
+This parameters as well as possible tags can be used for the Test case catalog.
vIMS test case was integrated to demonstrate the capability to deploy a
relatively complex NFV scenario on top of the OPNFV infrastructure.
-Functest considers OPNFV as a black box. As of Colorado release the OPNFV
+Functest considers OPNFV as a black box. As of Danube release the OPNFV
offers a lot of potential combinations:
* 3 controllers (OpenDaylight, ONOS, OpenContrail)
DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given
environment.
-In the Colorado OPNFV System release a convenience Functest CLI utility is also
-introduced to simplify setting up the Functest evironment, management of the
-OpenStack environment (e.g. resource clean-up) and for executing tests.
+A convenience Functest CLI utility is also available to simplify setting up the
+Functest evironment, management of the OpenStack environment (e.g. resource
+clean-up) and for executing tests.
The Functest CLI organised the testcase into logical Tiers, which contain in
turn one or more testcases. The CLI allows execution of a single specified
testcase, all test cases in a specified Tier, or the special case of execution
as they are a prerequisite and all the commands explained in this section **must** be
performed **inside the container**.
-Note: In Colorado release, the scripts **run_tests.sh** is now replaced with a
-new Functest CLI. One difference, is that tests run through the Functest CLI
-will always clean-up OpenStack resources. See the `Troubleshooting`_ section of this
-document, where this difference is discussed.
-
The Functest CLI offers two commands (functest tier ...) and (functest testcase ... )
for the execution of Test Tiers or Test Cases::
root@22e436918db0:~/repos/functest/ci# functest tier list
- 0. healthcheck:
- ['healthcheck']
+ ['healthcheck', 'connection_check', 'api_check',]
- 1. smoke:
- ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity']
- - 2. sdn_suites:
- ['odl']
- - 3. features:
+ ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity', 'snaps_smoke', 'odl']
+ - 2. features:
['doctor', 'security_scan']
- - 4. openstack:
+ - 3. components:
['tempest_full_parallel', 'rally_full']
- - 5. vnf:
- ['vims']
+ - 4. vnf:
+ ['cloudify_ims']
and
healthcheck
vping_ssh
vping_userdata
+ connection_check
+ api_check
+ snaps_smoke
tempest_smoke_serial
rally_sanity
odl
security_scan
tempest_full_parallel
rally_full
- vims
+ cloudify_ims
More specific details on specific Tiers or Test Cases can be seen wih the
'show' command::
To list the test cases which are part of a specific Test Tier, the 'get-tests'
command is used with 'functest tier'::
- root@22e436918db0:~/repos/functest/ci# functest tier get-tests sdn_suites
- Test cases in tier 'sdn_suites':
- ['odl']
+ root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck
+ Test cases in tier 'healthcheck':
+ ['healthcheck']
Please note that for some scenarios some test cases might not be launched.
* Run a single Test Case, specified by a valid choice of <testcase_name>
* Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all'
-Example::
-
- root@22e436918db0:~/repos/functest/ci# functest testcase run all
- Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t all'
- 2016-06-30 12:03:28,628 - run_tests - INFO - Sourcing the OpenStack RC file...
- 2016-06-30 12:03:28,634 - run_tests - INFO - Tiers to be executed:
- - 0. healthcheck:
- ['healthcheck']
- - 1. smoke:
- ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity']
- - 2. sdn_suites:
- ['odl']
- - 3. features:
- ['doctor', 'security_scan']
- - 4. openstack:
- ['tempest_full_parallel', 'rally_full']
- - 5. vnf:
- ['vims']
- 2016-06-30 12:03:28,634 - run_tests - INFO - ############################################
- 2016-06-30 12:03:28,635 - run_tests - INFO - Running tier 'healthcheck'
- 2016-06-30 12:03:28,635 - run_tests - INFO - ############################################
- 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================
- 2016-06-30 12:03:28,635 - run_tests - INFO - Running test case 'healthcheck'...
- 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================
- 2016-06-30 12:03:28,651 - healtcheck - INFO - Testing Keystone API...
- 2016-06-30 12:03:36,676 - healtcheck - INFO - ...Keystone OK!
- 2016-06-30 12:03:36,679 - healtcheck - INFO - Testing Glance API...
- :
- :
- etc.
-
Functest includes a cleaning mechanism in order to remove all the OpenStack
resources except those present before running any test. The script
*$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up
the test suite name. The user may also use either of these Jenkins jobs to execute
the desired test suites.
-One of the most challenging task in the Colorado release consists
+One of the most challenging task in the Danube release consists
in dealing with lots of scenarios and installers. Thus, when the tests are
automatically started from CI, a basic algorithm has been created in order to
detect whether a given test is runnable or not on the given scenario.
Some Functest test suites cannot be systematically run (e.g. ODL suite can not
-be run on an ONOS scenario). Moreover since Colorado, we also introduce the
-notion of daily/weekly in order to save CI time and avoid running systematically
-long duration tests.
+be run on an ONOS scenario). The daily/weekly notion has been introduces in
+Colorado in order to save CI time and avoid running systematically
+long duration tests. It was not used in Colorado due to CI resource shortage.
+The mechanism remains however as part of the CI evolution.
CI provides some useful information passed to the container as environment
variables:
- * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE
+ * Installer (apex|compass|daisy|fuel|joid), stored in INSTALLER_TYPE
* Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
* The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
dependencies:
installer: ''
scenario: '^((?!bgpvpn|odl_l3).)*$'
+ run:
+ module: 'functest.opnfv_tests.openstack.vping.vping_ssh'
+ class: 'VPingSSH'
....
We may distinguish 2 levels in the test case description:
* blocking: if set to true, if the test is failed, the execution of the following tests is canceled
* the description of the test case
* the dependencies: a combination of 2 regex on the scenario and the installer name
+ * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests
+
+For further details on abstraction classes, see developper guide.
+
+Additional parameters have been added in the desription in the Database.
+The target is to use the configuration stored in the Database and consider the
+local file as backup if the Database is not reachable.
+The additional fields related to a test case are:
+ * trust: we introduced this notion to put in place a mechanism of scenario promotion.
+ * Version: it indicates since which version you can run this test
+ * domains: the main domain covered by the test suite
+ * tags: a list of tags related to the test suite
The order of execution is the one defined in the file if all test cases are selected.
1) healthcheck (blocking)
2) smoke: both vPings are blocking
- 3) SDN controller suites (blocking)
- 4) Feature project tests cases
+ 3) Feature project tests cases
In CI weekly job we add 2 tiers:
- 5) vIMS suite
- 6) Rally suite
+ 4) VNFs (vIMS)
+ 5) Components (Rally and Tempest long duration suites)
As explained before, at the end of an automated execution, the OpenStack resources
might be eventually removed.
*$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_userdata.py*
Notes:
- #. In this Colorado Funtest Userguide, the use of the Functest CLI is
- emphasized. The Functest CLI replaces the earlier Bash shell script
- *run_tests.sh*.
#. There is one difference, between the Functest CLI based test case
execution compared to the earlier used Bash shell script, which is
NFV
---
-vIMS
-^^^^
+cloudify_ims
+^^^^^^^^^^^^
vIMS deployment may fail for several reasons, the most frequent ones are
described in the following table: