The current list of test suites can be distributed over 5 main domains:
* VIM (Virtualised Infrastructure Manager)
* Controllers (i.e. SDN Controllers)
- * Features
* VNF (Virtual Network Functions)
* Kubernetes
Functest test suites are also distributed in the OPNFV testing categories:
-healthcheck, smoke, features, components, performance, VNF, Stress tests.
+healthcheck, smoke, benchmarking, VNF, Stress tests.
-All the Healthcheck and smoke tests of a given scenario must be succesful to
+All the Healthcheck and smoke tests of a given scenario must be successful to
validate the scenario for the release.
+-------------+---------------+------------+----------------------------------+
| Domain | Tier | Test case | Comments |
+=============+===============+============+==================================+
| VIM | healthcheck | connection | Check OpenStack connectivity |
-| | | \_check | through SNAPS framework |
-| | +------------+----------------------------------+
-| | | api_check | Check OpenStack API through |
-| | | | SNAPS framework |
-| | +------------+----------------------------------+
-| | | snaps | basic instance creation, check |
-| | | \_health | DHCP |
| | | \_check | |
| +---------------+------------+----------------------------------+
| | smoke | vping_ssh | NFV "Hello World" using an SSH |
| | +------------+----------------------------------+
| | | tempest | Generate and run a relevant |
| | | \_smoke | Tempest Test Suite in smoke mode.|
-| | | \_serial | The generated test set is |
+| | | | The generated test set is |
| | | | dependent on the OpenStack |
| | | | deployment environment |
| | +------------+----------------------------------+
| | | rally | Run a subset of the OpenStack |
| | | \_sanity | Rally Test Suite in smoke mode |
| | +------------+----------------------------------+
-| | | snaps\ | Run the SNAPS-OO integration |
-| | | \_smoke | tests |
-| | +------------+----------------------------------+
| | | refstack | Reference RefStack suite |
| | | \_defcore | tempest selection for NFV |
| | +------------+----------------------------------+
| | | | upstream testcases. See below |
| | | | for details |
+-------------+---------------+------------+----------------------------------+
-| Features | features | bgpvpn | Implementation of the OpenStack |
-| | | | bgpvpn API from the SDNVPN |
-| | | | feature project. It allows for |
-| | | | the creation of BGP VPNs. |
-| | | | See `SDNVPN User Guide`_ for |
-| | | | details |
-| | +------------+----------------------------------+
-| | | doctor | Doctor platform, as of Colorado |
-| | | | release, provides the three |
-| | | | features: |
-| | | | * Immediate Notification |
-| | | | * Consistent resource state |
-| | | | awareness for compute host down |
-| | | | * Valid compute host status |
-| | | | given to VM owner |
-| | | | See `Doctor User Guide`_ for |
-| | | | details |
-| | +------------+----------------------------------+
-| | | odl-sfc | SFC testing for odl scenarios |
-| | | | See `SFC User Guide`_ for details|
-| | +------------+----------------------------------+
-| | | parser | Parser is an integration project |
-| | | | which aims to provide |
-| | | | placement/deployment templates |
-| | | | translation for OPNFV platform, |
-| | | | including TOSCA -> HOT, POLICY ->|
-| | | | TOSCA and YANG -> TOSCA. it |
-| | | | deals with a fake vRNC. |
-| | | | See `Parser User Guide`_ for |
-| | | | details |
-| | +------------+----------------------------------+
-| | | fds | Test Suite for the OpenDaylight |
-| | | | SDN Controller when the GBP |
-| | | | features are installed. It |
-| | | | integrates some test suites from |
-| | | | upstream using Robot as the test |
-| | | | framework |
-+-------------+---------------+------------+----------------------------------+
| VNF | vnf | cloudify | Example of a real VNF deployment |
| | | \_ims | to show the NFV capabilities of |
| | | | the platform. The IP Multimedia |
-| | | | Subsytem is a typical Telco test |
+| | | | Subsystem is a typical Telco test|
| | | | case, referenced by ETSI. |
| | | | It provides a fully functional |
| | | | VoIP System |
| | | | Technical Specs and are executed |
| | | | through protocol drivers provided|
| | | | by ABot. |
-| | +------------+----------------------------------+
-| | | cloudify | Based on cloudify_ims test case |
-| | | \_ims_perf | cloudify_ims_perf substitutes |
-| | | | the signaling test suite by an |
-| | | | automatic deployment of an Ixia |
-| | | | loader and generic SIP stress |
-| | | | tests. |
-| | | | This work has been initiated |
-| | | | during the plugfest and allows |
-| | | | realistic load tests on top of |
-| | | | cloudify_ims. Please note that |
-| | | | this test is available but not |
-| | | | declared in testcases.yaml as it |
-| | | | requires access to proprietary |
-| | | | resources (Ixia loader) |
+-------------+---------------+------------+----------------------------------+
| Kubernetes | healthcheck | k8s_smoke | Test a running Kubernetes |
| | | | cluster and ensure it satisfies |
| | smoke | k8s\_ | Run a subset of Kubernetes |
| | | conformance| End-to-End tests, expected to |
| | | | pass on any Kubernetes cluster |
-| +---------------+------------+----------------------------------+
-| | stor4nfv | stor4nfv | Run tests necessary to |
-| | | \_k8s | demonstrate conformance of the |
-| | | | K8s+Stor4NFV deployment |
-| +---------------+------------+----------------------------------+
-| | clover | clover_k8s | Test functionality of |
-| | | | K8s+Istio+Clover deployment. |
+-------------+---------------+------------+----------------------------------+
Some of the test cases are developed by Functest team members, whereas others
are integrated from upstream communities or other OPNFV projects. For example,
-`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
+`Tempest <https://docs.openstack.org/tempest/latest/>`_ is the
OpenStack integration test suite and Functest is in charge of the selection,
integration and automation of those tests that fit suitably to OPNFV.
Most of the tests are runnable by any combination, but some tests might have
restrictions imposed by the utilized installers or due to the available
-deployed features. The system uses the environment variables (INSTALLER_TYPE
-and DEPLOY_SCENARIO) to automatically determine the valid test cases, for each
-given environment.
+deployed services. The system uses the environment variables to automatically
+determine the valid test cases, for each given environment.
A convenience Functest CLI utility is also available to simplify setting up the
-Functest evironment, management of the OpenStack environment (e.g. resource
+Functest environment, management of the OpenStack environment (e.g. resource
clean-up) and for executing tests.
The Functest CLI organised the testcase into logical Tiers, which contain in
turn one or more testcases. The CLI allows execution of a single specified
of **ALL** testcases. The Functest CLI is introduced in more details in next
section.
-.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
-.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
-.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html
-.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html
-.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
-.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/
-.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html
+.. _`[2]`: https://docs.openstack.org/tempest/latest/
+.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html
+.. _`Functest Dashboard`: http://testresults.opnfv.org/
Code Review / functest.git / blobdiff