installation to validate and verify the basic functions of the
infrastructure.
-The current list of test suites can be distributed over 4 main domains: VIM
+The current list of test suites can be distributed over 5 main domains: VIM
(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers),
-Features and VNF (Virtual Network Functions).
+Features, VNF (Virtual Network Functions) and MANO stacks.
+
+Functest test suites are also distributed in the OPNFV testing categories:
+healthcheck, smoke, features, components, performance, VNF, Stress tests.
+
+All the Healthcheck and smoke tests of a given scenario must be succesful to
+validate the scenario for the release.
+-------------+---------------+----------------+----------------------------------+
| Domain | Tier | Test case | Comments |
+=============+===============+================+==================================+
| VIM | healthcheck | healthcheck | Verify basic operation in VIM |
+| | +----------------+----------------------------------+
+| | | connection | Check OpenStack connectivity |
+| | | _check | through SNAPS framework |
+| | +----------------+----------------------------------+
+| | | api_check | Check OpenStack API through |
+| | | | SNAPS framework |
| +---------------+----------------+----------------------------------+
| | smoke | vPing_SSH | NFV "Hello World" using an SSH |
| | | | connection to a destination VM |
| | +----------------+----------------------------------+
| | | rally_sanity | Run a subset of the OpenStack |
| | | | Rally Test Suite in smoke mode |
+| | +----------------+----------------------------------+
+| | | snaps_smoke | Run a subset of the OpenStack |
+| | | | Rally Test Suite in smoke mode |
| +---------------+----------------+----------------------------------+
-| | openstack | tempest_full | Generate and run a full set of |
+| | components | tempest_full | Generate and run a full set of |
| | | \_parallel | the OpenStack Tempest Test Suite.|
| | | | See the OpenStack reference test |
| | | | suite `[2]`_. The generated |
| | | | benchmarking OpenStack modules |
| | | | See the Rally documents `[3]`_. |
+-------------+---------------+----------------+----------------------------------+
-| Controllers | sdn_suites | odl | Opendaylight Test suite |
+| Controllers | smoke | odl | Opendaylight Test suite |
| | | | Limited test suite to check the |
| | | | basic neutron (Layer 2) |
| | | | operations mainly based on |
| | | | See `ONOSFW User Guide`_ for |
| | | | details. |
+-------------+---------------+----------------+----------------------------------+
-| Features | features | Promise | Resource reservation and |
+| Features | features | promise | Resource reservation and |
| | | | management project to identify |
| | | | NFV related requirements and |
| | | | realize resource reservation for |
| | | | See `Promise User Guide`_ for |
| | | | details. |
| | +----------------+----------------------------------+
-| | | Doctor | Doctor platform, as of Colorado |
-| | | | release, provides the two |
+| | | doctor | Doctor platform, as of Colorado |
+| | | | release, provides the three |
| | | | features: |
| | | | * Immediate Notification |
| | | | * Consistent resource state |
-| | | | awareness (compute) |
-| | | | * Get Valid Server state |
+| | | | awareness for compute host down |
+| | | | * Valid compute host status |
+| | | | given to VM owner |
| | | | See `Doctor User Guide`_ for |
| | | | details |
| | +----------------+----------------------------------+
| | | | installer environment) |
| | +----------------+----------------------------------+
| | | onos-sfc | SFC testing for onos scenarios |
-| | | | TODO See for details |
+| | | | See `ONOSFW User Guide`_ for |
+| | | | details |
| | +----------------+----------------------------------+
| | | odl-sfc | SFC testing for odl scenarios |
-| | | | TODO See for details |
+| | | | See `SFC User Guide`_ for details|
| | +----------------+----------------------------------+
-| | | domino | Service template publication |
-| | | | service based on OASIS |
-| | | | specification "TOSCA Simple |
-| | | | Profile for Network Function |
-| | | | Virtualization (NFV) Version 1.0"|
-| | | | TODO See for details |
+| | | domino | Domino provides TOSCA template |
+| | | | distribution service for network |
+| | | | service and VNF descriptors |
+| | | | among MANO components e.g., |
+| | | | NFVO, VNFM, VIM, SDN-C, etc., |
+| | | | as well as OSS/BSS functions. |
+| | | | See `Domino User Guide`_ for |
+| | | | details |
| | +----------------+----------------------------------+
-| | | copper | Deployment policy |
-| | | | TODO See for details |
+| | | copper | Copper develops OPNFV platform |
+| | | | support for policy management, |
+| | | | using open source projects such |
+| | | | as OpenStack Congress, focused |
+| | | | on helping ensure that virtual |
+| | | | infrastructure and the apps that |
+| | | | execute on it comply with the |
+| | | | configuration policy intent of |
+| | | | service providers, developers, |
+| | | | and end users. See more detail |
+| | | | in the `Copper User Guide`_. |
| | +----------------+----------------------------------+
| | | multisites | Multisites |
-| | | | TODO See for details |
+| | | | See `Multisite User Guide`_ for |
+| | | | details |
| | +----------------+----------------------------------+
| | | moon | Security management system |
-| | | | TODO See for details |
+| | | | See `Moon User Guide`_ for |
+| | | | details |
+-------------+---------------+----------------+----------------------------------+
-| VNF | vnf | vims | Example of a real VNF deployment |
+| VNF | vnf | cloudify_ims | Example of a real VNF deployment |
| | | | to show the NFV capabilities of |
| | | | the platform. The IP Multimedia |
| | | | Subsytem is a typical Telco test |
| | | | case, referenced by ETSI. |
| | | | It provides a fully functional |
| | | | VoIP System |
+| | +----------------+----------------------------------+
+| | | opera_ims | vIMS deployment using openBaton |
+| | +----------------+----------------------------------+
+| | | orchestra_ims | vIMS deployment using open-O |
+ +---------------+----------------+----------------------------------+
-| | | parser | Service template translation |
-| | | | TODO See for details |
+| | | parser | Parser is an integration project |
+| | | | which aims to provide |
+| | | | placement/deployment templates |
+| | | | translation for OPNFV platform, |
+| | | | including TOSCA -> HOT, POLICY ->|
+| | | | TOSCA and YANG -> TOSCA. |
+| | | | See `Parser User Guide`_ for |
+| | | | details |
+-------------+---------------+----------------+----------------------------------+
'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
Tier testcases, there is little point to launch a full testcase execution round.
+In Danube, we merged smoke and sdn controller tiers in smoke tier.
+
An overview of the Functest Structural Concept is depicted graphically below:
.. figure:: ../images/concepts_mapping_final.png
sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot
of a live Functest Dashboard is shown below:
-.. figure:: ../images/FunctestDashboard.png
+** TODO **
+.. figure:: ../images/FunctestDashboardDanube.png
:align: center
:alt: Functest Dashboard
-There is no real notion of Test domain or Test coverage. Basic components
-(VIM, SDN controllers) are tested through their own suites. Feature projects
-also provide their own test suites with different ways of running their tests.
+Basic components (VIM, SDN controllers) are tested through their own suites.
+Feature projects also provide their own test suites with different ways of
+running their tests.
+
+The notion of domain has been introduced in the description of the test cases
+stored in the Database.
+This parameters as well as possible tags can be used for the Test case catalog.
vIMS test case was integrated to demonstrate the capability to deploy a
relatively complex NFV scenario on top of the OPNFV infrastructure.
-Functest considers OPNFV as a black box. As of Colorado release the OPNFV
+Functest considers OPNFV as a black box. As of Danube release the OPNFV
offers a lot of potential combinations:
* 3 controllers (OpenDaylight, ONOS, OpenContrail)
Most of the tests are runnable by any combination, but some tests might have
restrictions imposed by the utilized installers or due to the available
deployed features. The system uses the environment variables (INSTALLER_IP and
-DEPLOY_SCENARIO) to automatically determine the valid test cases; for each given
+DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given
environment.
-In the Colorado OPNFV System release a convenience Functest CLI utility is also
-introduced to simplify setting up the Functest evironment, management of the
-OpenStack environment (e.g. resource clean-up) and for executing tests.
+A convenience Functest CLI utility is also available to simplify setting up the
+Functest evironment, management of the OpenStack environment (e.g. resource
+clean-up) and for executing tests.
The Functest CLI organised the testcase into logical Tiers, which contain in
-turn one or more testcases. The CLI allow execution of a single specified
+turn one or more testcases. The CLI allows execution of a single specified
testcase, all test cases in a specified Tier, or the special case of execution
of **ALL** testcases. The Functest CLI is introduced in more detail in the
section `Executing the functest suites`_ of this document.
.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
-.. _`Doctor User Guide`: http://artifacts.opnfv.org/opnfvdocs/brahmaputra/docs/userguide/featureusage-doctor.html
-.. _`Promise User Guide`: http://artifacts.opnfv.org/promise/brahmaputra/docs/userguide/index.html
-.. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/brahmaputra/docs/userguide/index.html
-.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/brahmaputra/docs/userguide/featureusage.html
+.. _`Copper User Guide`: http://artifacts.opnfv.org/copper/colorado/docs/userguide/index.html
+.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html
+.. _`Promise User Guide`: http://artifacts.opnfv.org/promise/colorado/docs/userguide/index.html
+.. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/colorado/userguide/index.html
+.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html
+.. _`Domino User Guide`: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html
+.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/
+.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html
+.. _`Multisite User Guide`: http://artifacts.opnfv.org/multisite/docs/userguide/index.html
+.. _`Moon User Guide`: http://artifacts.opnfv.org/moon/docs/userguide/index.html
Code Review / functest.git / blobdiff