-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. SPDX-License-Identifier: CC-BY-4.0
-******************************
-OPNFV FUNCTEST developer guide
-******************************
+************************
+Functest Developer Guide
+************************
.. toctree::
:numbered:
:maxdepth: 2
-Version history
-===============
-+------------+----------+------------------+----------------------------------+
-| **Date** | **Ver.** | **Author** | **Comment** |
-| | | | |
-+------------+----------+------------------+----------------------------------+
-| 2017-01-23 | 1.0.0 | Morgan Richomme | Creation for Danube |
-+------------+----------+------------------+----------------------------------+
-| 2017-08-16 | 1.0.1 | Morgan Richomme | Adaptations for Euphrates |
-| | | | - move generic part to Testing |
-| | | | developer guide |
-| | | | - move reporting part to functest|
-| | | | user guide |
-| | | | - update test case list |
-| | | | - include auto generated core |
-| | | | documentation |
-+------------+----------+------------------+----------------------------------+
-
============
Introduction
============
Functest is a project dealing with functional testing.
-Functest produces its own internal test cases but can also be considered
+The project produces its own internal test cases but can also be considered
as a framework to support feature and VNF onboarding project testing.
Therefore there are many ways to contribute to Functest. You can:
Additional tasks involving Functest but addressing all the test projects
may also be mentioned:
- * Develop the API / Test collection framework
- * Develop dashboards or automatic reporting portals
+ * The API / Test collection framework
+ * The dashboards
+ * The automatic reporting portals
+ * The testcase catalog
This document describes how, as a developer, you may interact with the
Functest project. The first section details the main working areas of
Functest is a project delivering test containers dedicated to OPNFV.
It includes the tools, the scripts and the test scenarios.
-Until Danube, Functest produced 2 docker files based on Ubuntu 14.04:
-
- * x86 Functest: https://hub.docker.com/r/opnfv/functest/
- * aarch64 Functest: https://hub.docker.com/r/opnfv/functest_aarch64/
-
In Euphrates Alpine containers have been introduced in order to lighten the
-container and manage testing slicing, the new containers are created according
+container and manage testing slicing. The new containers are created according
to the different tiers:
* functest-core: https://hub.docker.com/r/opnfv/functest-core/
* functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/
* functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/
- * functest-features: TODO
- * functest-components: TODO
- * functest-vnf: TODO
+ * functest-features: https://hub.docker.com/r/opnfv/functest-features/
+ * functest-components: https://hub.docker.com/r/opnfv/functest-components/
+ * functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/
+ * functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/
+
+Standalone functest dockers are maintained for Euphrates but Alpine containers
+are recommended.
Functest can be described as follow::
- +----------------------+
- | |
- | +--------------+ | +-------------------+
- | | | | Public | |
- | | Tools | +------------------+ OPNFV |
- | | Scripts | | | System Under Test |
- | | Scenarios | +------------------+ |
- | | | | Management | |
- | +--------------+ | +-------------------+
- | |
- | Functest Docker |
- | |
- +----------------------+
+ +----------------------+
+ | |
+ | +--------------+ | +-------------------+
+ | | | | Public | |
+ | | Tools | +------------------+ OPNFV |
+ | | Scripts | | | System Under Test |
+ | | Scenarios | | | |
+ | | | | | |
+ | +--------------+ | +-------------------+
+ | |
+ | Functest Docker |
+ | |
+ +----------------------+
Functest internal test cases
============================
* api_check
- * cloudify_ims
* connection_check
+ * snaps_health_check
* vping_ssh
* vping_userdata
* odl
- * odl-netvirt
- * odl-fds
* rally_full
* rally_sanity
- * snaps_health_check
- * tempest_full_parallel
* tempest_smoke_serial
+ * tempest_full_parallel
* cloudify_ims
By internal, we mean that this particular test cases have been developed and/or
The structure of this repository is detailed in `[1]`_.
The main internal test cases are in the opnfv_tests subfolder of the
-repository, the internal test cases are:
+repository, the internal test cases can be grouped by domain:
- * sdn: odl, odl_netvirt, odl_fds
- * openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_*
+ * sdn: odl, odl_fds
+ * openstack: api_check, connection_check, snaps_health_check, vping_ssh,
+ vping_userdata, tempest_*, rally_*
* vnf: cloudify_ims
-If you want to create a new test case you will have to create a new folder under
-the testcases directory (See next section for details).
+If you want to create a new test case you will have to create a new folder
+under the testcases directory (See next section for details).
Functest external test cases
============================
* doctor
* domino
* fds
- * orchestra_ims
- * parser
* promise
* refstack_defcore
* snaps_smoke
* functest-odl-sfc
+ * orchestra_clearwaterims
+ * orchestra_openims
* vyos_vrouter
+ * juju_vepc
External test cases integrated in previous versions but not released in
Euphrates:
* copper
+ * moon
* netready
* security_scan
The code to run these test cases is hosted in the repository of the project.
+Please note that orchestra test cases are hosted in Functest repository and not
+in orchestra repository. Vyos_vrouter and juju_vepc code is also hosted in
+functest as there are no dedicated projects.
Functest framework
Functest is a framework.
-Historically Functest is released as a docker file, including tools, scripts and
-a CLI to prepare the environment and run tests.
-It simplifies the integration of external test suites in CI pipeline and provide
-commodity tools to collect and display results.
+Historically Functest is released as a docker file, including tools, scripts
+and a CLI to prepare the environment and run tests.
+It simplifies the integration of external test suites in CI pipeline and
+provide commodity tools to collect and display results.
-Since Colorado, test categories also known as tiers have been created to
+Since Colorado, test categories also known as **tiers** have been created to
group similar tests, provide consistent sub-lists and at the end optimize
test duration for CI (see How To section).
* smoke
* features
* components
- * performance
* vnf
- * stress
-
-Note Functest deals with healthcheck, smoke, features, components and vnf tiers.
-Performance and stress tiers are out of scope.
Functest abstraction classes
============================
Feature, unit and vnf_base inherit from testcase::
- +-----------------------------------------+
- | |
- | TestCase |
- | |
- | - init() |
- | - run() |
- | - publish_report() |
- | - check_criteria() |
- | |
- +-----------------------------------------+
- | |
- V V
- +--------------------+ +--------------+ +--------------------------+
- | | | | | |
- | feature | | unit | | vnf |
- | | | | | |
- | | | | | - prepare() |
- | - execute() | | | | - deploy_orchestrator() |
- | BashFeature class | | | | - deploy_vnf() |
- | | | | | - test_vnf() |
- | | | | | - clean() |
- +--------------------+ +--------------+ +--------------------------+
+ +----------------------------------------------------------------+
+ | |
+ | TestCase |
+ | |
+ | - init() |
+ | - run() |
+ | - push_to_db() |
+ | - is_successful() |
+ | |
+ +----------------------------------------------------------------+
+ | | | |
+ V V V V
+ +--------------------+ +---------+ +------------------------+ +-----------------+
+ | | | | | | | |
+ | feature | | unit | | vnf | | robotframework |
+ | | | | | | | |
+ | | | | |- prepare() | | |
+ | - execute() | | | |- deploy_orchestrator() | | |
+ | BashFeature class | | | |- deploy_vnf() | | |
+ | | | | |- test_vnf() | | |
+ | | | | |- clean() | | |
+ +--------------------+ +---------+ +------------------------+ +-----------------+
Testcase
-========
+--------
.. raw:: html
:url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html
Feature
-=======
+-------
.. raw:: html
:url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html
Unit
-====
+----
.. raw:: html
:url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html
VNF
-===
+---
.. raw:: html
:url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html
+Robotframework
+--------------
+.. raw:: html
+ :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.robotframework.html
+
-see `Functest framework overview`_ to get code samples
+see `[5]`_ to get code samples
Functest util classes
=====================
In order to simplify the creation of test cases, Functest develops also some
-functions that can be used by any feature or internal test cases.
+functions that are used by internal test cases.
Several features are supported such as logger, configuration management and
-Openstack capabilities (snapshot, clean, tacker,..).
-These functions can be found under <repo>/functest/utils and can be described as
-follows::
+Openstack capabilities (tacker,..).
+These functions can be found under <repo>/functest/utils and can be described
+as follows::
functest/utils/
|-- config.py
|-- constants.py
- |-- decoratos.py
+ |-- decorators.py
|-- env.py
|-- functest_utils.py
- |-- openstack_clean.py
- |-- openstack_snapshot.py
|-- openstack_tacker.py
`-- openstack_utils.py
-Please note that it is possible to use snaps utils. SNAPS `[4]`_ is an OPNFV
-project providing OpenStack utils.
+It is recommended to use the SNAPS-OO library for deploying OpenStack
+instances. SNAPS `[4]`_ is an OPNFV project providing OpenStack utils.
TestAPI
=======
Functest is using the Test collection framework and the TestAPI developed by
-the OPNFV community. See `OPNFV Test collection framework`_ for details.
+the OPNFV community. See `[6]`_ for details.
Reporting
Dashboard
=========
-Additional dashboarding is managed at the testing group level, see
-`OPNFV Testing dashboard`_
+Additional dashboarding is managed at the testing group level, see `[7]`_ for
+details.
=======
How TOs
=======
-See `How to section`_ on Functest wiki
+See How to section on Functest wiki `[8]`_
==========
_`[4]`: https://git.opnfv.org/snaps/
-_`Functest framework overview` : http://testresults.opnfv.org/functest/framework/index.html
+_`[5]` : http://testresults.opnfv.org/functest/framework/index.html
-_`OPNFV Test collection framework`: TODO
+_`[6]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html
-_`OPNFV Testing dashboard`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba
+_`[7]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba
-_`How to section`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932
+_`[8]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932
IRC support chan: #opnfv-functest
Code Review / functest.git / blobdiff