X-Git-Url: https://gerrit.opnfv.org/gerrit/gitweb?a=blobdiff_plain;f=docs%2Ftesting%2Fdeveloper%2Fdevguide%2Findex.rst;h=d700ac1fe9743cd0cb87fa12cb37e39d54ce486a;hb=920e8d3cd4386e4d9085b5557b0c381d5f043830;hp=8a8b09cb19502b5921f879b10aa0cf150e3ea550;hpb=90d34ecea4e5eaa90db2092b831ece909d9c1477;p=functest.git diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index 8a8b09cb1..d700ac1fe 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -1,38 +1,19 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 -****************************** -OPNFV FUNCTEST developer guide -****************************** +************************ +Functest Developer Guide +************************ .. toctree:: :numbered: :maxdepth: 2 -Version history -=============== -+------------+----------+------------------+----------------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+----------------------------------+ -| 2017-01-23 | 1.0.0 | Morgan Richomme | Creation for Danube | -+------------+----------+------------------+----------------------------------+ -| 2017-08-16 | 1.0.1 | Morgan Richomme | Adaptations for Euphrates | -| | | | - move generic part to Testing | -| | | | developer guide | -| | | | - move reporting part to functest| -| | | | user guide | -| | | | - update test case list | -| | | | - include auto generated core | -| | | | documentation | -+------------+----------+------------------+----------------------------------+ - ============ Introduction ============ Functest is a project dealing with functional testing. -Functest produces its own internal test cases but can also be considered +The project produces its own internal test cases but can also be considered as a framework to support feature and VNF onboarding project testing. Therefore there are many ways to contribute to Functest. You can: @@ -44,8 +25,10 @@ Therefore there are many ways to contribute to Functest. You can: Additional tasks involving Functest but addressing all the test projects may also be mentioned: - * Develop the API / Test collection framework - * Develop dashboards or automatic reporting portals + * The API / Test collection framework + * The dashboards + * The automatic reporting portals + * The testcase catalog This document describes how, as a developer, you may interact with the Functest project. The first section details the main working areas of @@ -63,37 +46,37 @@ Functest High level architecture Functest is a project delivering test containers dedicated to OPNFV. It includes the tools, the scripts and the test scenarios. -Until Danube, Functest produced 2 docker files based on Ubuntu 14.04: - - * x86 Functest: https://hub.docker.com/r/opnfv/functest/ - * aarch64 Functest: https://hub.docker.com/r/opnfv/functest_aarch64/ - In Euphrates Alpine containers have been introduced in order to lighten the -container and manage testing slicing, the new containers are created according +container and manage testing slicing. The new containers are created according to the different tiers: * functest-core: https://hub.docker.com/r/opnfv/functest-core/ * functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/ * functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/ - * functest-features: TODO - * functest-components: TODO - * functest-vnf: TODO + * functest-features: https://hub.docker.com/r/opnfv/functest-features/ + * functest-components: https://hub.docker.com/r/opnfv/functest-components/ + * functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/ + * functest-parser: https://hub.docker.com/r/opnfv/functest-parser/ + * functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/ + +Standalone functest dockers are maintained for Euphrates but Alpine containers +are recommended. Functest can be described as follow:: - +----------------------+ - | | - | +--------------+ | +-------------------+ - | | | | Public | | - | | Tools | +------------------+ OPNFV | - | | Scripts | | | System Under Test | - | | Scenarios | +------------------+ | - | | | | Management | | - | +--------------+ | +-------------------+ - | | - | Functest Docker | - | | - +----------------------+ + +----------------------+ + | | + | +--------------+ | +-------------------+ + | | | | Public | | + | | Tools | +------------------+ OPNFV | + | | Scripts | | | System Under Test | + | | Scenarios | | | | + | | | | | | + | +--------------+ | +-------------------+ + | | + | Functest Docker | + | | + +----------------------+ Functest internal test cases ============================ @@ -101,18 +84,15 @@ The internal test cases in Euphrates are: * api_check - * cloudify_ims * connection_check + * snaps_health_check * vping_ssh * vping_userdata * odl - * odl-netvirt - * odl-fds * rally_full * rally_sanity - * snaps_health_check - * tempest_full_parallel * tempest_smoke_serial + * tempest_full_parallel * cloudify_ims By internal, we mean that this particular test cases have been developed and/or @@ -124,14 +104,15 @@ just integrated in Functest). The structure of this repository is detailed in `[1]`_. The main internal test cases are in the opnfv_tests subfolder of the -repository, the internal test cases are: +repository, the internal test cases can be grouped by domain: - * sdn: odl, odl_netvirt, odl_fds - * openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_* + * sdn: odl, odl_fds + * openstack: api_check, connection_check, snaps_health_check, vping_ssh, + vping_userdata, tempest_*, rally_* * vnf: cloudify_ims -If you want to create a new test case you will have to create a new folder under -the testcases directory (See next section for details). +If you want to create a new test case you will have to create a new folder +under the testcases directory (See next section for details). Functest external test cases ============================ @@ -145,23 +126,29 @@ The external test cases are: * doctor * domino * fds - * orchestra_ims * parser * promise * refstack_defcore * snaps_smoke * functest-odl-sfc + * orchestra_clearwaterims + * orchestra_openims * vyos_vrouter + * juju_vepc External test cases integrated in previous versions but not released in Euphrates: * copper + * moon * netready * security_scan The code to run these test cases is hosted in the repository of the project. +Please note that orchestra test cases are hosted in Functest repository and not +in orchestra repository. Vyos_vrouter and juju_vepc code is also hosted in +functest as there are no dedicated projects. Functest framework @@ -169,12 +156,12 @@ Functest framework Functest is a framework. -Historically Functest is released as a docker file, including tools, scripts and -a CLI to prepare the environment and run tests. -It simplifies the integration of external test suites in CI pipeline and provide -commodity tools to collect and display results. +Historically Functest is released as a docker file, including tools, scripts +and a CLI to prepare the environment and run tests. +It simplifies the integration of external test suites in CI pipeline and +provide commodity tools to collect and display results. -Since Colorado, test categories also known as tiers have been created to +Since Colorado, test categories also known as **tiers** have been created to group similar tests, provide consistent sub-lists and at the end optimize test duration for CI (see How To section). @@ -185,12 +172,7 @@ The tiers are: * smoke * features * components - * performance * vnf - * stress - -Note Functest deals with healthcheck, smoke, features, components and vnf tiers. -Performance and stress tiers are out of scope. Functest abstraction classes ============================ @@ -207,83 +189,86 @@ The goal is to unify the way to run tests in Functest. Feature, unit and vnf_base inherit from testcase:: - +-----------------------------------------+ - | | - | TestCase | - | | - | - init() | - | - run() | - | - publish_report() | - | - check_criteria() | - | | - +-----------------------------------------+ - | | - V V - +--------------------+ +--------------+ +--------------------------+ - | | | | | | - | feature | | unit | | vnf | - | | | | | | - | | | | | - prepare() | - | - execute() | | | | - deploy_orchestrator() | - | BashFeature class | | | | - deploy_vnf() | - | | | | | - test_vnf() | - | | | | | - clean() | - +--------------------+ +--------------+ +--------------------------+ + +----------------------------------------------------------------+ + | | + | TestCase | + | | + | - init() | + | - run() | + | - push_to_db() | + | - is_successful() | + | | + +----------------------------------------------------------------+ + | | | | + V V V V + +--------------------+ +---------+ +------------------------+ +-----------------+ + | | | | | | | | + | feature | | unit | | vnf | | robotframework | + | | | | | | | | + | | | | |- prepare() | | | + | - execute() | | | |- deploy_orchestrator() | | | + | BashFeature class | | | |- deploy_vnf() | | | + | | | | |- test_vnf() | | | + | | | | |- clean() | | | + +--------------------+ +---------+ +------------------------+ +-----------------+ Testcase -======== +-------- .. raw:: html :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html Feature -======= +------- .. raw:: html :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html Unit -==== +---- .. raw:: html :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html VNF -=== +--- .. raw:: html :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html +Robotframework +-------------- +.. raw:: html + :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.robotframework.html + -see `Functest framework overview`_ to get code samples +see `[5]`_ to get code samples Functest util classes ===================== In order to simplify the creation of test cases, Functest develops also some -functions that can be used by any feature or internal test cases. +functions that are used by internal test cases. Several features are supported such as logger, configuration management and -Openstack capabilities (snapshot, clean, tacker,..). -These functions can be found under /functest/utils and can be described as -follows:: +Openstack capabilities (tacker,..). +These functions can be found under /functest/utils and can be described +as follows:: functest/utils/ |-- config.py |-- constants.py - |-- decoratos.py + |-- decorators.py |-- env.py |-- functest_utils.py - |-- openstack_clean.py - |-- openstack_snapshot.py |-- openstack_tacker.py `-- openstack_utils.py -Please note that it is possible to use snaps utils. SNAPS `[4]`_ is an OPNFV -project providing OpenStack utils. +It is recommended to use the SNAPS-OO library for deploying OpenStack +instances. SNAPS `[4]`_ is an OPNFV project providing OpenStack utils. TestAPI ======= Functest is using the Test collection framework and the TestAPI developed by -the OPNFV community. See `OPNFV Test collection framework`_ for details. +the OPNFV community. See `[6]`_ for details. Reporting @@ -295,15 +280,15 @@ jinja2 templates `[3]`_. Dashboard ========= -Additional dashboarding is managed at the testing group level, see -`OPNFV Testing dashboard`_ +Additional dashboarding is managed at the testing group level, see `[7]`_ for +details. ======= How TOs ======= -See `How to section`_ on Functest wiki +See How to section on Functest wiki `[8]`_ ========== @@ -318,12 +303,12 @@ _`[3]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting _`[4]`: https://git.opnfv.org/snaps/ -_`Functest framework overview` : http://testresults.opnfv.org/functest/framework/index.html +_`[5]` : http://testresults.opnfv.org/functest/framework/index.html -_`OPNFV Test collection framework`: TODO +_`[6]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html -_`OPNFV Testing dashboard`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba +_`[7]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba -_`How to section`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932 +_`[8]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932 IRC support chan: #opnfv-functest