1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
4 ************************
5 Functest Developer Guide
6 ************************
16 Functest is a project dealing with functional testing.
17 The project produces its own internal test cases but can also be considered
18 as a framework to support feature and VNF onboarding project testing.
20 Therefore there are many ways to contribute to Functest. You can:
22 * Develop new internal test cases
23 * Integrate the tests from your feature project
24 * Develop the framework to ease the integration of external test cases
26 Additional tasks involving Functest but addressing all the test projects
27 may also be mentioned:
29 * The API / Test collection framework
31 * The automatic reporting portals
32 * The testcase catalog
34 This document describes how, as a developer, you may interact with the
35 Functest project. The first section details the main working areas of
36 the project. The Second part is a list of "How to" to help you to join
37 the Functest family whatever your field of interest is.
40 ========================
41 Functest developer areas
42 ========================
45 Functest High level architecture
46 ================================
48 Functest is a project delivering test containers dedicated to OPNFV.
49 It includes the tools, the scripts and the test scenarios.
50 In Euphrates Alpine containers have been introduced in order to lighten the
51 container and manage testing slicing. The new containers are created according
52 to the different tiers:
54 * functest-core: https://hub.docker.com/r/opnfv/functest-core/
55 * functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/
56 * functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/
57 * functest-features: https://hub.docker.com/r/opnfv/functest-features/
58 * functest-components: https://hub.docker.com/r/opnfv/functest-components/
59 * functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/
60 * functest-parser: https://hub.docker.com/r/opnfv/functest-parser/
61 * functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/
63 Standalone functest dockers are maintained for Euphrates but Alpine containers
66 Functest can be described as follow::
68 +----------------------+
70 | +--------------+ | +-------------------+
72 | | Tools | +------------------+ OPNFV |
73 | | Scripts | | | System Under Test |
76 | +--------------+ | +-------------------+
80 +----------------------+
82 Functest internal test cases
83 ============================
84 The internal test cases in Euphrates are:
96 * tempest_smoke_serial
97 * tempest_full_parallel
100 By internal, we mean that this particular test cases have been developed and/or
101 integrated by functest contributors and the associated code is hosted in the
103 An internal case can be fully developed or a simple integration of
104 upstream suites (e.g. Tempest/Rally developed in OpenStack, or odl suites are
105 just integrated in Functest).
107 The structure of this repository is detailed in `[1]`_.
108 The main internal test cases are in the opnfv_tests subfolder of the
109 repository, the internal test cases can be grouped by domain:
111 * sdn: odl, odl_netvirt, odl_fds
112 * openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_*
115 If you want to create a new test case you will have to create a new folder under
116 the testcases directory (See next section for details).
118 Functest external test cases
119 ============================
120 The external test cases are inherited from other OPNFV projects, especially the
123 The external test cases are:
135 * orchestra_clearwaterims
140 External test cases integrated in previous versions but not released in
149 The code to run these test cases is hosted in the repository of the project.
150 Please note that orchestra test cases are hosted in Functest repository and not
151 in orchestra repository. Vyos_vrouter and juju_vepc code is also hosted in
152 functest as there are no dedicated projects.
158 Functest is a framework.
160 Historically Functest is released as a docker file, including tools, scripts and
161 a CLI to prepare the environment and run tests.
162 It simplifies the integration of external test suites in CI pipeline and provide
163 commodity tools to collect and display results.
165 Since Colorado, test categories also known as **tiers** have been created to
166 group similar tests, provide consistent sub-lists and at the end optimize
167 test duration for CI (see How To section).
169 The definition of the tiers has been agreed by the testing working group.
178 Functest abstraction classes
179 ============================
181 In order to harmonize test integration, abstraction classes have been
184 * testcase: base for any test case
185 * unit: run unit tests as test case
186 * feature: abstraction for feature project
187 * vnf: abstraction for vnf onboarding
189 The goal is to unify the way to run tests in Functest.
191 Feature, unit and vnf_base inherit from testcase::
193 +----------------------------------------------------------------+
200 | - is_successful() |
202 +----------------------------------------------------------------+
205 +--------------------+ +---------+ +------------------------+ +-----------------+
207 | feature | | unit | | vnf | | robotframework |
209 | | | | |- prepare() | | |
210 | - execute() | | | |- deploy_orchestrator() | | |
211 | BashFeature class | | | |- deploy_vnf() | | |
212 | | | | |- test_vnf() | | |
213 | | | | |- clean() | | |
214 +--------------------+ +---------+ +------------------------+ +-----------------+
220 :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html
225 :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html
230 :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html
235 :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html
240 :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.robotframework.html
243 see `[5]`_ to get code samples
246 Functest util classes
247 =====================
249 In order to simplify the creation of test cases, Functest develops also some
250 functions that are used by internal test cases.
251 Several features are supported such as logger, configuration management and
252 Openstack capabilities (snapshot, clean, tacker,..).
253 These functions can be found under <repo>/functest/utils and can be described as
261 |-- functest_utils.py
262 |-- openstack_clean.py
263 |-- openstack_snapshot.py
264 |-- openstack_tacker.py
265 `-- openstack_utils.py
267 It is recommended to use the SNAPS-OO library for deploying OpenStack instances.
268 SNAPS `[4]`_ is an OPNFV project providing OpenStack utils.
273 Functest is using the Test collection framework and the TestAPI developed by
274 the OPNFV community. See `[6]`_ for details.
279 A web page is automatically generated every day to display the status based on
280 jinja2 templates `[3]`_.
286 Additional dashboarding is managed at the testing group level, see `[7]`_ for
294 See How to section on Functest wiki `[8]`_
301 _`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide
303 _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide
305 _`[3]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting
307 _`[4]`: https://git.opnfv.org/snaps/
309 _`[5]` : http://testresults.opnfv.org/functest/framework/index.html
311 _`[6]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html
313 _`[7]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba
315 _`[8]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932
317 IRC support chan: #opnfv-functest