1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
4 ******************************
5 OPNFV FUNCTEST developer guide
6 ******************************
14 +------------+----------+------------------+----------------------------------+
15 | **Date** | **Ver.** | **Author** | **Comment** |
17 +------------+----------+------------------+----------------------------------+
18 | 2017-01-23 | 1.0.0 | Morgan Richomme | Creation for Danube |
19 +------------+----------+------------------+----------------------------------+
20 | 2017-08-16 | 1.0.1 | Morgan Richomme | Adaptations for Euphrates |
21 | | | | - move generic part to Testing |
22 | | | | developer guide |
23 | | | | - move reporting part to functest|
25 | | | | - update test case list |
26 | | | | - include auto generated core |
27 | | | | documentation |
28 +------------+----------+------------------+----------------------------------+
34 Functest is a project dealing with functional testing.
35 Functest produces its own internal test cases but can also be considered
36 as a framework to support feature and VNF onboarding project testing.
38 Therefore there are many ways to contribute to Functest. You can:
40 * Develop new internal test cases
41 * Integrate the tests from your feature project
42 * Develop the framework to ease the integration of external test cases
44 Additional tasks involving Functest but addressing all the test projects
45 may also be mentioned:
47 * Develop the API / Test collection framework
48 * Develop dashboards or automatic reporting portals
50 This document describes how, as a developer, you may interact with the
51 Functest project. The first section details the main working areas of
52 the project. The Second part is a list of "How to" to help you to join
53 the Functest family whatever your field of interest is.
56 ========================
57 Functest developer areas
58 ========================
61 Functest High level architecture
62 ================================
64 Functest is a project delivering test containers dedicated to OPNFV.
65 It includes the tools, the scripts and the test scenarios.
66 Until Danube, Functest produced 2 docker files based on Ubuntu 14.04:
68 * x86 Functest: https://hub.docker.com/r/opnfv/functest/
69 * aarch64 Functest: https://hub.docker.com/r/opnfv/functest_aarch64/
71 In Euphrates Alpine containers have been introduced in order to lighten the
72 container and manage testing slicing, the new containers are created according
73 to the different tiers:
75 * functest-core: https://hub.docker.com/r/opnfv/functest-core/
76 * functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/
77 * functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/
78 * functest-features: TODO
79 * functest-components: TODO
82 Functest can be described as follow::
84 +----------------------+
86 | +--------------+ | +-------------------+
88 | | Tools | +------------------+ OPNFV |
89 | | Scripts | | | System Under Test |
90 | | Scenarios | +------------------+ |
91 | | | | Management | |
92 | +--------------+ | +-------------------+
96 +----------------------+
98 Functest internal test cases
99 ============================
100 The internal test cases in Euphrates are:
114 * tempest_full_parallel
115 * tempest_smoke_serial
118 By internal, we mean that this particular test cases have been developed and/or
119 integrated by functest contributors and the associated code is hosted in the
121 An internal case can be fully developed or a simple integration of
122 upstream suites (e.g. Tempest/Rally developed in OpenStack, or odl suites are
123 just integrated in Functest).
125 The structure of this repository is detailed in `[1]`_.
126 The main internal test cases are in the opnfv_tests subfolder of the
127 repository, the internal test cases are:
129 * sdn: odl, odl_netvirt, odl_fds, onos
130 * openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_*
133 If you want to create a new test case you will have to create a new folder under
134 the testcases directory (See next section for details).
136 Functest external test cases
137 ============================
138 The external test cases are inherited from other OPNFV projects, especially the
141 The external test cases are:
157 External test cases integrated in previous versions but not released in
165 The code to run these test cases is hosted in the repository of the project.
171 Functest is a framework.
173 Historically Functest is released as a docker file, including tools, scripts and
174 a CLI to prepare the environment and run tests.
175 It simplifies the integration of external test suites in CI pipeline and provide
176 commodity tools to collect and display results.
178 Since Colorado, test categories also known as tiers have been created to
179 group similar tests, provide consistent sub-lists and at the end optimize
180 test duration for CI (see How To section).
182 The definition of the tiers has been agreed by the testing working group.
193 Note Functest deals with healthcheck, smoke, features, components and vnf tiers.
194 Performance and stress tiers are out of scope.
196 Functest abstraction classes
197 ============================
199 In order to harmonize test integration, abstraction classes have been
202 * testcase: base for any test case
203 * unit: run unit tests as test case
204 * feature: abstraction for feature project
205 * vnf: abstraction for vnf onboarding
207 The goal is to unify the way to run tests in Functest.
209 Feature, unit and vnf_base inherit from testcase::
211 +-----------------------------------------+
217 | - publish_report() |
218 | - check_criteria() |
220 +-----------------------------------------+
223 +--------------------+ +--------------+ +--------------------------+
225 | feature | | unit | | vnf |
227 | | | | | - prepare() |
228 | - execute() | | | | - deploy_orchestrator() |
229 | BashFeature class | | | | - deploy_vnf() |
230 | | | | | - test_vnf() |
231 | | | | | - clean() |
232 +--------------------+ +--------------+ +--------------------------+
238 :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html
243 :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html
248 :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html
253 :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html
256 see `Functest framework overview`_ to get code samples
259 Functest util classes
260 =====================
262 In order to simplify the creation of test cases, Functest develops also some
263 functions that can be used by any feature or internal test cases.
264 Several features are supported such as logger, configuration management and
265 Openstack capabilities (snapshot, clean, tacker,..).
266 These functions can be found under <repo>/functest/utils and can be described as
274 |-- functest_utils.py
275 |-- openstack_clean.py
276 |-- openstack_snapshot.py
277 |-- openstack_tacker.py
278 `-- openstack_utils.py
280 Please note that it is possible to use snaps utils. SNAPS `[4]`_ is an OPNFV
281 project providing OpenStack utils.
286 Functest is using the Test collection framework and the TestAPI developed by
287 the OPNFV community. See `OPNFV Test collection framework`_ for details.
292 A web page is automatically generated every day to display the status based on
293 jinja2 templates `[3]`_.
299 Additional dashboarding is managed at the testing group level, see
300 `OPNFV Testing dashboard`_
307 See `How to section`_ on Functest wiki
314 _`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide
316 _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide
318 _`[3]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting
320 _`[4]`: https://git.opnfv.org/snaps/
322 _`Functest framework overview` : http://testresults.opnfv.org/functest/framework/index.html
324 _`OPNFV Test collection framework`: TODO
326 _`OPNFV Testing dashboard`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba
328 _`How to section`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932
330 IRC support chan: #opnfv-functest