1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
8 OPNFV Euphrates release note for Functest Docs
9 are licensed under a Creative Commons Attribution 4.0 International License.
10 You should have received a copy of the license along with this.
11 If not, see <http://creativecommons.org/licenses/by/4.0/>.
13 =============================================
14 OPNFV Euphrates 5.0 release note for Functest
15 =============================================
20 This document contains the release notes of the Functest project.
23 OPNFV Euphrates Release
24 ======================
26 Functest deals with functional testing of the OPNFV solution.
27 It includes test cases developed within the project, test cases developed in
28 other OPNFV projects and it also intgrates test cases from other upstream
31 The internal test cases are:
38 * tempest_smoke_serial
43 * tempest_full_parallel
48 The OPNFV projects integrated into Functest framework for automation are:
60 * orchestra_clearwaterims
66 +--------------------------------------+--------------------------------------+
67 | **Project** | functest |
69 +--------------------------------------+--------------------------------------+
70 | **Repo/tag** | opnfv-5.0.0 |
72 +--------------------------------------+--------------------------------------+
73 | **Release designation** | Euphrates initial release |
75 +--------------------------------------+--------------------------------------+
76 | **Release date** | October 20th 2017 |
78 +--------------------------------------+--------------------------------------+
79 | **Purpose of the delivery** | Euphrates first release |
81 +--------------------------------------+--------------------------------------+
89 Functest Docker images:
91 * https://hub.docker.com/r/opnfv/functest
92 * https://hub.docker.com/r/opnfv/functest-healthcheck
93 * https://hub.docker.com/r/opnfv/functest-smoke
94 * https://hub.docker.com/r/opnfv/functest-features
95 * https://hub.docker.com/r/opnfv/functest-components
96 * https://hub.docker.com/r/opnfv/functest-vnf
97 * https://hub.docker.com/r/opnfv/functest-parser
98 * https://hub.docker.com/r/opnfv/functest-restapi
100 TestAPI Docker image:
102 * https://hub.docker.com/r/opnfv/testapi
104 Docker tag to be pulled: opnfv-5.0.0
109 - Installation/configuration guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/user/configguide/index.html
111 - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/user/userguide/index.html
113 - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/developer/devguide/index.html
115 - API Docs: http://artifacts.opnfv.org/functest/docs/index.html
117 - Functest Framework presentation: http://testresults.opnfv.org/functest/framework/index.html
123 Functest now delivers light-weigth Docker images based on Alpine 3.6. The test cases are grouped into several categories
124 or tiers and must be run from the corresponding container. For example, to run the test case healthcheck, the image
125 opnfv/functest-healthcheck shall be used. The tiers and the tests within them are explained in detail in the User Guide.
127 For ARM (aarch64), the former Ubuntu image opnfv/functest shall be used since there are not any Alpine images built
128 for this architecture yet. It will probably be supported for Euphrates 5.1.
130 The Parser test case has its own dedicated Docker image since it requires libraries released for OpenStack Pike and
131 Euphrates is based on Ocata.
133 The Docker images do not contain OS images (Cirros, Ubuntu, Centos, ..) anymore. A script has been created under the
134 ci directory (download_images.sh) which contains all the needed images for all the tests. This file can be modified by
135 the user since not all the images might be used. It must be executed before starting Functest and attach the needed
136 images as a Docker volume. See Configuration Guide for more information.
138 The requirements have been split into 3 files:
139 * requirements.txt : lists all abstract dependencies of the OPNFV packages
140 * test-requirements.txt : lists all abstract dependencies required by Functest Unit Tests
141 * upper-constraints.txt : lists all concrete upstream dependencies required by Functest Docker container
143 OPNFV (test-)requirements.txt have been updated according to stable/ocata global-requirements.txt.
144 Functest uses (and completes) stable/ocata upper-constraints.txt in Dockerfiles and tox configuration.
145 The project relies on pbr, which injects requirements into the install_requires, tests_require and/or dependency_links
146 arguments to setup. It also supports conditional dependencies which can be added to the requirements (e.g. dnspython>=1.14.0;python_version=='2.7')
148 The way to manage logging has been centralized to a configuration file (logging.ini) which might be modified by the user.
149 By default, the output of executing the test cases is redirected to log files and is not displayed on the console, only result
150 messages and summary tables are displayed.
152 The framework has been refactored and all the test cases inherit from a core class TestCase. For Feature projects who develop
153 test cases, 2 sub-classes have been created:
154 - Feature: it implements all the needed functions and the developer must only overwrite the method "execute" (e.g. Barometer)
155 - BashFeature: it is used if the third party test case is a shell script. This way, the execution command must be specified in
156 testcases.yaml as the argument (e.g. Domino, Doctor)
158 An internal REST API has been introduced in Euphrates. The goal is to trigger Functest operations through an API in addition of the CLI.
159 This could be considered as a first step towards a pseudo micro services approach where the different test projects could expose and
160 consume APIs to the other test projects.
163 Euphrates known restrictions/issues
164 ===================================
165 +--------------+-----------+----------------------------------------------+
166 | Installer | Scenario | Issue |
167 +==============+===========+==============================================+
168 | fuel@aarch64 | any | Alpine containers not supported yet for ARM |
169 | | | The former Ubuntu Docker image shall be |
170 | | | still used for this architecture. |
171 +--------------+-----------+----------------------------------------------+
172 | fuel@aarch64 | any | VNF tier not supported yet. |
173 +--------------+-----------+----------------------------------------------+
174 | | | The test cases belonging to the VNF tier |
175 | any | any | have been only tested on os-nosdn-nofeature |
176 | | | scenarios and baremetal deployments. |
177 +--------------+-----------+----------------------------------------------+
178 | Joid | k8 | Functest does not offer test suites for |
179 | Compass | | Kubernetes scenarios yet. |
180 +--------------+-----------+----------------------------------------------+
183 Test and installer/scenario dependencies
184 ========================================
186 It is not always possible to run all the test cases on all the scenarios.
187 The scenario dependencies (installer or scenario) are detailed
188 in the different testcases.yaml for each tier:
190 * https://git.opnfv.org/functest/tree/docker/healthcheck/testcases.yaml?h=stable/euphrates
191 * https://git.opnfv.org/functest/tree/docker/smoke/testcases.yaml?h=stable/euphrates
192 * https://git.opnfv.org/functest/tree/docker/features/testcases.yaml?h=stable/euphrates
193 * https://git.opnfv.org/functest/tree/docker/components/testcases.yaml?h=stable/euphrates
194 * https://git.opnfv.org/functest/tree/docker/vnf/testcases.yaml?h=stable/euphrates
195 * https://git.opnfv.org/functest/tree/docker/parser/testcases.yaml?h=stable/euphrates
201 The Functest scenario status on October 20, 2017 can be seen on
202 http://testresults.opnfv.org/functest/euphrates/
204 Test logs are available in:
206 - test results logs from CI: http://artifacts.opnfv.org (within different directories 'logs_functest_X')
208 - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
210 - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
217 +------------------+-----------------------------------------------+
218 | JIRA | Description |
219 +==================+===============================================+
222 +------------------+-----------------------------------------------+
224 All the tickets that are not blocking have been fixed or postponed
231 - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
233 - wiki Functest Euphrates page: https://wiki.opnfv.org/display/functest/5.+Euphrates
235 - Functest repo: https://git.opnfv.org/cgit/functest
237 - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
239 - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
241 - Functest IRC chan: #opnfv-functest
243 - Reporting page: http://testresults.opnfv.org/reporting/euphrates.html