-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. SPDX-License-Identifier: CC-BY-4.0
-=======
-License
-=======
-
-OPNFV Danube release note for Functest Docs
-(c) by Jose Lausuch (Ericsson)
-
-OPNFV Danube release note for Functest Docs
-are licensed under a Creative Commons Attribution 4.0 International License.
-You should have received a copy of the license along with this.
-If not, see <http://creativecommons.org/licenses/by/4.0/>.
-
-===========================================
-OPNFV Danube1.0 release note for Functest
-===========================================
+======================================
+OPNFV master release note for Functest
+======================================
Abstract
========
-This document describes the release note of Functest project.
-
-
-Version history
-===============
+This document contains the release notes of the Functest project.
-+------------+----------+------------------+------------------------+
-| **Date** | **Ver.** | **Author** | **Comment** |
-| | | | |
-+------------+----------+------------------+------------------------+
-| 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado |
-| | | (Orange) | release |
-+------------+----------+------------------+------------------------+
-| 2017-03-27 | 4.0.0 | Jose Lausuch | Functest for Danube |
-| | | (Ericsson) | release |
-+------------+----------+------------------+------------------------+
-
-OPNFV Danube Release
-======================
+OPNFV master Release
+====================
Functest deals with functional testing of the OPNFV solution.
-It includes test cases developed within the project and test cases developed in
-other OPNFV projects and other upstream communities.
+It includes test cases developed within the project, test cases developed in
+other OPNFV projects and it also integrates test cases from other upstream
+communities.
+
+OpenStack
+---------
The internal test cases are:
* vping_ssh
* vping_userdata
* tempest_smoke_serial
- * refstack_defcore
- * snaps_smoke
* rally_sanity
+ * refstack_defcore
+ * patrole
* odl
+ * odl-netvirt
+ * snaps_smoke
+ * neutron_trunk
* tempest_full_parallel
* rally_full
* cloudify_ims
+ * vyos_vrouter
+ * juju_epc
The OPNFV projects integrated into Functest framework for automation are:
- * barometer
- * bgpvpn
* doctor
- * domino
- * fds
- * multisite
- * netready
- * onos
+ * bgpvpn
* odl-sfc
- * odl-netvirt
- * orchestra_ims
+ * barometer
+ * fds
* parser
- * promise
- * security scan
- * vyos_vrouter
-The validation of a scenario requires a subset of these tests depending
-on the installer and the scenario.
+Kubernetes
+----------
+
+The internal test cases are:
+
+ * k8s_smoke
+ * k8s_conformance
+
+The OPNFV projects integrated into Functest framework for automation are:
-The test cases from vnf (cloudify_ims, orchestra_ims, vyos_vrouter) and
-component categories (tempest full parallel, Rally full) are not considered for
-scenario validation.
+ * stor4nfv
+ * clover
Release Data
============
+--------------------------------------+--------------------------------------+
| **Project** | functest |
-| | |
+--------------------------------------+--------------------------------------+
-| **Repo/tag** | danube.1.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release designation** | Danube base release |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release date** | April 3rd 2017 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | Danube base release |
-| | |
+| **Repository branch** | master |
+--------------------------------------+--------------------------------------+
Deliverables
Software
--------
- - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: danube.1.0)
-
- - The TestAPI Docker image: https://hub.docker.com/r/opnfv/testapi (tag:danube.1.0)
+ Functest Docker images (OpenStack):
+ * https://hub.docker.com/r/opnfv/functest-healthcheck
+ * https://hub.docker.com/r/opnfv/functest-smoke
+ * https://hub.docker.com/r/opnfv/functest-features
+ * https://hub.docker.com/r/opnfv/functest-components
+ * https://hub.docker.com/r/opnfv/functest-vnf
+ * https://hub.docker.com/r/opnfv/functest-parser
-Documents
----------
+ Functest Docker images (Kubernetes):
- - Installation/configuration guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
+ * https://hub.docker.com/r/opnfv/functest-kubernetes-healthcheck
+ * https://hub.docker.com/r/opnfv/functest-kubernetes-smoke
+ * https://hub.docker.com/r/opnfv/functest-kubernetes-features
- - User Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
+Docker tag for master: latest
- - Developer Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/developer/devguide/index.html
+Documents
+---------
+ * Config Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
+ * User Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
+ * Developer Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/developer/devguide/index.html
+ * API Docs: http://functest.readthedocs.io/en/latest/
Version change
==============
-Feature evolution
------------------
-
-- Adoption of SNAPS as middleware in 4 new test cases (connection_check, api_check,
-snaps_health_check and snaps_smoke)
-
-- Introduction of refstack suite
-
-- Support new odl suites (odl-netvirt, fds)
-
-- Introduction of VNF onboarding capabilities
-
-- Support of new feature projects (fds, netready, barometer, orchestra, vyos_vrouter)
-
-
-
-Framework
----------
-
- - Harmonization of the naming, better adoption of OpenStack coding conventions
-
- - Pythonization, rewritting of bash scripts
-
- - Introduction of abstraction classes to ease and harmonize the integration of
- test cases (internal or from feature projects)
-
- - New management of logger, env variables and configuration files
-
- - Creation of unit tests on the whole framework to ensure stability
-
- - Creation or ARM Functest docker
-
-
-Test API
----------
-
-- automatic documentation (html & pdf)
-
-- full dockerization and automation of the deployment on testresults.opnfv.org
-
-- automation of test database backup on artifact
-
-
-
-
-Scenario Matrix
-===============
-
-For Danube 1.0, Functest was tested on the following HA scenarios (new
-dabube scenarios in bold):
-
-+---------------------+---------+---------+---------+---------+
-| Scenario | Apex | Compass | Fuel | Joid |
-+=====================+=========+=========+=========+=========+
-| nosdn | X | X | X | X |
-+---------------------+---------+---------+---------+---------+
-| **fdio** | X | | | |
-+---------------------+---------+---------+---------+---------+
-| kvm | X | | X | |
-+---------------------+---------+---------+---------+---------+
-| **kvm_ovs_dpdk** | | | X | |
-+---------------------+---------+---------+---------+---------+
-| **kvm_ovs_dpdk-bar**| | | X | |
-+---------------------+---------+---------+---------+---------+
-| lxd | | | | X |
-+---------------------+---------+---------+---------+---------+
-| **ovs** | | | X | |
-+---------------------+---------+---------+---------+---------+
-| **openo** | | X | | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2 | | X | X | X |
-+---------------------+---------+---------+---------+---------+
-| odl-bgpvpn | X | | | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-bgpvpn | | | X | |
-+---------------------+---------+---------+---------+---------+
-| **odl_l2-fdio** | X | | | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-sfc | | | X | |
-+---------------------+---------+---------+---------+---------+
-| odl_l3 | X | X | X | |
-+---------------------+---------+---------+---------+---------+
-| **ocl** | | X | | |
-+---------------------+---------+---------+---------+---------+
-| onos | | X | | |
-+---------------------+---------+---------+---------+---------+
-| multisite | | | X | |
-+---------------------+---------+---------+---------+---------+
-
-Non HA scenarios:
-
-+---------------------+---------+---------+---------+---------+
-| Scenario | Apex | Compass | Fuel | Joid |
-+=====================+=========+=========+=========+=========+
-| nosdn | | | X | X |
-+---------------------+---------+---------+---------+---------+
-| kvm | | | X | |
-+---------------------+---------+---------+---------+---------+
-| **kvm_ovs_dpdk** | | | X | |
-+---------------------+---------+---------+---------+---------+
-| **kvm_ovs_dpdk-bar**| | | X | |
-+---------------------+---------+---------+---------+---------+
-| lxd | | | | X |
-+---------------------+---------+---------+---------+---------+
-| ovs | | | X | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2 | | | X | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-bgpvpn | | | X | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-fdio | X | | | |
-+---------------------+---------+---------+---------+---------+
-| **odl_l3-fdio** | X | | | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-sfc | | | X | |
-+---------------------+---------+---------+---------+---------+
-| odl_l3 | | | X | |
-+---------------------+---------+---------+---------+---------+
-| **odl_gluon** | X | | | |
-+---------------------+---------+---------+---------+---------+
-
-Colorado deprecated scenarios:
-
- * odl_l2-moon
- * onos-sfc
- * onos-noha
- * onos-sfc-noha
-
-For the scenario validation, we consider only the healthcheck, smoke and
-features categories. These tests are run systematically in the CI daily loops.
-
-Success criteria have been defined for these test cases, they can be
-PASS/FAIL or a success rate may be declared (100%, > 90%)
-All the criteria, as well as the test dependencies are declared in the
-ci/testcases.yaml file.
-
-The scoring for the Danube release per installer can be described as
-follows.
-
-The scoring is an indicator showing how many feature project test suites
-have been integrated on the scenario.
-
-The scoring equals the number of tests * succesful iteration of each
-test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
-
- e.g.
- apex/odl_l2-nofeature-ha
- tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
- Scoring = 21/21 = 7 * 3
-
-By default, if not specified, the scenarios are HA.
-HA means OpenStack High Availability (main services). Note that not
-all VIM (e.g. OpenStack) services are deployed in HA mode, as that
-depends upon support of the specific service for HA deployment.
-For example, in the Danube release, the Congress service
-is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
-support for HA operation has not yet been verified.
-See the release notes of the installers for more details.
-
-
-Apex
-----
-
-+------------------+---------+---------+-------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+==================+=========+=========+===================+
-************* TODO *****************************************
-| nosdn | 17/18 | 95% | `apex-res-174`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2 | 21/21 | 100% | `apex-res-175`_ |
-+------------------+---------+---------+-------------------+
-| odl_l3 | 15/18 | 83% | `apex-res-176`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn | 14/18 | 78% | `apex-res-235`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha | 18/24 | 75% | `apex-res-217`_ |
-+------------------+---------+---------+-------------------+
-| onos-nofeature-ha| 20/21 | 95% | `apex-res-423`_ |
-+------------------+---------+---------+-------------------+
-
-
-Compass
--------
-
-+------------------+---------+---------+-------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+==================+=========+=========+===================+
-************* TODO *****************************************
-| nosdn | 12/12 | 100% | `compass-res-55`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2 | 15/15 | 100% | `compass-res-59`_ |
-+------------------+---------+---------+-------------------+
-| odl_l3 | 9/12 | 75% | `compass-res-73`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-moon | 15/18 | 83% | `compass-res-567`_|
-+------------------+---------+---------+-------------------+
-| onos-ha | 15/15 | 100% | `compass-res-285`_|
-+------------------+---------+---------+-------------------+
-| onos-sfc-ha | 17/18 | 95% | `compass-res-76`_ |
-+------------------+---------+---------+-------------------+
-
-Note: all the Compass tests for Danube have been executed on virtual
-environment. Bare metal resources were used for Master branch.
-
-
-Fuel
-----
-
-+---------------------+---------+---------+-------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+=====================+=========+=========+===================+
-************* TODO *****************************************
-| nosdn | 18/18 | 100% | `fuel-res-129`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-noha | 15/15 | 100% | `fuel-res-154`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-ovs | 12/18 | 67%* | `fuel-res-213`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2 | 21/21 | 100% | `fuel-res-123`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn | 14/18 | 78% | `fuel-res-119`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn-noha | 14/15 | 93% | `fuel-res-160`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha | 6/21 | 29% | `fuel-res-219`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-sfc-ha | 16/21 | 76% | `fuel-res-376`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l3 | 15/18 | 83% | `fuel-res-115`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ |
-+---------------------+---------+---------+-------------------+
-| onos | 20/21 | 95% | `fuel-res-492`_ |
-+---------------------+---------+---------+-------------------+
-| onos-noha | 18/18 | 100% | `fuel-res-166`_ |
-+---------------------+---------+---------+-------------------+
-| onos-sfc | 24/24 | 100% | `fuel-res-124`_ |
-+---------------------+---------+---------+-------------------+
-| onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ |
-+---------------------+---------+---------+-------------------+
-| multisite | N.R | 100% | `fuel-res-8`_ |
-+---------------------+---------+---------+-------------------+
-
-*: all results passed, lacking iterations to reach the full score
-
-Results of Functest on AArch64 Danube 3.0
-
-+---------------------+---------+---------+----------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+=====================+=========+=========+======================+
-************* TODO *****************************************
-| nosdn | 18/18 | 100% | `fuel-arm-res-128`_ |
-+---------------------+---------+---------+----------------------+
-| odl_l2 | 21/21 | 100% | `fuel-arm-res-122`_ |
-+---------------------+---------+---------+----------------------+
-| odl_l2-noha | 18/18 | 100% | `fuel-arm-res-129`_ |
-+---------------------+---------+---------+----------------------+
-| odl_l3 | 17/18 | 95% | `fuel-arm-res-135`_ |
-+---------------------+---------+---------+----------------------+
-
-
-Joid
-----
-
-+---------------------+---------+---------+-----------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+=====================+=========+=========+=================+
-************* TODO *****************************************
-| nosdn | 18/18 | 100% | `joid-res-102`_ |
-+---------------------+---------+---------+-----------------+
-| nosdn-noha | 17/18 | 95% | `joid-res-93`_ |
-+---------------------+---------+---------+-----------------+
-| nosdn-lxd | 12/12 | 100% | `joid-res-104`_ |
-+---------------------+---------+---------+-----------------+
-| nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ |
-+---------------------+---------+---------+-----------------+
-| odl_l2 | 21/21 | 100% | `joid-res-103`_ |
-+---------------------+---------+---------+-----------------+
-| onos | 21/21 | 100% | `joid-res-345`_ |
-+---------------------+---------+---------+-----------------+
-| onos-sfc | 24/24 | 100% | `joid-res-97`_ |
-+---------------------+---------+---------+-----------------+
-
-It is highly recommended to install a json viewer in your browser
-(e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
-
-You can get additional details through test logs on http://artifacts.opnfv.org/.
-As no search engine is available on the OPNFV artifact web site you must
-retrieve the pod identifier on which the tests have been executed (see
-field pod in any of the results) then click on the selected POD and look
-for the date of the test you are interested in.
-
-The reporting pages can be found at:
-
- * apex: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-apex.html
- * compass: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-compass.html
- * fuel: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html
- * joid: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-joid.html
-
-Danube known restrictions/issues
-==================================
-
-************* TODO *****************************************
-
-+-----------+-----------+----------------------------------------------+
-| Installer | Scenario | Issue |
-+===========+===========+==============================================+
-| any | odl_l3-* | Tempest cases related to using floating IP |
-| | | addresses fail because of a known ODL bug. |
-| | | vPing_ssh test case is excluded for the same |
-| | | reason. |
-| | | https://jira.opnfv.org/browse/APEX-112 |
-| | | https://jira.opnfv.org/browse/FUNCTEST-445 |
-+-----------+-----------+----------------------------------------------+
-| apex/fuel | *-bgpvpn | vPing_ssh (floating ips not supported) and |
-| | | vIMS excluded. Some Tempest cases related to |
-| | | floating ips also excluded. Some performance |
-| | | issues have been detected in this scenario |
-| | | (i.e. BGPVPN extension enabled) when running |
-| | | commands against the OpenStack APIs, thus |
-| | | Rally sanity test case has been disabled. |
-| | | Performance issues seem to be connected to |
-| | | the ODL version. It is planned to reintroduce|
-| | | Rally sanity in Danube 2.0 with the |
-| | | adoption of ODL Boron release. |
-+-----------+-----------+----------------------------------------------+
-| fuel | odl | TestServerBasicOps test case skipped |
-| | | Tempest ssh client is hanging on opendaylight|
-| | | enabled envs (getting deadlock in paramiko |
-| | | recv_exit_status method) while trying to |
-| | | execute a command on a vm. |
-+-----------+-----------+----------------------------------------------+
-| joid | any | Tempest cases related to object storage |
-| | | excluded |
-| | | https://gerrit.opnfv.org/gerrit/#/c/17871/ |
-+-----------+-----------+----------------------------------------------+
-
-
-Test and installer/scenario dependencies
-========================================
-
-It is not always possible to run all the test cases on all the scenarios.
-The following table details the dependencies of the test cases per
-scenario. The scenario dependencies (installer or scenario) are detailed
-in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
-
-Test results
-============
-
-Test results are available in:
-
- - test results document: http://artifacts.opnfv.org/functest
-
- - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
-
- - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
-
-
-
-Open JIRA tickets
-=================
-
-+------------------+-----------------------------------------------+
-| JIRA | Description |
-+==================+===============================================+
-************* TODO *****************************************
-| | |
-| | |
-+------------------+-----------------------------------------------+
+New test cases
+--------------
+
+ * neutron_trunk
+ * patrole
+ * juju_epc
+ * k8s_smoke
+ * k8s_conformance
+ * stor4nfv
+ * clover
+
+Key changes
+-----------
+
+ * update test cases and containers to `OpenStack Pike`_
+ * move the framework into a separate project: Xtesting_
+ * ease running all containers thanks to default values
+ * clean interfaces with OPNFV Installers and Features
+ * rewrite all vnfs to allow multiple tests in parallel
+ * fully support non-default region names and Keystone v3 domains
+ * refactor all tempest-based test cases (e.g. refstack_defcore)
+ * remove obsolete OpenStack and Functest utils
+ * verify all changes via doc8 and yamllint too
+ * generate reports for all tempest-based test cases
+
+.. _`OpenStack Pike`: https://raw.githubusercontent.com/openstack/requirements/stable/pike/upper-constraints.txt
+.. _Xtesting: http://xtesting.readthedocs.io/en/latest/
+
+Key benefits
+------------
+
+ * the enduser can run all tests by setting only one input (EXTERNAL_NETWORK)
+ * the developer can only work on the test suites without diving into CI/CD
+ integration
+ * both OpenStack and Kubernetes deployments can be verified
+ * Functest test cases are trustable as they meet the best coding rules (unit
+ tests, coverage, linters, etc.)
+ * Functest can be reused in other projects (e.g.
+ `ONS: Re-using OPNFV framework tests for LFN projects`_)
+
+.. _`ONS: Re-using OPNFV framework tests for LFN projects`: https://wiki.lfnetworking.org/display/LN/LFN+Developer+Forum+Schedule?preview=/328197/328329/ONS-OPNFV%20framework%20tests%20for%20LFN%20projects.pdf
+
+Code quality
+------------
+
+ * pylint: ~9.5/10
+ * code coverage: ~70%
Useful links
============
- - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
-
- - wiki Functest Danube page: https://wiki.opnfv.org/display/functest/Functest+Danube
-
- - Functest repo: https://git.opnfv.org/cgit/functest
-
- - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
-
- - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
-
- - Functest IRC chan: #opnfv-functest
-
- - Reporting page: http://testresults.opnfv.org/reporting/danube.html
-
- - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/functest/ci/testcases.yaml
+ * wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
+ * Functest git repository: https://git.opnfv.org/cgit/functest
+ * Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
+ * JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
+ * Functest IRC channel: #opnfv-functest
+ * Reporting page: http://testresults.opnfv.org/reporting/master/functest/functest.html
Code Review / functest.git / blobdiff