X-Git-Url: https://gerrit.opnfv.org/gerrit/gitweb?a=blobdiff_plain;f=docs%2Frelease%2Frelease-notes%2Ffunctest-release.rst;h=bb7600d352cd798cb2b837010595b21c08e49d5a;hb=refs%2Fchanges%2F91%2F59491%2F6;hp=423b9f9b84bf889dae93b8d900d585954fe674d6;hpb=19f9f2a99641cca98b896aa975a2182c43ba7f97;p=functest.git diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst index 423b9f9b8..bb7600d35 100644 --- a/docs/release/release-notes/functest-release.rst +++ b/docs/release/release-notes/functest-release.rst @@ -1,48 +1,24 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 -======= -License -======= - -OPNFV Danube release note for Functest Docs -(c) by Jose Lausuch (Ericsson) - -OPNFV Danube release note for Functest Docs -are licensed under a Creative Commons Attribution 4.0 International License. -You should have received a copy of the license along with this. -If not, see . - -=========================================== -OPNFV Danube1.0 release note for Functest -=========================================== +====================================== +OPNFV master release note for Functest +====================================== Abstract ======== -This document describes the release note of Functest project. - - -Version history -=============== +This document contains the release notes of the Functest project. -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado | -| | | (Orange) | release | -+------------+----------+------------------+------------------------+ -| 2017-03-27 | 4.0.0 | Jose Lausuch | Functest for Danube | -| | | (Ericsson) | release | -+------------+----------+------------------+------------------------+ - -OPNFV Danube Release -====================== +OPNFV master Release +==================== Functest deals with functional testing of the OPNFV solution. -It includes test cases developed within the project and test cases developed in -other OPNFV projects and other upstream communities. +It includes test cases developed within the project, test cases developed in +other OPNFV projects and it also integrates test cases from other upstream +communities. + +OpenStack +--------- The internal test cases are: @@ -52,57 +28,46 @@ The internal test cases are: * vping_ssh * vping_userdata * tempest_smoke_serial - * refstack_defcore - * snaps_smoke * rally_sanity + * refstack_defcore + * patrole * odl - * tempest_full_parallel + * snaps_smoke + * neutron_trunk + * tempest_full * rally_full * cloudify_ims + * vyos_vrouter + * juju_epc The OPNFV projects integrated into Functest framework for automation are: - * barometer - * bgpvpn * doctor - * domino - * fds - * multisite - * netready - * onos + * bgpvpn * odl-sfc - * odl-netvirt - * orchestra_ims - * parser - * promise - * security scan - * vyos_vrouter + * barometer + * fds -The validation of a scenario requires a subset of these tests depending -on the installer and the scenario. +Kubernetes +---------- + +The internal test cases are: -The test cases from vnf (cloudify_ims, orchestra_ims, vyos_vrouter) and -component categories (tempest full parallel, Rally full) are not considered for -scenario validation. + * k8s_smoke + * k8s_conformance + +The OPNFV projects integrated into Functest framework for automation are: + + * stor4nfv + * clover Release Data ============ +--------------------------------------+--------------------------------------+ | **Project** | functest | -| | | -+--------------------------------------+--------------------------------------+ -| **Repo/tag** | danube.1.0 | -| | | +--------------------------------------+--------------------------------------+ -| **Release designation** | Danube base release | -| | | -+--------------------------------------+--------------------------------------+ -| **Release date** | April 3rd 2017 | -| | | -+--------------------------------------+--------------------------------------+ -| **Purpose of the delivery** | Danube base release | -| | | +| **Repository branch** | master | +--------------------------------------+--------------------------------------+ Deliverables @@ -111,424 +76,87 @@ Deliverables Software -------- - - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: danube.1.0) + Functest Docker images (OpenStack): - - The TestAPI Docker image: https://hub.docker.com/r/opnfv/testapi (tag:danube.1.0) + * https://hub.docker.com/r/opnfv/functest-healthcheck + * https://hub.docker.com/r/opnfv/functest-smoke + * https://hub.docker.com/r/opnfv/functest-features + * https://hub.docker.com/r/opnfv/functest-components + * https://hub.docker.com/r/opnfv/functest-vnf + Functest Docker images (Kubernetes): -Documents ---------- - - - Installation/configuration guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html + * https://hub.docker.com/r/opnfv/functest-kubernetes-healthcheck + * https://hub.docker.com/r/opnfv/functest-kubernetes-smoke + * https://hub.docker.com/r/opnfv/functest-kubernetes-features - - User Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html +Docker tag for master: latest - - Developer Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/developer/devguide/index.html +Documents +--------- + * Config Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html + * User Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html + * Developer Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/developer/devguide/index.html + * API Docs: http://functest.readthedocs.io/en/latest/ Version change ============== -Feature evolution ------------------ - -- Adoption of SNAPS as middleware in 4 new test cases (connection_check, api_check, -snaps_health_check and snaps_smoke) - -- Introduction of refstack suite - -- Support new odl suites (odl-netvirt, fds) - -- Introduction of VNF onboarding capabilities - -- Support of new feature projects (fds, netready, barometer, orchestra, vyos_vrouter) - - - -Framework ---------- - - - Harmonization of the naming, better adoption of OpenStack coding conventions - - - Pythonization, rewritting of bash scripts - - - Introduction of abstraction classes to ease and harmonize the integration of - test cases (internal or from feature projects) - - - New management of logger, env variables and configuration files - - - Creation of unit tests on the whole framework to ensure stability - - - Creation or ARM Functest docker - - -Test API ---------- - -- automatic documentation (html & pdf) - -- full dockerization and automation of the deployment on testresults.opnfv.org - -- automation of test database backup on artifact - - - - -Scenario Matrix -=============== - -For Danube 1.0, Functest was tested on the following HA scenarios (new -dabube scenarios in bold): - -+---------------------+---------+---------+---------+---------+ -| Scenario | Apex | Compass | Fuel | Joid | -+=====================+=========+=========+=========+=========+ -| nosdn | X | X | X | X | -+---------------------+---------+---------+---------+---------+ -| **fdio** | X | | | | -+---------------------+---------+---------+---------+---------+ -| kvm | X | | X | | -+---------------------+---------+---------+---------+---------+ -| **kvm_ovs_dpdk** | | | X | | -+---------------------+---------+---------+---------+---------+ -| **kvm_ovs_dpdk-bar**| | | X | | -+---------------------+---------+---------+---------+---------+ -| lxd | | | | X | -+---------------------+---------+---------+---------+---------+ -| **ovs** | | | X | | -+---------------------+---------+---------+---------+---------+ -| **openo** | | X | | | -+---------------------+---------+---------+---------+---------+ -| odl_l2 | | X | X | X | -+---------------------+---------+---------+---------+---------+ -| odl-bgpvpn | X | | | | -+---------------------+---------+---------+---------+---------+ -| odl_l2-bgpvpn | | | X | | -+---------------------+---------+---------+---------+---------+ -| **odl_l2-fdio** | X | | | | -+---------------------+---------+---------+---------+---------+ -| odl_l2-sfc | | | X | | -+---------------------+---------+---------+---------+---------+ -| odl_l3 | X | X | X | | -+---------------------+---------+---------+---------+---------+ -| **ocl** | | X | | | -+---------------------+---------+---------+---------+---------+ -| onos | | X | | | -+---------------------+---------+---------+---------+---------+ -| multisite | | | X | | -+---------------------+---------+---------+---------+---------+ - -Non HA scenarios: - -+---------------------+---------+---------+---------+---------+ -| Scenario | Apex | Compass | Fuel | Joid | -+=====================+=========+=========+=========+=========+ -| nosdn | | | X | X | -+---------------------+---------+---------+---------+---------+ -| kvm | | | X | | -+---------------------+---------+---------+---------+---------+ -| **kvm_ovs_dpdk** | | | X | | -+---------------------+---------+---------+---------+---------+ -| **kvm_ovs_dpdk-bar**| | | X | | -+---------------------+---------+---------+---------+---------+ -| lxd | | | | X | -+---------------------+---------+---------+---------+---------+ -| ovs | | | X | | -+---------------------+---------+---------+---------+---------+ -| odl_l2 | | | X | | -+---------------------+---------+---------+---------+---------+ -| odl_l2-bgpvpn | | | X | | -+---------------------+---------+---------+---------+---------+ -| odl_l2-fdio | X | | | | -+---------------------+---------+---------+---------+---------+ -| **odl_l3-fdio** | X | | | | -+---------------------+---------+---------+---------+---------+ -| odl_l2-sfc | | | X | | -+---------------------+---------+---------+---------+---------+ -| odl_l3 | | | X | | -+---------------------+---------+---------+---------+---------+ -| **odl_gluon** | X | | | | -+---------------------+---------+---------+---------+---------+ - -Colorado deprecated scenarios: - - * odl_l2-moon - * onos-sfc - * onos-noha - * onos-sfc-noha - -For the scenario validation, we consider only the healthcheck, smoke and -features categories. These tests are run systematically in the CI daily loops. - -Success criteria have been defined for these test cases, they can be -PASS/FAIL or a success rate may be declared (100%, > 90%) -All the criteria, as well as the test dependencies are declared in the -ci/testcases.yaml file. - -The scoring for the Danube release per installer can be described as -follows. - -The scoring is an indicator showing how many feature project test suites -have been integrated on the scenario. - -The scoring equals the number of tests * succesful iteration of each -test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617 - - e.g. - apex/odl_l2-nofeature-ha - tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper - Scoring = 21/21 = 7 * 3 - -By default, if not specified, the scenarios are HA. -HA means OpenStack High Availability (main services). Note that not -all VIM (e.g. OpenStack) services are deployed in HA mode, as that -depends upon support of the specific service for HA deployment. -For example, in the Danube release, the Congress service -is deployed in non-HA mode even for HA OPNFV scenarios, as explicit -support for HA operation has not yet been verified. -See the release notes of the installers for more details. - - -Apex ----- - -+------------------+---------+---------+-------------------+ -| Scenario | Scoring | Success | Results | -| | | rate | | -+==================+=========+=========+===================+ -************* TODO ***************************************** -| nosdn | 17/18 | 95% | `apex-res-174`_ | -+------------------+---------+---------+-------------------+ -| odl_l2 | 21/21 | 100% | `apex-res-175`_ | -+------------------+---------+---------+-------------------+ -| odl_l3 | 15/18 | 83% | `apex-res-176`_ | -+------------------+---------+---------+-------------------+ -| odl_l2-bgpvpn | 14/18 | 78% | `apex-res-235`_ | -+------------------+---------+---------+-------------------+ -| odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ | -+------------------+---------+---------+-------------------+ -| odl_l2-sfc-noha | 18/24 | 75% | `apex-res-217`_ | -+------------------+---------+---------+-------------------+ -| onos-nofeature-ha| 20/21 | 95% | `apex-res-423`_ | -+------------------+---------+---------+-------------------+ - - -Compass -------- - -+------------------+---------+---------+-------------------+ -| Scenario | Scoring | Success | Results | -| | | rate | | -+==================+=========+=========+===================+ -************* TODO ***************************************** -| nosdn | 12/12 | 100% | `compass-res-55`_ | -+------------------+---------+---------+-------------------+ -| odl_l2 | 15/15 | 100% | `compass-res-59`_ | -+------------------+---------+---------+-------------------+ -| odl_l3 | 9/12 | 75% | `compass-res-73`_ | -+------------------+---------+---------+-------------------+ -| odl_l2-moon | 15/18 | 83% | `compass-res-567`_| -+------------------+---------+---------+-------------------+ -| onos-ha | 15/15 | 100% | `compass-res-285`_| -+------------------+---------+---------+-------------------+ -| onos-sfc-ha | 17/18 | 95% | `compass-res-76`_ | -+------------------+---------+---------+-------------------+ - -Note: all the Compass tests for Danube have been executed on virtual -environment. Bare metal resources were used for Master branch. - - -Fuel ----- - -+---------------------+---------+---------+-------------------+ -| Scenario | Scoring | Success | Results | -| | | rate | | -+=====================+=========+=========+===================+ -************* TODO ***************************************** -| nosdn | 18/18 | 100% | `fuel-res-129`_ | -+---------------------+---------+---------+-------------------+ -| nosdn-noha | 15/15 | 100% | `fuel-res-154`_ | -+---------------------+---------+---------+-------------------+ -| nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ | -+---------------------+---------+---------+-------------------+ -| nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ | -+---------------------+---------+---------+-------------------+ -| nosdn-ovs | 12/18 | 67%* | `fuel-res-213`_ | -+---------------------+---------+---------+-------------------+ -| nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2 | 21/21 | 100% | `fuel-res-123`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2-bgpvpn | 14/18 | 78% | `fuel-res-119`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2-bgpvpn-noha | 14/15 | 93% | `fuel-res-160`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2-sfc-noha | 6/21 | 29% | `fuel-res-219`_ | -+---------------------+---------+---------+-------------------+ -| odl_l2-sfc-ha | 16/21 | 76% | `fuel-res-376`_ | -+---------------------+---------+---------+-------------------+ -| odl_l3 | 15/18 | 83% | `fuel-res-115`_ | -+---------------------+---------+---------+-------------------+ -| odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ | -+---------------------+---------+---------+-------------------+ -| onos | 20/21 | 95% | `fuel-res-492`_ | -+---------------------+---------+---------+-------------------+ -| onos-noha | 18/18 | 100% | `fuel-res-166`_ | -+---------------------+---------+---------+-------------------+ -| onos-sfc | 24/24 | 100% | `fuel-res-124`_ | -+---------------------+---------+---------+-------------------+ -| onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ | -+---------------------+---------+---------+-------------------+ -| multisite | N.R | 100% | `fuel-res-8`_ | -+---------------------+---------+---------+-------------------+ - -*: all results passed, lacking iterations to reach the full score - -Results of Functest on AArch64 Danube 3.0 - -+---------------------+---------+---------+----------------------+ -| Scenario | Scoring | Success | Results | -| | | rate | | -+=====================+=========+=========+======================+ -************* TODO ***************************************** -| nosdn | 18/18 | 100% | `fuel-arm-res-128`_ | -+---------------------+---------+---------+----------------------+ -| odl_l2 | 21/21 | 100% | `fuel-arm-res-122`_ | -+---------------------+---------+---------+----------------------+ -| odl_l2-noha | 18/18 | 100% | `fuel-arm-res-129`_ | -+---------------------+---------+---------+----------------------+ -| odl_l3 | 17/18 | 95% | `fuel-arm-res-135`_ | -+---------------------+---------+---------+----------------------+ - - -Joid ----- - -+---------------------+---------+---------+-----------------+ -| Scenario | Scoring | Success | Results | -| | | rate | | -+=====================+=========+=========+=================+ -************* TODO ***************************************** -| nosdn | 18/18 | 100% | `joid-res-102`_ | -+---------------------+---------+---------+-----------------+ -| nosdn-noha | 17/18 | 95% | `joid-res-93`_ | -+---------------------+---------+---------+-----------------+ -| nosdn-lxd | 12/12 | 100% | `joid-res-104`_ | -+---------------------+---------+---------+-----------------+ -| nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ | -+---------------------+---------+---------+-----------------+ -| odl_l2 | 21/21 | 100% | `joid-res-103`_ | -+---------------------+---------+---------+-----------------+ -| onos | 21/21 | 100% | `joid-res-345`_ | -+---------------------+---------+---------+-----------------+ -| onos-sfc | 24/24 | 100% | `joid-res-97`_ | -+---------------------+---------+---------+-----------------+ - -It is highly recommended to install a json viewer in your browser -(e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/) - -You can get additional details through test logs on http://artifacts.opnfv.org/. -As no search engine is available on the OPNFV artifact web site you must -retrieve the pod identifier on which the tests have been executed (see -field pod in any of the results) then click on the selected POD and look -for the date of the test you are interested in. - -The reporting pages can be found at: - - * apex: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-apex.html - * compass: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-compass.html - * fuel: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html - * joid: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-joid.html - -Danube known restrictions/issues -================================== - -************* TODO ***************************************** - -+-----------+-----------+----------------------------------------------+ -| Installer | Scenario | Issue | -+===========+===========+==============================================+ -| any | odl_l3-* | Tempest cases related to using floating IP | -| | | addresses fail because of a known ODL bug. | -| | | vPing_ssh test case is excluded for the same | -| | | reason. | -| | | https://jira.opnfv.org/browse/APEX-112 | -| | | https://jira.opnfv.org/browse/FUNCTEST-445 | -+-----------+-----------+----------------------------------------------+ -| apex/fuel | *-bgpvpn | vPing_ssh (floating ips not supported) and | -| | | vIMS excluded. Some Tempest cases related to | -| | | floating ips also excluded. Some performance | -| | | issues have been detected in this scenario | -| | | (i.e. BGPVPN extension enabled) when running | -| | | commands against the OpenStack APIs, thus | -| | | Rally sanity test case has been disabled. | -| | | Performance issues seem to be connected to | -| | | the ODL version. It is planned to reintroduce| -| | | Rally sanity in Danube 2.0 with the | -| | | adoption of ODL Boron release. | -+-----------+-----------+----------------------------------------------+ -| fuel | odl | TestServerBasicOps test case skipped | -| | | Tempest ssh client is hanging on opendaylight| -| | | enabled envs (getting deadlock in paramiko | -| | | recv_exit_status method) while trying to | -| | | execute a command on a vm. | -+-----------+-----------+----------------------------------------------+ -| joid | any | Tempest cases related to object storage | -| | | excluded | -| | | https://gerrit.opnfv.org/gerrit/#/c/17871/ | -+-----------+-----------+----------------------------------------------+ - - -Test and installer/scenario dependencies -======================================== - -It is not always possible to run all the test cases on all the scenarios. -The following table details the dependencies of the test cases per -scenario. The scenario dependencies (installer or scenario) are detailed -in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml - -Test results -============ - -Test results are available in: - - - test results document: http://artifacts.opnfv.org/functest - - - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/ - - - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/ - - - -Open JIRA tickets -================= - -+------------------+-----------------------------------------------+ -| JIRA | Description | -+==================+===============================================+ -************* TODO ***************************************** -| | | -| | | -+------------------+-----------------------------------------------+ +New test cases +-------------- + + * neutron_trunk + * patrole + * juju_epc + * k8s_smoke + * k8s_conformance + * stor4nfv + * clover + +Key changes +----------- + + * update test cases and containers to `OpenStack Pike`_ + * move the framework into a separate project: Xtesting_ + * ease running all containers thanks to default values + * clean interfaces with OPNFV Installers and Features + * rewrite all vnfs to allow multiple tests in parallel + * fully support non-default region names and Keystone v3 domains + * refactor all tempest-based test cases (e.g. refstack_defcore) + * remove obsolete OpenStack and Functest utils + * verify all changes via doc8 and yamllint too + * generate reports for all tempest-based test cases + +.. _`OpenStack Pike`: https://raw.githubusercontent.com/openstack/requirements/stable/pike/upper-constraints.txt +.. _Xtesting: http://xtesting.readthedocs.io/en/latest/ + +Key benefits +------------ + + * the enduser can run all tests by setting only one input (EXTERNAL_NETWORK) + * the developer can only work on the test suites without diving into CI/CD + integration + * both OpenStack and Kubernetes deployments can be verified + * Functest test cases are trustable as they meet the best coding rules (unit + tests, coverage, linters, etc.) + * Functest can be reused in other projects (e.g. + `ONS: Re-using OPNFV framework tests for LFN projects`_) + +.. _`ONS: Re-using OPNFV framework tests for LFN projects`: https://wiki.lfnetworking.org/display/LN/LFN+Developer+Forum+Schedule?preview=/328197/328329/ONS-OPNFV%20framework%20tests%20for%20LFN%20projects.pdf + +Code quality +------------ + + * pylint: ~9.5/10 + * code coverage: ~70% Useful links ============ - - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing - - - wiki Functest Danube page: https://wiki.opnfv.org/display/functest/Functest+Danube - - - Functest repo: https://git.opnfv.org/cgit/functest - - - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/ - - - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611 - - - Functest IRC chan: #opnfv-functest - - - Reporting page: http://testresults.opnfv.org/reporting/danube.html - - - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/functest/ci/testcases.yaml + * wiki project page: https://wiki.opnfv.org/opnfv_functional_testing + * Functest git repository: https://git.opnfv.org/cgit/functest + * Functest CI dashboard: https://build.opnfv.org/ci/view/functest/ + * JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611 + * Functest IRC channel: #opnfv-functest + * Reporting page: http://testresults.opnfv.org/reporting/master/functest/functest.html