+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. SPDX-License-Identifier: CC-BY-4.0
+
=======
License
=======
-OPNFV Colorado release note for Functest Docs
-(c) by Morgan Richomme (Orange)
+OPNFV Danube release note for Functest Docs
+(c) by Jose Lausuch (Ericsson)
-OPNFV Colorado release note for Functest Docs
+OPNFV Danube release note for Functest Docs
are licensed under a Creative Commons Attribution 4.0 International License.
You should have received a copy of the license along with this.
If not, see <http://creativecommons.org/licenses/by/4.0/>.
===========================================
-OPNFV Colorado1.0 release note for Functest
+OPNFV Danube1.0 release note for Functest
===========================================
Abstract
| 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado |
| | | (Orange) | release |
+------------+----------+------------------+------------------------+
+| 2017-03-29 | 4.0.0 | Jose Lausuch | Functest for Danube |
+| | | (Ericsson) | release |
++------------+----------+------------------+------------------------+
+
-OPNFV Colorado Release
-=========================
+OPNFV Danube Release
+======================
Functest deals with functional testing of the OPNFV solution.
It includes test cases developed within the project and test cases developed in
The internal test cases are:
- * healthcheck
- * vPing ssh
- * vPing userdata
- * Tempest Smoke Serial
- * Rally Sanity
- * ODL
- * Tempest full parallel
- * Rally full
- * vIMS
+ * connection_check
+ * api_check
+ * snaps_health_check
+ * vping_ssh
+ * vping_userdata
+ * tempest_smoke_serial
+ * refstack_defcore
+ * snaps_smoke
+ * rally_sanity
+ * odl
+ * tempest_full_parallel
+ * rally_full
+ * cloudify_ims
The OPNFV projects integrated into Functest framework for automation are:
+ * barometer
* bgpvpn
- * Copper
- * Doctor
- * Domino
- * Moon
- * Multisite
- * ONOSFW
- * ONOS-sfc
- * ODL-sfc
- * Parser
- * Promise
- * Security scan
+ * doctor
+ * domino
+ * fds
+ * multisite
+ * netready
+ * onos
+ * odl-sfc
+ * odl-netvirt
+ * parser
+ * promise
+ * security scan
+ * orchestra_ims
+ * vyos_vrouter
The validation of a scenario requires a subset of these tests depending
on the installer and the scenario.
-The 3 last internal test cases (tempest full parallel, Rally full and
-vIMS) are not considered for scenario validation.
+The test cases from vnf (cloudify_ims, orchestra_ims, vyos_vrouter) and
+component categories (tempest full parallel, Rally full) are not considered for
+scenario validation.
Release Data
============
| **Project** | functest |
| | |
+--------------------------------------+--------------------------------------+
-| **Repo/tag** | colorado.1.0 |
+| **Repo/tag** | danube.1.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Release designation** | Colorado base release |
+| **Release designation** | Danube base release |
| | |
+--------------------------------------+--------------------------------------+
-| **Release date** | September 22 2016 |
+| **Release date** | March 31st 2017 |
| | |
+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | Colorado base release |
+| **Purpose of the delivery** | Danube base release |
| | |
+--------------------------------------+--------------------------------------+
Software
--------
- - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0)
+ - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: danube.1.0)
- - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0)
+ - The TestAPI Docker image: https://hub.docker.com/r/opnfv/testapi (tag:danube.1.0)
Documents
---------
- - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
+ - Installation/configuration guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
- - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
+ - User Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
- - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
+ - Developer Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/developer/devguide/index.html
Version change
Feature evolution
-----------------
- - refactoring of ODL functional tests (with upstream modifications)
+- Adoption of SNAPS as middleware in 4 new test cases (connection_check, api_check,
+ snaps_health_check and snaps_smoke)
- - refactoring of testapi (update, swagger documentation, dockerization)
+- Introduction of refstack suite
- - jenkins logs improvement
+- Support new odl suites (odl-netvirt, fds)
- - update integration of Doctor, Promise and SDNVPN projects
+- Introduction of VNF onboarding capabilities
- - split Tempest and rally into 2 different tests: smoke and full
+- Support of new feature projects (fds, netready, barometer, orchestra, vyos_vrouter)
- - vIMS test suite integration
- - adoption of Kibana for dashboarding
+Framework
+---------
+
+ - Harmonization of the naming, better adoption of OpenStack coding conventions
-New features
-------------
+ - Enhanced code to be more Object Oriented, removed bash scripts
- - Functest CLI to prepare and run the tests
+ - Introduction of abstraction classes to ease and harmonize the integration of
+ test cases (internal or from feature projects)
- - creation of the healthcheck test case
+ - New management of logger, env variables and configuration files
- - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
+ - Creation of unit tests on the whole framework to ensure stability
+
+ - Creation or ARM Functest docker
+
+
+Test API
+---------
- - integration of new OPNFV feature projects (copper, domino, multisite,
- moon, parser, onos-sfc, odl-sfc, security scan)
+- Automatic documentation (html & pdf)
- - introduction of test tiers in functest framework
+- Full dockerization and automation of the deployment on testresults.opnfv.org
- - automatic reporting
+- Automation of test database backup on artifact
- - introduction of a jenkins summary table
- - support of ARM architecture
+New internal tests cases
+------------------------
+
+- connection_check
+
+- api_check
+
+- snaps_health_check (replacing shell script healtcheck)
+
+- refstack_defcore
+
+- snaps_smoke
+
+- vyos_vrouter
Scenario Matrix
===============
-For Colorado 1.0, Functest was tested on the following scenarios (if not
-precised, the scenario is a HA scenario):
+For Danube 1.0, Functest was tested on the following HA scenarios (new
+dabube scenarios in bold):
+---------------------+---------+---------+---------+---------+
| Scenario | Apex | Compass | Fuel | Joid |
+=====================+=========+=========+=========+=========+
| nosdn | X | X | X | X |
+---------------------+---------+---------+---------+---------+
-| nosdn-noha | | | X | X |
+| **fdio** | X | | | |
+---------------------+---------+---------+---------+---------+
-| kvm | | | X | |
+| kvm | X | | X | |
+---------------------+---------+---------+---------+---------+
-| kvm-noha | | | X | |
+| **kvm_ovs_dpdk** | | | X | |
+---------------------+---------+---------+---------+---------+
-| lxd | | | | X |
+| **kvm_ovs_dpdk-bar**| | | X | |
+---------------------+---------+---------+---------+---------+
-| lxd-noha | | | | X |
-+---------------------+---------+---------+---------+---------+
-| ovs-noha (dpdk) | | | X | |
+| lxd | | | | X |
+---------------------+---------+---------+---------+---------+
-| odl_l2 | X | X | X | X |
+| **ovs** | | | X | |
+---------------------+---------+---------+---------+---------+
-| odl_l2-noha | | | X | |
+| **openo** | | X | | |
+---------------------+---------+---------+---------+---------+
-| odl_l2-bgpvpn | X | | X | |
+| odl_l2 | | X | X | X |
+---------------------+---------+---------+---------+---------+
-| odl_l2-bgpvpn-noha| | | X | |
+| odl-bgpvpn | X | | | |
+---------------------+---------+---------+---------+---------+
-| odl_l2-fdio-noha | X | | | |
+| odl_l2-bgpvpn | | | X | |
+---------------------+---------+---------+---------+---------+
-| odl_l2-moon | | X | | |
+| **odl_l2-fdio** | X | | | |
+---------------------+---------+---------+---------+---------+
-| odl_l2-sfc-noha | X | | X | |
+| odl_l2-sfc | | | X | |
+---------------------+---------+---------+---------+---------+
| odl_l3 | X | X | X | |
+---------------------+---------+---------+---------+---------+
-| odl_l3-noha | | | X | |
-+---------------------+---------+---------+---------+---------+
-| onos | | X | X | X |
-+---------------------+---------+---------+---------+---------+
-| onos-noha | | | X | |
+| **ocl** | | X | | |
+---------------------+---------+---------+---------+---------+
-| onos-sfc | | X | X | X |
-+---------------------+---------+---------+---------+---------+
-| onos-sfc-noha | | | X | |
+| onos | | X | | |
+---------------------+---------+---------+---------+---------+
| multisite | | | X | |
+---------------------+---------+---------+---------+---------+
-For Colorado 2.0, the following scenarios have been re-released:
+Non HA scenarios:
+---------------------+---------+---------+---------+---------+
| Scenario | Apex | Compass | Fuel | Joid |
+=====================+=========+=========+=========+=========+
-| onos | X | X | X | X |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-sfc-ha | | | X | |
-+---------------------+---------+---------+---------+---------+
-
-
-In Colorado, the functional tests have been sliced in 6 different
-categories:
-
-+----------------+-----------------------------------------------+
-| Category | Description |
-+================+===============================================+
-| healthcheck | Basic OpenStack commands |
-+----------------+-----------------------------------------------+
-| smoke | vPings, Tempest and rally smoke tests |
-+----------------+-----------------------------------------------+
-| sdn_suites | Specific SDN feature tests |
-+----------------+-----------------------------------------------+
-| features | OPNFV feature project functional test suites |
-+----------------+-----------------------------------------------+
-| openstack | Advanced, long duration OpenStack tests |
-| | (Tempest and Rally full suite). Each test may |
-| | last several hours |
-+----------------+-----------------------------------------------+
-| vnf | Complex scenarios dealing with orchestration, |
-| | VNF deployment and tests (e.g. vIMS) |
-+----------------+-----------------------------------------------+
-
-For the scenario validation, we consider only the healthcheck, smoke,
-sdn_suites and features categories. These tests are run systematically
-in the CI daily loops.
-
-Success criteria have been defined for these test cases, they can be
-PASS/FAIL or a success rate may be declared (100%, > 90%)
+| nosdn | | | X | X |
++---------------------+---------+---------+---------+---------+
+| kvm | | | X | |
++---------------------+---------+---------+---------+---------+
+| **kvm_ovs_dpdk** | | | X | |
++---------------------+---------+---------+---------+---------+
+| **kvm_ovs_dpdk-bar**| | | X | |
++---------------------+---------+---------+---------+---------+
+| lxd | | | | X |
++---------------------+---------+---------+---------+---------+
+| ovs | | | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l2 | | | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l2-bgpvpn | | | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l2-fdio | X | | | |
++---------------------+---------+---------+---------+---------+
+| **odl_l3-fdio** | X | | | |
++---------------------+---------+---------+---------+---------+
+| odl_l2-sfc | | | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l3 | | | X | |
++---------------------+---------+---------+---------+---------+
+| **odl_gluon** | X | | | |
++---------------------+---------+---------+---------+---------+
+
+Colorado deprecated scenarios:
+
+ * odl_l2-moon
+ * onos-sfc
+ * onos-noha
+ * onos-sfc-noha
+
+For the scenario validation, we consider only the healthcheck, smoke and
+features categories. These tests are run systematically in the CI daily loops.
+
+Success criterias have been defined for these test cases, they can be
+PASS/FAIL or a success rate may be declared (100%, > 90%).
All the criteria, as well as the test dependencies are declared in the
-ci/testcases.yaml file.
+testcases.yaml file located in the ci directory of the repository.
-The scoring for the Colorado release per installer can be described as
+The scoring for the Danube release per installer can be described as
follows.
The scoring is an indicator showing how many feature project test suites
-have been integrated on the scenario.
+have been integrated in the scenario.
-The scoring equals the number of tests * succesful iteration of each
+The scoring equals the number of tests * successful iteration of each
test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
e.g.
HA means OpenStack High Availability (main services). Note that not
all VIM (e.g. OpenStack) services are deployed in HA mode, as that
depends upon support of the specific service for HA deployment.
-For example, in the Colorado release, the Congress service
+For example, in the Danube release, the Congress service
is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
support for HA operation has not yet been verified.
See the release notes of the installers for more details.
Apex
----
-+------------------+---------+---------+-------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+==================+=========+=========+===================+
-| nosdn | 17/18 | 95% | `apex-res-174`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2 | 21/21 | 100% | `apex-res-175`_ |
-+------------------+---------+---------+-------------------+
-| odl_l3 | 15/18 | 83% | `apex-res-176`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn | 14/18 | 78% | `apex-res-235`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha | 18/24 | 75% | `apex-res-217`_ |
-+------------------+---------+---------+-------------------+
-| onos-nofeature-ha| 20/21 | 95% | `apex-res-423`_ |
-+------------------+---------+---------+-------------------+
-
++------------------+---------+---------+-----------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++==================+=========+=========+=================+
+| nosdn | 33/33 | 100% | `apex-res-1`_ |
++------------------+---------+---------+-----------------+
+| odl_l3 | 27/33 | 82% | `apex-res-2`_ |
++------------------+---------+---------+-----------------+
+| odl-bgpvpn | 26/30 | 87% | `apex-res-3`_ |
++------------------+---------+---------+-----------------+
+| odl-gluon | 30/36 | 83% | `apex-res-4`_ |
++------------------+---------+---------+-----------------+
+| kvm | 32/33 | 97% | `apex-res-5`_ |
++------------------+---------+---------+-----------------+
+| odl_l2-fdio | 28/36 | 78% | `apex-res-6`_ |
++------------------+---------+---------+-----------------+
+| odl_l2-fdio-noha | 30/36 | 83% | `apex-res-7`_ |
++------------------+---------+---------+-----------------+
+| odl_l3-fdio-noha | 26/30 | 87% | `apex-res-8`_ |
++------------------+---------+---------+-----------------+
+| fdio | 6/30 | 20% | `apex-res-9`_ |
++------------------+---------+---------+-----------------+
Compass
-------
-+------------------+---------+---------+-------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+==================+=========+=========+===================+
-| nosdn | 12/12 | 100% | `compass-res-55`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2 | 15/15 | 100% | `compass-res-59`_ |
-+------------------+---------+---------+-------------------+
-| odl_l3 | 9/12 | 75% | `compass-res-73`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-moon | 15/18 | 83% | `compass-res-567`_|
-+------------------+---------+---------+-------------------+
-| onos-ha | 15/15 | 100% | `compass-res-285`_|
-+------------------+---------+---------+-------------------+
-| onos-sfc-ha | 17/18 | 95% | `compass-res-76`_ |
-+------------------+---------+---------+-------------------+
-
-Note: all the Compass tests for Colorado have been executed on virtual
++------------------+---------+---------+------------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++==================+=========+=========+==================+
+| nosdn | 29/30 | 97% | `compass-res-1`_ |
++------------------+---------+---------+------------------+
+| odl_l2 | 28/33 | 84% | `compass-res-2`_ |
++------------------+---------+---------+------------------+
+| odl_l3 | 21/30 | 70% | `compass-res-3`_ |
++------------------+---------+---------+------------------+
+| onos | 28/33 | 84% | `compass-res-4`_ |
++------------------+---------+---------+------------------+
+| openo | 28/30 | 93% | `compass-res-5`_ |
++------------------+---------+---------+------------------+
+| ocl | 4/30 | 13% | `compass-res-6`_ |
++------------------+---------+---------+------------------+
+
+Note: all the Compass tests for Danube have been executed on virtual
environment. Bare metal resources were used for Master branch.
Fuel
----
-+---------------------+---------+---------+-------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+=====================+=========+=========+===================+
-| nosdn | 18/18 | 100% | `fuel-res-129`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-noha | 15/15 | 100% | `fuel-res-154`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-ovs | 12/18 | 67%* | `fuel-res-213`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2 | 21/21 | 100% | `fuel-res-123`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn | 14/18 | 78% | `fuel-res-119`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn-noha | 14/15 | 93% | `fuel-res-160`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha | 6/21 | 29% | `fuel-res-219`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-sfc-ha | 16/21 | 76% | `fuel-res-376`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l3 | 15/18 | 83% | `fuel-res-115`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ |
-+---------------------+---------+---------+-------------------+
-| onos | 20/21 | 95% | `fuel-res-492`_ |
-+---------------------+---------+---------+-------------------+
-| onos-noha | 18/18 | 100% | `fuel-res-166`_ |
-+---------------------+---------+---------+-------------------+
-| onos-sfc | 24/24 | 100% | `fuel-res-124`_ |
-+---------------------+---------+---------+-------------------+
-| onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ |
-+---------------------+---------+---------+-------------------+
-| multisite | N.R | 100% | `fuel-res-8`_ |
-+---------------------+---------+---------+-------------------+
-
-*: all results passed, lacking iterations to reach the full score
-
-Results of Functest on AArch64 Colorado 3.0
-
-+---------------------+---------+---------+----------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+=====================+=========+=========+======================+
-| nosdn | 18/18 | 100% | `fuel-arm-res-128`_ |
-+---------------------+---------+---------+----------------------+
-| odl_l2 | 21/21 | 100% | `fuel-arm-res-122`_ |
-+---------------------+---------+---------+----------------------+
-| odl_l2-noha | 18/18 | 100% | `fuel-arm-res-129`_ |
-+---------------------+---------+---------+----------------------+
-| odl_l3 | 17/18 | 95% | `fuel-arm-res-135`_ |
-+---------------------+---------+---------+----------------------+
++----------------------+---------+---------+----------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++======================+=========+=========+================+
+| nosdn | 37/39 | 95% | `fuel-res-1`_ |
++----------------------+---------+---------+----------------+
+| nosdn-noha | 36/36 | 100% | `fuel-res-2`_ |
++----------------------+---------+---------+----------------+
+| nosdn-kvm | 37/39 | 95% | `fuel-res-3`_ |
++----------------------+---------+---------+----------------+
+| nosdn-kvm-noha | 36/36 | 100% | `fuel-res-4`_ |
++----------------------+---------+---------+----------------+
+| nosdn-ovs | 38/39 | 97% | `fuel-res-5`_ |
++----------------------+---------+---------+----------------+
+| nosdn-ovs-noha | 36/36 | 100% | `fuel-res-6`_ |
++----------------------+---------+---------+----------------+
+| odl_l2 | 42/42 | 100% | `fuel-res-7`_ |
++----------------------+---------+---------+----------------+
+| odl_l2-noha | 36/39 | 92% | `fuel-res-8`_ |
++----------------------+---------+---------+----------------+
+| odl_l2-sfc | 40/45 | 89% | `fuel-res-11`_ |
++----------------------+---------+---------+----------------+
+| odl_l2-sfc-noha | 36/42 | 86% | `fuel-res-12`_ |
++----------------------+---------+---------+----------------+
+| odl_l3 | 34/39 | 87% | `fuel-res-13`_ |
++----------------------+---------+---------+----------------+
+| odl_l3-noha | 34/36 | 94% | `fuel-res-14`_ |
++----------------------+---------+---------+----------------+
+| kvm_ovs_dpdk | 6/39 | 15% | `fuel-res-15`_ |
++----------------------+---------+---------+----------------+
+| kvm_ovs_dpdk_noha | 36/36 | 100% | `fuel-res-16`_ |
++----------------------+---------+---------+----------------+
+| kvm_ovs_dpdk_bar | 6/42 | 14% | `fuel-res-17`_ |
++----------------------+---------+---------+----------------+
+| kvm_ovs_dpdk_bar_noha| 38/39 | 97% | `fuel-res-18`_ |
++----------------------+---------+---------+----------------+
+
+
Joid
----
-+---------------------+---------+---------+-----------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+=====================+=========+=========+=================+
-| nosdn | 18/18 | 100% | `joid-res-102`_ |
-+---------------------+---------+---------+-----------------+
-| nosdn-noha | 17/18 | 95% | `joid-res-93`_ |
-+---------------------+---------+---------+-----------------+
-| nosdn-lxd | 12/12 | 100% | `joid-res-104`_ |
-+---------------------+---------+---------+-----------------+
-| nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ |
-+---------------------+---------+---------+-----------------+
-| odl_l2 | 21/21 | 100% | `joid-res-103`_ |
-+---------------------+---------+---------+-----------------+
-| onos | 21/21 | 100% | `joid-res-345`_ |
-+---------------------+---------+---------+-----------------+
-| onos-sfc | 24/24 | 100% | `joid-res-97`_ |
-+---------------------+---------+---------+-----------------+
++---------------------+---------+---------+---------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++=====================+=========+=========+===============+
+| nosdn | 32/33 | 97% | `joid-res-1`_ |
++---------------------+---------+---------+---------------+
+| nosdn-noha | 31/33 | 94% | `joid-res-2`_ |
++---------------------+---------+---------+---------------+
+| nosdn-lxd | 18/24 | 75% | `joid-res-3`_ |
++---------------------+---------+---------+---------------+
+| nosdn-lxd-noha | 17/24 | 71% | `joid-res-4`_ |
++---------------------+---------+---------+---------------+
+| odl_l2 | 9/36 | 25% | `joid-res-5`_ |
++---------------------+---------+---------+---------------+
It is highly recommended to install a json viewer in your browser
(e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
The reporting pages can be found at:
- * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
- * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
- * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
- * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html
+ * apex: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-apex.html
+ * compass: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-compass.html
+ * fuel: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html
+ * joid: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-joid.html
-Colorado known restrictions/issues
+Danube known restrictions/issues
==================================
+************* TODO *****************************************
+
+-----------+-----------+----------------------------------------------+
| Installer | Scenario | Issue |
+===========+===========+==============================================+
-| any | odl_l3-* | Tempest cases related to using floating IP |
-| | | addresses fail because of a known ODL bug. |
-| | | vPing_ssh test case is excluded for the same |
-| | | reason. |
-| | | https://jira.opnfv.org/browse/APEX-112 |
-| | | https://jira.opnfv.org/browse/FUNCTEST-445 |
-+-----------+-----------+----------------------------------------------+
-| apex/fuel | *-bgpvpn | vPing_ssh (floating ips not supported) and |
-| | | vIMS excluded. Some Tempest cases related to |
-| | | floating ips also excluded. Some performance |
-| | | issues have been detected in this scenario |
-| | | (i.e. BGPVPN extension enabled) when running |
-| | | commands against the OpenStack APIs, thus |
-| | | Rally sanity test case has been disabled. |
-| | | Performance issues seem to be connected to |
-| | | the ODL version. It is planned to reintroduce|
-| | | Rally sanity in Colorado 2.0 with the |
-| | | adoption of ODL Boron release. |
+| fuel | odl_-* | Tempest test case "TestServerBasicOps" |
+| | | disabled due to bug `tempest-bug`_ |
+-----------+-----------+----------------------------------------------+
-| apex | *-fdio | Due to late integration, fdio decided to |
-| | | focus on mandatory tests and exclude feature |
-| | | tests (copper, doctor, security_scan) from |
-| | | its scenarios |
+| apex/fuel | *-bgpvpn | Due to some instabilities in the bgpvpn |
+| | | test case, the scenario has been postponed |
+| | | to Danube 2.0 |
+-----------+-----------+----------------------------------------------+
-| compass | moon | First ODL test FAILS because ODL/Openstack |
-| | | federation done in moon is partial. Only |
-| | | MD-SAL is federated (not AD-SAL) |
-+-----------+-----------+----------------------------------------------+
-| fuel | any | TestServerBasicOps test case skipped |
-| | | https://gerrit.opnfv.org/gerrit/#/c/19635/ |
-+-----------+-----------+----------------------------------------------+
-| fuel | kvm | vPing_ssh and vIMS excluded (metadata related|
-| | | scenarios) |
-+-----------+-----------+----------------------------------------------+
-| fuel | multisite | random errors running multisite. A known bug |
-| | | in keystone mitaka, due to which memcache |
-| | | raises exception and keystone becomes |
-| | | unresponsive |
-| | | bugs.launchpad.net/keystone/+bug/1600394 |
-| | | workaround consists in restarting memcache on|
-| | | server |
+| apex | *-gluon | vPing_ssh disabled due to floating ips |
+| | | not working 100% of the times. |
+| | | Tempest test "test_reboot_server_hard" |
+| | | disabled due to bug `gluon-bug`_ |
+-----------+-----------+----------------------------------------------+
| joid | any | Tempest cases related to object storage |
| | | excluded |
-| | | https://gerrit.opnfv.org/gerrit/#/c/17871/ |
+-----------+-----------+----------------------------------------------+
-| joid | domino | Domino tests are skipped in CI. However the |
-| | | test case can be run by manually setting |
-| | | IS_IPandKEY_CONFIGURED=true after manually |
-| | | setting properly the IP addresses of the 3 |
-| | | Controller nodes in the configuration file |
-| | | /repos/domino/tests/run_multinode.sh |
+| any | any | The VNF tier has not been fully tested |
+| | | since it has not been run in daily loops |
+| | | in CI. Weekly jobs have been activated |
+| | | a bit late in the process and have not been |
+| | | used to validate the scenarios. |
+-----------+-----------+----------------------------------------------+
-
Test and installer/scenario dependencies
========================================
+------------------+-----------------------------------------------+
| JIRA | Description |
+==================+===============================================+
-| `FUNCTEST-446`_ | Cleanup ODL-SFC output in Functest execution |
-| | Impact on odl_l2-sfc scenarios |
-+------------------+-----------------------------------------------+
-| `FUNCTEST-454`_ | Cleanup failures when using HA networks in |
-| | Neutron |
+| | |
+| | |
+------------------+-----------------------------------------------+
+All the tickets that are not blocking have been fixed or postponed
+the next release.
+
+Functest Danube 1.0 is released without known bugs.
+
+
+
Useful links
============
- wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
- - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado
+ - wiki Functest Danube page: https://wiki.opnfv.org/display/functest/Functest+Danube
- Functest repo: https://git.opnfv.org/cgit/functest
- Functest IRC chan: #opnfv-functest
- - Functest reporting: http://testresults.opnfv.org/reporting
-
- - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
-
- - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
-
- - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
-
- - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
-
-.. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
-
-.. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
-
-.. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
-
-.. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
-
-.. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
-
-.. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
-
-.. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
+ - Reporting page: http://testresults.opnfv.org/reporting/danube.html
-.. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
+ - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/functest/ci/testcases.yaml
-.. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423
+.. _`tempest-bug`: https://bugs.launchpad.net/tempest/+bug/1577632
-.. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
+.. _`gluon-bug`: https://bugs.opendaylight.org/show_bug.cgi?id=5586
-.. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
+.. _`apex-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-68
-.. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
+.. _`apex-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-69
-.. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
+.. _`apex-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-70
-.. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
+.. _`apex-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-66
-.. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
+.. _`apex-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-60
-.. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285
+.. _`apex-res-6`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-73
-.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
+.. _`apex-res-7`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-72
-.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
+.. _`apex-res-8`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-69
-.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
+.. _`apex-res-9`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-62
-.. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
+.. _`compass-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-60
-.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
+.. _`compass-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-59
-.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
+.. _`compass-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-baremetal-daily-danube-69
-.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
+.. _`compass-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-57
-.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
+.. _`compass-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-baremetal-daily-danube-67
-.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
+.. _`compass-res-6`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-baremetal-daily-danube-65
-.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
+.. _`fuel-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-54
-.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
+.. _`fuel-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-46
-.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
+.. _`fuel-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-53
-.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
+.. _`fuel-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-44
-.. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
+.. _`fuel-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-55
-.. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
+.. _`fuel-res-6`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-45
-.. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213
+.. _`fuel-res-7`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-zte-pod1-daily-danube-4
-.. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219
+.. _`fuel-res-8`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-48
-.. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376
+.. _`fuel-res-9`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-52
-.. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492
+.. _`fuel-res-10`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-43
-.. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console
+.. _`fuel-res-11`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-50
-.. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console
+.. _`fuel-res-12`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-42
-.. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console
+.. _`fuel-res-13`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-48
-.. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console
+.. _`fuel-res-14`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-50
-.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
+.. _`fuel-res-15`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-51
-.. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
+.. _`fuel-res-16`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-49
-.. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
+.. _`fuel-res-17`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-49
-.. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
+.. _`fuel-res-18`: http://testresults.opnfv.org/test/api/v1/results?build_tag= jenkins-functest-fuel-virtual-daily-danube-51
-.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
+.. _`joid-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-54
-.. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
+.. _`joid-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-55
-.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
+.. _`joid-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-56
-.. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345
+.. _`joid-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-57
+.. _`joid-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-46
Code Review / functest.git / blobdiff