-===========================================
-OPNFV Colorado1.0 release note for Functest
-===========================================
-
-Abstract
-========
-
-This document describes the release note of Functest project.
-
License
=======
You should have received a copy of the license along with this.
If not, see <http://creativecommons.org/licenses/by/4.0/>.
+===========================================
+OPNFV Colorado1.0 release note for Functest
+===========================================
+
+Abstract
+========
+
+This document describes the release note of Functest project.
+
+
Version history
===============
| **Date** | **Ver.** | **Author** | **Comment** |
| | | | |
+------------+----------+------------------+------------------------+
-| 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for C release |
-| | | (Orange) | |
+| 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado |
+| | | (Orange) | release |
+------------+----------+------------------+------------------------+
OPNFV Colorado Release
Documents
---------
- - Installation/configuration guide: * TODO link *
+ - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
- - User Guide: * TODO link *
+ - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
- - Developer Guide: * TODO link *
+ - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
Version change
- creation of the healthcheck test case
- - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, multisite)
+ - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
- integration of new OPNFV feature projects (copper, domino, multisite,
moon, parser, onos-sfc, odl-sfc, security scan)
===============
For Colorado 1.0, Functest was tested on the following scenarios (if not
-precised, the scenario is a na scenario):
+precised, the scenario is a HA scenario):
+---------------------+---------+---------+---------+---------+
| Scenario | Apex | Compass | Fuel | Joid |
+=====================+=========+=========+=========+=========+
| nosdn | X | X | X | X |
+---------------------+---------+---------+---------+---------+
+| nosdn-noha | | | X | X |
++---------------------+---------+---------+---------+---------+
| odl_l2 | X | X | X | X |
+---------------------+---------+---------+---------+---------+
+| odl_l2-noha | | | X | |
++---------------------+---------+---------+---------+---------+
| odl_l3 | X | X | X | |
+---------------------+---------+---------+---------+---------+
-| onos | X | X | X | X |
+| odl_l3-noha | | | X | |
+---------------------+---------+---------+---------+---------+
-| ocl | | X | | |
+| odl_l2-bgpvpn | X | | X | |
+---------------------+---------+---------+---------+---------+
-| ovs-noha (dpdk) | | | X | |
+| odl_l2-bgpvpn-noha| | | X | |
+---------------------+---------+---------+---------+---------+
-| kvm-noha | | | X | |
+| odl_l2-fdio-noha | X | | | |
+---------------------+---------+---------+---------+---------+
-| odl_l2-bgpvpn | X | | X | |
+| odl_l2-moon | | X | | |
+---------------------+---------+---------+---------+---------+
| odl_l2-sfc | | | X | |
+---------------------+---------+---------+---------+---------+
-| onos-sfc | X | X | X | X |
+| odl_l2-sfc-noha | X | | X | |
+---------------------+---------+---------+---------+---------+
-| odl_l2-moon | | X | | |
+| onos | | X | X | X |
++---------------------+---------+---------+---------+---------+
+| onos-noha | | | X | |
++---------------------+---------+---------+---------+---------+
+| onos-sfc | | X | X | X |
+---------------------+---------+---------+---------+---------+
-| multisite | | | | |
+| onos-sfc-noha | | | X | |
++---------------------+---------+---------+---------+---------+
+| ovs-noha (dpdk) | | | X | |
++---------------------+---------+---------+---------+---------+
+| kvm | | | X | |
++---------------------+---------+---------+---------+---------+
+| kvm-noha | | | X | |
++---------------------+---------+---------+---------+---------+
+| multisite | | | X | |
+---------------------+---------+---------+---------+---------+
| lxd | | | | X |
+---------------------+---------+---------+---------+---------+
+| lxd-noha | | | | X |
++---------------------+---------+---------+---------+---------+
-Functest defines a scenario scoring based on the sum of the unitary test
-cases run in CI.
-The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
-
-In Colorado, the functional tests have been sliced in different
+In Colorado, the functional tests have been sliced in 6 different
categories:
+----------------+-----------------------------------------------+
| | VNF deployment and tests (e.g. vIMS) |
+----------------+-----------------------------------------------+
-For the scenario validation, we consider only the categories healthcheck,
-smoke, sdn_suites and features. These tests are run systematically in
-the CI daily loops.
+For the scenario validation, we consider only the healthcheck, smoke,
+sdn_suites and features categories. These tests are run systematically
+in the CI daily loops.
Success criteria have been defined for these test cases, they can be
-PASS/FAIl or a success rate may be declared (100%, > 90%)
+PASS/FAIL or a success rate may be declared (100%, > 90%)
All the criteria, as well as the test dependencies are declared in the
ci/testcases.yaml file.
-* TODO scoring table *
+The scoring for the Colorado release per installer can be described as
+follows.
+
+The scoring is an indicator showing how many feature project test suites
+have been integrated on the scenario.
+
+The scoring equals the number of tests * succesful iteration of each
+test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
+
+ e.g.
+ apex/odl_l2-nofeature-ha
+ tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
+ Scoring = 21/21 = 7 * 3
+
+By default, if not precised, the scenarios are HA.
+HA means OpenStack High Availability (main services). For copper test,
+the OpenStack congress module is not HA. See the release notes of the
+installers for details.
+
+
+apex
+----
+
++------------------+---------+---------+-------------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++==================+=========+=========+===================+
+| nosdn | 17/18 | 95% | `apex-res-174`_ |
++------------------+---------+---------+-------------------+
+| odl_l2 | 21/21 | 100% | `apex-res-175`_ |
++------------------+---------+---------+-------------------+
+| odl_l3 | 15/18 | 83% | `apex-res-176`_ |
++------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn | 14/18 | 78% | ?? |
++------------------+---------+---------+-------------------+
+| odl_l2-fdio-noha | 11/15 | 73% | ?? |
++------------------+---------+---------+-------------------+
+| odl_l2-sfc-noha | 18/21 | 86% | ?? |
++------------------+---------+---------+-------------------+
+
+compass
+-------
+
++------------------+---------+---------+-------------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++==================+=========+=========+===================+
+| nosdn | 12/12 | 100% | `compass-res-55`_ |
++------------------+---------+---------+-------------------+
+| odl_l2 | 15/15 | 100% | `compass-res-59`_ |
++------------------+---------+---------+-------------------+
+| odl_l3 | 9/12 | 75% | `compass-res-73`_ |
++------------------+---------+---------+-------------------+
+| odl_l2-moon | 9/18 | % | ?? |
++------------------+---------+---------+-------------------+
+| onos-ha | 15/15 | 100% | `compass-res-77`_ |
++------------------+---------+---------+-------------------+
+| onos-sfc-ha | 16/18 | 100% | `compass-res-76`_ |
++------------------+---------+---------+-------------------+
+
+Note: all the Compass tests for Colorado have been executed on virtual
+environment. Bare metal resources were used for Master branch.
+
+
+fuel
+----
+
++---------------------+---------+---------+-------------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++=====================+=========+=========+===================+
+| nosdn | 18/18 | 100% | `fuel-res-129`_ |
++---------------------+---------+---------+-------------------+
+| nosdn-noha | 15/15 | 100% | `fuel-res-154`_ |
++---------------------+---------+---------+-------------------+
+| nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ |
++---------------------+---------+---------+-------------------+
+| nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ |
++---------------------+---------+---------+-------------------+
+| nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ |
++---------------------+---------+---------+-------------------+
+| odl_l2-sfc | 21/21 | 100% | |
++---------------------+---------+---------+-------------------+
+| odl_l2-sfc-noha | 16/18 | 88% | |
++---------------------+---------+---------+-------------------+
+| odl_l2 | 21/21 | 100% | `fuel-res-123`_ |
++---------------------+---------+---------+-------------------+
+| odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ |
++---------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn | 17/18 | 94% | `fuel-res-119`_ |
++---------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn-noha | 15/15 | 100% | `fuel-res-160`_ |
++---------------------+---------+---------+-------------------+
+| odl_l3 | 15/18 | 67% | `fuel-res-115`_ |
++---------------------+---------+---------+-------------------+
+| odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ |
++---------------------+---------+---------+-------------------+
+| onos | 20/21 | 95% | `fuel-res-117`_ |
++---------------------+---------+---------+-------------------+
+| onos-noha | 18/18 | 100% | `fuel-res-166`_ |
++---------------------+---------+---------+-------------------+
+| onos-sfc | 24/24 | 100% | fuel-res-124`_ |
++---------------------+---------+---------+-------------------+
+| onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ |
++---------------------+---------+---------+-------------------+
+| multisite | N.R | 100% | `fuel-res-8`_ |
++---------------------+---------+---------+-------------------+
+
+joid
+----
+
++---------------------+---------+---------+-----------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++=====================+=========+=========+=================+
+| nosdn | 18/18 | 100% | `joid-res-102`_ |
++---------------------+---------+---------+-----------------+
+| nosdn-noha | 17/18 | 95% | `joid-res-93`_ |
++---------------------+---------+---------+-----------------+
+| nosdn-lxd | 12/12 | 100% | `joid-res-104`_ |
++---------------------+---------+---------+-----------------+
+| nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ |
++---------------------+---------+---------+-----------------+
+| odl_l2 | 19/21 | 91% | `joid-res-103`_ |
++---------------------+---------+---------+-----------------+
+| onos | 21/21 | 100% | `joid-res-99`_ |
++---------------------+---------+---------+-----------------+
+| onos-sfc | 24/24 | 100% | `joid-res-97`_ |
++---------------------+---------+---------+-----------------+
+
+It is highly recommended to install a json viewer in your browser
+(e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
+
+You can get additional details through test logs on http://artifacts.opnfv.org/.
+As no search engine is available on the OPNFV artifact web site you must
+retrieve the pod identifier on which the tests have been executed (see
+field pod in any of the results) then click on the selected POD and look
+for the date of the test you are interested in.
The reporting pages can be found at:
+-----------+-----------+----------------------------------------------+
| Installer | Scenario | Issue |
+===========+===========+==============================================+
-| any | onos-* | vPing userdata and Tempest cases related to |
-| | | metadata service excluded from onos scenarios|
-| | | https://gerrit.opnfv.org/gerrit/#/c/18729/ |
-+-----------+-----------+----------------------------------------------+
| any | odl_l3-* | Tempest cases related to using floating IP |
| | | addresses fail because of a known ODL bug. |
| | | vPing_ssh test case is excluded for the same |
| | | Rally sanity in Colorado 2.0 with the |
| | | adoption of ODL Boron release. |
+-----------+-----------+----------------------------------------------+
+| apex | *-fdio | Due to late integration, fdio decided to |
+| | | focus on mandatory tests and exclude feature |
+| | | tests (copper, doctor, security_scan) from |
+| | | its scenarios |
++-----------+-----------+----------------------------------------------+
| compass | moon | First ODL test FAILS because ODL/Openstack |
| | | federation done in moon is partial. Only |
| | | MD-SAL is federated (not AD-SAL) |
+------------------+-----------------------------------------------+
| JIRA | Description |
+==================+===============================================+
+| `FUNCTEST-419`_ | do not try to Remove docker |
+| | image opnfv/functest:<none> |
+| | reported by joid on Intel POD |
+| | may impact CI |
+| | not reproducible |
++------------------+-----------------------------------------------+
+| `FUNCTEST-446`_ | Cleanup ODL-SFC output in Functest execution |
+| | Impact on odl_l2-sfc scenarios |
+------------------+-----------------------------------------------+
+| `FUNCTEST-450`_ | Functest is Failing to get the token using |
+| | keystone client |
+------------------+-----------------------------------------------+
+| `FUNCTEST-454`_ | Cleanup failures when using HA networks in |
+| | Neutron |
+------------------+-----------------------------------------------+
+| `FUNCTEST-460`_ | Wrong image format used in rally cases |
+------------------+-----------------------------------------------+
+| `FUNCTEST-462`_ | OLD test fails after forcing the clone |
+| | release/beryllium-sr3 branch |
+------------------+-----------------------------------------------+
+
Useful links
============
- Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
- - Functest Colorado user guide: * TODO *
+ - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
+
+ - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
+
+ - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
+
+.. _`FUNCTEST-419` : https://jira.opnfv.org/browse/FUNCTEST-419
+
+.. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
+
+.. _`FUNCTEST-450` : https://jira.opnfv.org/browse/FUNCTEST-450
+
+.. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
+
+.. _`FUNCTEST-460` : https://jira.opnfv.org/browse/FUNCTEST-460
+
+.. _`FUNCTEST-462` : https://jira.opnfv.org/browse/FUNCTEST-462
+
+.. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
+
+.. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
+
+.. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
+
+.. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
+
+.. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
+
+.. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
+
+.. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
+
+.. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
+
+.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
+
+.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
+
+.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
+
+.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
+
+.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
+
+.. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
+
+.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
+
+.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
+
+.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
+
+.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
+
+.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
+
+.. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
+
+.. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
+
+.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
+
+.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
+
+.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
+
+.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
+
+.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
- - Functest installation/configuration guide: * TODO *
+.. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
- - Functest developer guide: * TODO *
+.. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
+.. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
+.. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
Code Review / functest.git / commitdiff