+=======
License
=======
Software
--------
- - The Functest Docker image: * TODO link *
+ - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0)
- - The testapi Docker image: * TODO link *
+ - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0)
Documents
+---------------------+---------+---------+---------+---------+
| nosdn-noha | | | X | X |
+---------------------+---------+---------+---------+---------+
-| odl_l2 | X | X | X | X |
+| kvm | | | X | |
+---------------------+---------+---------+---------+---------+
-| odl_l2-noha | | | X | |
+| kvm-noha | | | X | |
+---------------------+---------+---------+---------+---------+
-| odl_l3 | X | X | X | |
+| lxd | | | | X |
+---------------------+---------+---------+---------+---------+
-| odl_l3-noha | | | X | |
+| lxd-noha | | | | X |
++---------------------+---------+---------+---------+---------+
+| ovs-noha (dpdk) | | | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l2 | X | X | X | X |
++---------------------+---------+---------+---------+---------+
+| odl_l2-noha | | | X | |
+---------------------+---------+---------+---------+---------+
| odl_l2-bgpvpn | X | | X | |
+---------------------+---------+---------+---------+---------+
+---------------------+---------+---------+---------+---------+
| odl_l2-moon | | X | | |
+---------------------+---------+---------+---------+---------+
-| odl_l2-sfc | | | X | |
-+---------------------+---------+---------+---------+---------+
| odl_l2-sfc-noha | X | | X | |
+---------------------+---------+---------+---------+---------+
+| odl_l3 | X | X | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l3-noha | | | X | |
++---------------------+---------+---------+---------+---------+
| onos | | X | X | X |
+---------------------+---------+---------+---------+---------+
| onos-noha | | | X | |
+---------------------+---------+---------+---------+---------+
| onos-sfc-noha | | | X | |
+---------------------+---------+---------+---------+---------+
-| ovs-noha (dpdk) | | | X | |
-+---------------------+---------+---------+---------+---------+
-| kvm | | | X | |
-+---------------------+---------+---------+---------+---------+
-| kvm-noha | | | X | |
-+---------------------+---------+---------+---------+---------+
| multisite | | | X | |
+---------------------+---------+---------+---------+---------+
-| lxd | | | | X |
+
+For Colorado 2.0, the following scenarios have been re-released:
+
+---------------------+---------+---------+---------+---------+
-| lxd-noha | | | | X |
+| Scenario | Apex | Compass | Fuel | Joid |
++=====================+=========+=========+=========+=========+
+| onos | X | X | X | X |
++---------------------+---------+---------+---------+---------+
+| odl_l2-sfc-ha | | | X | |
+---------------------+---------+---------+---------+---------+
+
In Colorado, the functional tests have been sliced in 6 different
categories:
tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
Scoring = 21/21 = 7 * 3
-By default, if not precised, the scenarios are HA.
-HA means OpenStack High Availability (main services). For copper test,
-the OpenStack congress module is not HA. See the release notes of the
-installers for details.
+By default, if not specified, the scenarios are HA.
+HA means OpenStack High Availability (main services). Note that not
+all VIM (e.g. OpenStack) services are deployed in HA mode, as that
+depends upon support of the specific service for HA deployment.
+For example, in the Colorado release, the Congress service
+is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
+support for HA operation has not yet been verified.
+See the release notes of the installers for more details.
-apex
+Apex
----
+------------------+---------+---------+-------------------+
+------------------+---------+---------+-------------------+
| odl_l3 | 15/18 | 83% | `apex-res-176`_ |
+------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn | 14/18 | 78% | ?? |
+| odl_l2-bgpvpn | 14/18 | 78% | `apex-res-235`_ |
+------------------+---------+---------+-------------------+
-| odl_l2-fdio-noha | 11/15 | 73% | ?? |
+| odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ |
+------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha | 18/21 | 86% | ?? |
+| odl_l2-sfc-noha | 18/24 | 75% | `apex-res-217`_ |
+------------------+---------+---------+-------------------+
+| onos-nofeature-ha| 20/21 | 95% | `apex-res-423`_ |
++------------------+---------+---------+-------------------+
+
-compass
+Compass
-------
+------------------+---------+---------+-------------------+
+------------------+---------+---------+-------------------+
| odl_l3 | 9/12 | 75% | `compass-res-73`_ |
+------------------+---------+---------+-------------------+
-| odl_l2-moon | 9/18 | % | ?? |
+| odl_l2-moon | 15/18 | 83% | `compass-res-567`_|
+------------------+---------+---------+-------------------+
-| onos-ha | 15/15 | 100% | `compass-res-77`_ |
+| onos-ha | 15/15 | 100% | `compass-res-285`_|
+------------------+---------+---------+-------------------+
-| onos-sfc-ha | 16/18 | 100% | `compass-res-76`_ |
+| onos-sfc-ha | 17/18 | 95% | `compass-res-76`_ |
+------------------+---------+---------+-------------------+
Note: all the Compass tests for Colorado have been executed on virtual
environment. Bare metal resources were used for Master branch.
-fuel
+Fuel
----
+---------------------+---------+---------+-------------------+
+---------------------+---------+---------+-------------------+
| nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ |
+---------------------+---------+---------+-------------------+
-| nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-sfc | 21/21 | 100% | |
+| nosdn-ovs | 12/18 | 67%* | `fuel-res-213`_ |
+---------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha | 16/18 | 88% | |
+| nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ |
+---------------------+---------+---------+-------------------+
| odl_l2 | 21/21 | 100% | `fuel-res-123`_ |
+---------------------+---------+---------+-------------------+
| odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ |
+---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn | 17/18 | 94% | `fuel-res-119`_ |
+| odl_l2-bgpvpn | 14/18 | 78% | `fuel-res-119`_ |
++---------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn-noha | 14/15 | 93% | `fuel-res-160`_ |
+---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn-noha | 15/15 | 100% | `fuel-res-160`_ |
+| odl_l2-sfc-noha | 6/21 | 29% | `fuel-res-219`_ |
+---------------------+---------+---------+-------------------+
-| odl_l3 | 15/18 | 67% | `fuel-res-115`_ |
+| odl_l2-sfc-ha | 16/21 | 76% | `fuel-res-376`_ |
++---------------------+---------+---------+-------------------+
+| odl_l3 | 15/18 | 83% | `fuel-res-115`_ |
+---------------------+---------+---------+-------------------+
| odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ |
+---------------------+---------+---------+-------------------+
-| onos | 20/21 | 95% | `fuel-res-117`_ |
+| onos | 20/21 | 95% | `fuel-res-492`_ |
+---------------------+---------+---------+-------------------+
| onos-noha | 18/18 | 100% | `fuel-res-166`_ |
+---------------------+---------+---------+-------------------+
-| onos-sfc | 24/24 | 100% | fuel-res-124`_ |
+| onos-sfc | 24/24 | 100% | `fuel-res-124`_ |
+---------------------+---------+---------+-------------------+
| onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ |
+---------------------+---------+---------+-------------------+
| multisite | N.R | 100% | `fuel-res-8`_ |
+---------------------+---------+---------+-------------------+
-joid
+*: all results passed, lacking iterations to reach the full score
+
+Results of Functest on AArch64 Colorado 3.0
+
++---------------------+---------+---------+----------------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++=====================+=========+=========+======================+
+| nosdn | 18/18 | 100% | `fuel-arm-res-128`_ |
++---------------------+---------+---------+----------------------+
+| odl_l2 | 21/21 | 100% | `fuel-arm-res-122`_ |
++---------------------+---------+---------+----------------------+
+| odl_l2-noha | 18/18 | 100% | `fuel-arm-res-129`_ |
++---------------------+---------+---------+----------------------+
+| odl_l3 | 17/18 | 95% | `fuel-arm-res-135`_ |
++---------------------+---------+---------+----------------------+
+
+
+Joid
----
+---------------------+---------+---------+-----------------+
+---------------------+---------+---------+-----------------+
| nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ |
+---------------------+---------+---------+-----------------+
-| odl_l2 | 19/21 | 91% | `joid-res-103`_ |
+| odl_l2 | 21/21 | 100% | `joid-res-103`_ |
+---------------------+---------+---------+-----------------+
-| onos | 21/21 | 100% | `joid-res-99`_ |
+| onos | 21/21 | 100% | `joid-res-345`_ |
+---------------------+---------+---------+-----------------+
| onos-sfc | 24/24 | 100% | `joid-res-97`_ |
+---------------------+---------+---------+-----------------+
+------------------+-----------------------------------------------+
| JIRA | Description |
+==================+===============================================+
-| `FUNCTEST-419`_ | do not try to Remove docker |
-| | image opnfv/functest:<none> |
-| | reported by joid on Intel POD |
-| | may impact CI |
-| | not reproducible |
-+------------------+-----------------------------------------------+
| `FUNCTEST-446`_ | Cleanup ODL-SFC output in Functest execution |
| | Impact on odl_l2-sfc scenarios |
+------------------+-----------------------------------------------+
-| `FUNCTEST-450`_ | Functest is Failing to get the token using |
-| | keystone client |
-+------------------+-----------------------------------------------+
| `FUNCTEST-454`_ | Cleanup failures when using HA networks in |
| | Neutron |
+------------------+-----------------------------------------------+
-| `FUNCTEST-460`_ | Wrong image format used in rally cases |
-+------------------+-----------------------------------------------+
-| `FUNCTEST-462`_ | OLD test fails after forcing the clone |
-| | release/beryllium-sr3 branch |
-+------------------+-----------------------------------------------+
-
Useful links
============
- Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
-.. _`FUNCTEST-419` : https://jira.opnfv.org/browse/FUNCTEST-419
-
.. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
-.. _`FUNCTEST-450` : https://jira.opnfv.org/browse/FUNCTEST-450
-
.. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
-.. _`FUNCTEST-460` : https://jira.opnfv.org/browse/FUNCTEST-460
-
-.. _`FUNCTEST-462` : https://jira.opnfv.org/browse/FUNCTEST-462
+.. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
.. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
.. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
+.. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
+
+.. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
+
+.. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423
+
.. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
.. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
.. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
-.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
+.. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
-.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
+.. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285
-.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
+.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
-.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
+.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
-.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
+.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
.. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
-.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
+.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
-.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
+.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
-.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
+.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
-.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
+.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
-.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
+.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
+
+.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
+
+.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
+
+.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
+
+.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
.. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
.. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
-.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
+.. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213
-.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
+.. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219
-.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
+.. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376
-.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
+.. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492
-.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
+.. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console
+
+.. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console
+
+.. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console
+
+.. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console
+
+.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
.. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
.. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
+.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
+
.. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
+
+.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
+
+.. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345
+