Code Review / functest.git / blobdiff
? search:
summary | shortlog | log | commit | commitdiff | review | tree
raw | inline | side by side
Merge changes from topic 'convert_unix_format'
[functest.git] / docs / release-notes / functest-release.rst
diff --git a/docs/release-notes/functest-release.rst b/docs/release-notes/functest-release.rst
index 2d87253..07389bb 100644 (file)
--- a/docs/release-notes/functest-release.rst
+++ b/docs/release-notes/functest-release.rst
@@ -1,3 +1,4 @@
+=======
 License
 =======
 
@@ -96,9 +97,9 @@ Deliverables
 Software
 --------
 
- - The Functest Docker image: * TODO link *
+ - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0)
 
- - The testapi Docker image: * TODO link *
+ - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0)
 
 
 Documents
@@ -166,13 +167,19 @@ precised, the scenario is a HA scenario):
 +---------------------+---------+---------+---------+---------+
 |   nosdn-noha        |         |         |    X    |    X    |
 +---------------------+---------+---------+---------+---------+
-|   odl_l2            |    X    |    X    |    X    |    X    |
+|   kvm               |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   odl_l2-noha       |         |         |    X    |         |
+|   kvm-noha          |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   odl_l3            |    X    |    X    |    X    |         |
+|   lxd               |         |         |         |    X    |
 +---------------------+---------+---------+---------+---------+
-|   odl_l3-noha       |         |         |    X    |         |
+|   lxd-noha          |         |         |         |    X    |
++---------------------+---------+---------+---------+---------+
+|   ovs-noha (dpdk)   |         |         |    X    |         |
++---------------------+---------+---------+---------+---------+
+|   odl_l2            |    X    |    X    |    X    |    X    |
++---------------------+---------+---------+---------+---------+
+|   odl_l2-noha       |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
 |   odl_l2-bgpvpn     |    X    |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
@@ -182,10 +189,12 @@ precised, the scenario is a HA scenario):
 +---------------------+---------+---------+---------+---------+
 |   odl_l2-moon       |         |    X    |         |         |
 +---------------------+---------+---------+---------+---------+
-|   odl_l2-sfc        |         |         |    X    |         |
-+---------------------+---------+---------+---------+---------+
 |   odl_l2-sfc-noha   |    X    |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
+|   odl_l3            |    X    |    X    |    X    |         |
++---------------------+---------+---------+---------+---------+
+|   odl_l3-noha       |         |         |    X    |         |
++---------------------+---------+---------+---------+---------+
 |   onos              |         |    X    |    X    |    X    |
 +---------------------+---------+---------+---------+---------+
 |   onos-noha         |         |         |    X    |         |
@@ -194,19 +203,20 @@ precised, the scenario is a HA scenario):
 +---------------------+---------+---------+---------+---------+
 |   onos-sfc-noha     |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   ovs-noha (dpdk)   |         |         |    X    |         |
-+---------------------+---------+---------+---------+---------+
-|   kvm               |         |         |    X    |         |
-+---------------------+---------+---------+---------+---------+
-|   kvm-noha          |         |         |    X    |         |
-+---------------------+---------+---------+---------+---------+
 |   multisite         |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   lxd               |         |         |         |    X    |
+
+For Colorado 2.0, the following scenarios have been re-released:
+
 +---------------------+---------+---------+---------+---------+
-|   lxd-noha          |         |         |         |    X    |
+|    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
++=====================+=========+=========+=========+=========+
+|   onos              |    X    |    X    |    X    |    X    |
++---------------------+---------+---------+---------+---------+
+|   odl_l2-sfc-ha     |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
 
+
 In Colorado, the functional tests have been sliced in 6 different
 categories:
 
@@ -252,13 +262,17 @@ test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/view
  tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
  Scoring = 21/21 = 7 * 3
 
-By default, if not precised, the scenarios are HA.
-HA means OpenStack High Availability (main services). For copper test,
-the OpenStack congress module is not HA. See the release notes of the
-installers for details.
+By default, if not specified, the scenarios are HA.
+HA means OpenStack High Availability (main services). Note that not
+all VIM (e.g. OpenStack) services are deployed in HA mode, as that
+depends upon support of the specific service for HA deployment.
+For example, in the Colorado release, the Congress service
+is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
+support for HA operation has not yet been verified.
+See the release notes of the installers for more details.
 
 
-apex
+Apex
 ----
 
 +------------------+---------+---------+-------------------+
@@ -271,14 +285,17 @@ apex
 +------------------+---------+---------+-------------------+
 | odl_l3           |  15/18  |    83%  | `apex-res-176`_   |
 +------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn    |  14/18  |    78%  |     ??            |
+| odl_l2-bgpvpn    |  14/18  |    78%  | `apex-res-235`_   |
 +------------------+---------+---------+-------------------+
-| odl_l2-fdio-noha |  11/15  |    73%  |     ??            |
+| odl_l2-fdio-noha |  12/15  |    80%  | `apex-res-6`_     |
 +------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha  |  18/21  |    86%  |     ??            |
+| odl_l2-sfc-noha  |  18/24  |    75%  | `apex-res-217`_   |
 +------------------+---------+---------+-------------------+
+| onos-nofeature-ha|  20/21  |    95%  | `apex-res-423`_   |
++------------------+---------+---------+-------------------+
+
 
-compass
+Compass
 -------
 
 +------------------+---------+---------+-------------------+
@@ -291,18 +308,18 @@ compass
 +------------------+---------+---------+-------------------+
 | odl_l3           |  9/12   |    75%  | `compass-res-73`_ |
 +------------------+---------+---------+-------------------+
-| odl_l2-moon      |  9/18   |      %  |    ??             |
+| odl_l2-moon      |  15/18  |    83%  | `compass-res-567`_|
 +------------------+---------+---------+-------------------+
-| onos-ha          |  15/15  |   100%  | `compass-res-77`_ |
+| onos-ha          |  15/15  |   100%  | `compass-res-285`_|
 +------------------+---------+---------+-------------------+
-| onos-sfc-ha      |  16/18  |   100%  | `compass-res-76`_ |
+| onos-sfc-ha      |  17/18  |    95%  | `compass-res-76`_ |
 +------------------+---------+---------+-------------------+
 
 Note: all the Compass tests for Colorado have been executed on virtual
 environment. Bare metal resources were used for Master branch.
 
 
-fuel
+Fuel
 ----
 
 +---------------------+---------+---------+-------------------+
@@ -317,36 +334,56 @@ fuel
 +---------------------+---------+---------+-------------------+
 | nosdn-kvm-noha      |  15/15  |  100%   | `fuel-res-161`_   |
 +---------------------+---------+---------+-------------------+
-| nosdn-ovs-noha      |  15/15  |  100%   | `fuel-res-162`_   |
-+---------------------+---------+---------+-------------------+
-| odl_l2-sfc          |  21/21  |  100%   |                   |
+| nosdn-ovs           |  12/18  |   67%*  |  `fuel-res-213`_  |
 +---------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha     |  16/18  |   88%   |                   |
+| nosdn-ovs-noha      |  15/15  |  100%   | `fuel-res-162`_   |
 +---------------------+---------+---------+-------------------+
 | odl_l2              |  21/21  |  100%   |  `fuel-res-123`_  |
 +---------------------+---------+---------+-------------------+
 | odl_l2-noha         |  17/18  |   94%   | `fuel-res-155`_   |
 +---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn       |  17/18  |   94%   | `fuel-res-119`_   |
+| odl_l2-bgpvpn       |  14/18  |   78%   | `fuel-res-119`_   |
++---------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn-noha  |  14/15  |   93%   | `fuel-res-160`_   |
 +---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn-noha  |  15/15  |  100%   | `fuel-res-160`_   |
+| odl_l2-sfc-noha     |   6/21  |   29%   | `fuel-res-219`_   |
 +---------------------+---------+---------+-------------------+
-| odl_l3              |  15/18  |   67%   | `fuel-res-115`_   |
+| odl_l2-sfc-ha       |  16/21  |   76%   | `fuel-res-376`_   |
++---------------------+---------+---------+-------------------+
+| odl_l3              |  15/18  |   83%   | `fuel-res-115`_   |
 +---------------------+---------+---------+-------------------+
 | odl_l3-noha         |  12/15  |   80%   | `fuel-res-164`_   |
 +---------------------+---------+---------+-------------------+
-| onos                |  20/21  |   95%   | `fuel-res-117`_   |
+| onos                |  20/21  |   95%   | `fuel-res-492`_   |
 +---------------------+---------+---------+-------------------+
 | onos-noha           |  18/18  |  100%   | `fuel-res-166`_   |
 +---------------------+---------+---------+-------------------+
-| onos-sfc            |  24/24  |  100%   | fuel-res-124`_    |
+| onos-sfc            |  24/24  |  100%   | `fuel-res-124`_   |
 +---------------------+---------+---------+-------------------+
 | onos-sfc-noha       |  21/21  |  100%   | `fuel-res-129`_   |
 +---------------------+---------+---------+-------------------+
 | multisite           |  N.R    |  100%   | `fuel-res-8`_     |
 +---------------------+---------+---------+-------------------+
 
-joid
+*: all results passed, lacking iterations to reach the full score
+
+Results of Functest on AArch64 Colorado 3.0
+
++---------------------+---------+---------+----------------------+
+|  Scenario           | Scoring | Success |  Results             |
+|                     |         | rate    |                      |
++=====================+=========+=========+======================+
+| nosdn               |  18/18  |  100%   | `fuel-arm-res-128`_  |
++---------------------+---------+---------+----------------------+
+| odl_l2              |  21/21  |  100%   | `fuel-arm-res-122`_  |
++---------------------+---------+---------+----------------------+
+| odl_l2-noha         |  18/18  |  100%   | `fuel-arm-res-129`_  |
++---------------------+---------+---------+----------------------+
+| odl_l3              |  17/18  |   95%   | `fuel-arm-res-135`_  |
++---------------------+---------+---------+----------------------+
+
+
+Joid
 ----
 
 +---------------------+---------+---------+-----------------+
@@ -361,9 +398,9 @@ joid
 +---------------------+---------+---------+-----------------+
 | nosdn-lxd-noha      |  12/12  |  100%   | `joid-res-91`_  |
 +---------------------+---------+---------+-----------------+
-| odl_l2              |  19/21  |   91%   | `joid-res-103`_ |
+| odl_l2              |  21/21  |  100%   | `joid-res-103`_ |
 +---------------------+---------+---------+-----------------+
-| onos                |  21/21  |  100%   | `joid-res-99`_  |
+| onos                |  21/21  |  100%   | `joid-res-345`_ |
 +---------------------+---------+---------+-----------------+
 | onos-sfc            |  24/24  |  100%   | `joid-res-97`_  |
 +---------------------+---------+---------+-----------------+
@@ -472,27 +509,12 @@ Open JIRA tickets
 +------------------+-----------------------------------------------+
 |   JIRA           |         Description                           |
 +==================+===============================================+
-| `FUNCTEST-419`_  |  do not try to Remove docker                  |
-|                  |  image opnfv/functest:<none>                  |
-|                  |  reported by joid on Intel POD                |
-|                  |  may impact CI                                |
-|                  |  not reproducible                             |
-+------------------+-----------------------------------------------+
 | `FUNCTEST-446`_  |  Cleanup ODL-SFC output in Functest execution |
 |                  |  Impact on odl_l2-sfc scenarios               |
 +------------------+-----------------------------------------------+
-| `FUNCTEST-450`_  |  Functest is Failing to get the token using   |
-|                  |  keystone client                              |
-+------------------+-----------------------------------------------+
 | `FUNCTEST-454`_  |  Cleanup failures when using HA networks in   |
 |                  |  Neutron                                      |
 +------------------+-----------------------------------------------+
-| `FUNCTEST-460`_  |  Wrong image format used in rally cases       |
-+------------------+-----------------------------------------------+
-| `FUNCTEST-462`_  |  OLD test fails after forcing the clone       |
-|                  |  release/beryllium-sr3 branch                 |
-+------------------+-----------------------------------------------+
-
 
 Useful links
 ============
@@ -519,17 +541,11 @@ Useful links
 
  - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 
-.. _`FUNCTEST-419` : https://jira.opnfv.org/browse/FUNCTEST-419
-
 .. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
 
-.. _`FUNCTEST-450` : https://jira.opnfv.org/browse/FUNCTEST-450
-
 .. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
 
-.. _`FUNCTEST-460` : https://jira.opnfv.org/browse/FUNCTEST-460
-
-.. _`FUNCTEST-462` : https://jira.opnfv.org/browse/FUNCTEST-462
+.. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
 
 .. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
 
@@ -537,6 +553,12 @@ Useful links
 
 .. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
 
+.. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
+
+.. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
+
+.. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423
+
 .. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
 
 .. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
@@ -547,41 +569,57 @@ Useful links
 
 .. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
 
-.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
+.. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
 
-.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
+.. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285
 
-.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
+.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
 
-.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
+.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
 
-.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
+.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
 
 .. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
 
-.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
+.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
 
-.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
+.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
 
-.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
+.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
 
-.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
+.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
 
-.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
+.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
+
+.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
+
+.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
+
+.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
+
+.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
 
 .. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
 
 .. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
 
-.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
+.. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213
 
-.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
+.. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219
 
-.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
+.. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376
 
-.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
+.. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492
 
-.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
+.. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console
+
+.. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console
+
+.. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console
+
+.. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console
+
+.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
 
 .. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
 
@@ -589,4 +627,11 @@ Useful links
 
 .. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
 
+.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
+
 .. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
+
+.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
+
+.. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345
+
Test suites and cases to verify OPNFV Platform functionality