Code Review / functest.git / blobdiff
? search:
summary | shortlog | log | commit | commitdiff | review | tree
raw | inline | side by side
Logic to trigger Security Scan project
[functest.git] / docs / release-notes / functest-release.rst
diff --git a/docs/release-notes/functest-release.rst b/docs/release-notes/functest-release.rst
index 034c27b..07389bb 100644 (file)
--- a/docs/release-notes/functest-release.rst
+++ b/docs/release-notes/functest-release.rst
@@ -97,9 +97,9 @@ Deliverables
 Software
 --------
 
- - The Functest Docker image: * TODO link *
+ - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0)
 
- - The testapi Docker image: * TODO link *
+ - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0)
 
 
 Documents
@@ -167,13 +167,19 @@ precised, the scenario is a HA scenario):
 +---------------------+---------+---------+---------+---------+
 |   nosdn-noha        |         |         |    X    |    X    |
 +---------------------+---------+---------+---------+---------+
-|   odl_l2            |    X    |    X    |    X    |    X    |
+|   kvm               |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   odl_l2-noha       |         |         |    X    |         |
+|   kvm-noha          |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   odl_l3            |    X    |    X    |    X    |         |
+|   lxd               |         |         |         |    X    |
 +---------------------+---------+---------+---------+---------+
-|   odl_l3-noha       |         |         |    X    |         |
+|   lxd-noha          |         |         |         |    X    |
++---------------------+---------+---------+---------+---------+
+|   ovs-noha (dpdk)   |         |         |    X    |         |
++---------------------+---------+---------+---------+---------+
+|   odl_l2            |    X    |    X    |    X    |    X    |
++---------------------+---------+---------+---------+---------+
+|   odl_l2-noha       |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
 |   odl_l2-bgpvpn     |    X    |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
@@ -183,10 +189,12 @@ precised, the scenario is a HA scenario):
 +---------------------+---------+---------+---------+---------+
 |   odl_l2-moon       |         |    X    |         |         |
 +---------------------+---------+---------+---------+---------+
-|   odl_l2-sfc        |         |         |    X    |         |
-+---------------------+---------+---------+---------+---------+
 |   odl_l2-sfc-noha   |    X    |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
+|   odl_l3            |    X    |    X    |    X    |         |
++---------------------+---------+---------+---------+---------+
+|   odl_l3-noha       |         |         |    X    |         |
++---------------------+---------+---------+---------+---------+
 |   onos              |         |    X    |    X    |    X    |
 +---------------------+---------+---------+---------+---------+
 |   onos-noha         |         |         |    X    |         |
@@ -195,19 +203,20 @@ precised, the scenario is a HA scenario):
 +---------------------+---------+---------+---------+---------+
 |   onos-sfc-noha     |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   ovs-noha (dpdk)   |         |         |    X    |         |
-+---------------------+---------+---------+---------+---------+
-|   kvm               |         |         |    X    |         |
-+---------------------+---------+---------+---------+---------+
-|   kvm-noha          |         |         |    X    |         |
-+---------------------+---------+---------+---------+---------+
 |   multisite         |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   lxd               |         |         |         |    X    |
+
+For Colorado 2.0, the following scenarios have been re-released:
+
 +---------------------+---------+---------+---------+---------+
-|   lxd-noha          |         |         |         |    X    |
+|    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
++=====================+=========+=========+=========+=========+
+|   onos              |    X    |    X    |    X    |    X    |
++---------------------+---------+---------+---------+---------+
+|   odl_l2-sfc-ha     |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
 
+
 In Colorado, the functional tests have been sliced in 6 different
 categories:
 
@@ -276,13 +285,16 @@ Apex
 +------------------+---------+---------+-------------------+
 | odl_l3           |  15/18  |    83%  | `apex-res-176`_   |
 +------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn    |  14/18  |    78%  |     ??            |
+| odl_l2-bgpvpn    |  14/18  |    78%  | `apex-res-235`_   |
 +------------------+---------+---------+-------------------+
 | odl_l2-fdio-noha |  12/15  |    80%  | `apex-res-6`_     |
 +------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha  |  18/21  |    86%  |     ??            |
+| odl_l2-sfc-noha  |  18/24  |    75%  | `apex-res-217`_   |
++------------------+---------+---------+-------------------+
+| onos-nofeature-ha|  20/21  |    95%  | `apex-res-423`_   |
 +------------------+---------+---------+-------------------+
 
+
 Compass
 -------
 
@@ -296,11 +308,11 @@ Compass
 +------------------+---------+---------+-------------------+
 | odl_l3           |  9/12   |    75%  | `compass-res-73`_ |
 +------------------+---------+---------+-------------------+
-| odl_l2-moon      |  9/18   |      %  |    ??             |
+| odl_l2-moon      |  15/18  |    83%  | `compass-res-567`_|
 +------------------+---------+---------+-------------------+
-| onos-ha          |  15/15  |   100%  | `compass-res-77`_ |
+| onos-ha          |  15/15  |   100%  | `compass-res-285`_|
 +------------------+---------+---------+-------------------+
-| onos-sfc-ha      |  16/18  |   100%  | `compass-res-76`_ |
+| onos-sfc-ha      |  17/18  |    95%  | `compass-res-76`_ |
 +------------------+---------+---------+-------------------+
 
 Note: all the Compass tests for Colorado have been executed on virtual
@@ -322,49 +334,52 @@ Fuel
 +---------------------+---------+---------+-------------------+
 | nosdn-kvm-noha      |  15/15  |  100%   | `fuel-res-161`_   |
 +---------------------+---------+---------+-------------------+
-| nosdn-ovs-noha      |  15/15  |  100%   | `fuel-res-162`_   |
-+---------------------+---------+---------+-------------------+
-| odl_l2-sfc          |  21/21  |  100%   |                   |
+| nosdn-ovs           |  12/18  |   67%*  |  `fuel-res-213`_  |
 +---------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha     |  16/18  |   88%   |                   |
+| nosdn-ovs-noha      |  15/15  |  100%   | `fuel-res-162`_   |
 +---------------------+---------+---------+-------------------+
 | odl_l2              |  21/21  |  100%   |  `fuel-res-123`_  |
 +---------------------+---------+---------+-------------------+
 | odl_l2-noha         |  17/18  |   94%   | `fuel-res-155`_   |
 +---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn       |  17/18  |   94%   | `fuel-res-119`_   |
+| odl_l2-bgpvpn       |  14/18  |   78%   | `fuel-res-119`_   |
++---------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn-noha  |  14/15  |   93%   | `fuel-res-160`_   |
++---------------------+---------+---------+-------------------+
+| odl_l2-sfc-noha     |   6/21  |   29%   | `fuel-res-219`_   |
 +---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn-noha  |  15/15  |  100%   | `fuel-res-160`_   |
+| odl_l2-sfc-ha       |  16/21  |   76%   | `fuel-res-376`_   |
 +---------------------+---------+---------+-------------------+
-| odl_l3              |  15/18  |   67%   | `fuel-res-115`_   |
+| odl_l3              |  15/18  |   83%   | `fuel-res-115`_   |
 +---------------------+---------+---------+-------------------+
 | odl_l3-noha         |  12/15  |   80%   | `fuel-res-164`_   |
 +---------------------+---------+---------+-------------------+
-| onos                |  20/21  |   95%   | `fuel-res-117`_   |
+| onos                |  20/21  |   95%   | `fuel-res-492`_   |
 +---------------------+---------+---------+-------------------+
 | onos-noha           |  18/18  |  100%   | `fuel-res-166`_   |
 +---------------------+---------+---------+-------------------+
-| onos-sfc            |  24/24  |  100%   | fuel-res-124`_    |
+| onos-sfc            |  24/24  |  100%   | `fuel-res-124`_   |
 +---------------------+---------+---------+-------------------+
 | onos-sfc-noha       |  21/21  |  100%   | `fuel-res-129`_   |
 +---------------------+---------+---------+-------------------+
 | multisite           |  N.R    |  100%   | `fuel-res-8`_     |
 +---------------------+---------+---------+-------------------+
 
+*: all results passed, lacking iterations to reach the full score
 
-Results of Functest on AArch64 Colorado 1.0
+Results of Functest on AArch64 Colorado 3.0
 
 +---------------------+---------+---------+----------------------+
 |  Scenario           | Scoring | Success |  Results             |
 |                     |         | rate    |                      |
 +=====================+=========+=========+======================+
-| nosdn               |  18/18  |  100%   | `fuel-arm-res-29`_   |
+| nosdn               |  18/18  |  100%   | `fuel-arm-res-128`_  |
 +---------------------+---------+---------+----------------------+
-| odl_l2              |  21/21  |  100%   | `fuel-arm-res-24`_   |
+| odl_l2              |  21/21  |  100%   | `fuel-arm-res-122`_  |
 +---------------------+---------+---------+----------------------+
-| odl_l2-noha         |  17/18  |   94%   | `fuel-arm-res-30`_   |
+| odl_l2-noha         |  18/18  |  100%   | `fuel-arm-res-129`_  |
 +---------------------+---------+---------+----------------------+
-| odl_l3              |  14/18  |   78%   | `fuel-arm-res-28`_   |
+| odl_l3              |  17/18  |   95%   | `fuel-arm-res-135`_  |
 +---------------------+---------+---------+----------------------+
 
 
@@ -383,9 +398,9 @@ Joid
 +---------------------+---------+---------+-----------------+
 | nosdn-lxd-noha      |  12/12  |  100%   | `joid-res-91`_  |
 +---------------------+---------+---------+-----------------+
-| odl_l2              |  19/21  |   91%   | `joid-res-103`_ |
+| odl_l2              |  21/21  |  100%   | `joid-res-103`_ |
 +---------------------+---------+---------+-----------------+
-| onos                |  21/21  |  100%   | `joid-res-99`_  |
+| onos                |  21/21  |  100%   | `joid-res-345`_ |
 +---------------------+---------+---------+-----------------+
 | onos-sfc            |  24/24  |  100%   | `joid-res-97`_  |
 +---------------------+---------+---------+-----------------+
@@ -494,27 +509,12 @@ Open JIRA tickets
 +------------------+-----------------------------------------------+
 |   JIRA           |         Description                           |
 +==================+===============================================+
-| `FUNCTEST-419`_  |  do not try to Remove docker                  |
-|                  |  image opnfv/functest:<none>                  |
-|                  |  reported by joid on Intel POD                |
-|                  |  may impact CI                                |
-|                  |  not reproducible                             |
-+------------------+-----------------------------------------------+
 | `FUNCTEST-446`_  |  Cleanup ODL-SFC output in Functest execution |
 |                  |  Impact on odl_l2-sfc scenarios               |
 +------------------+-----------------------------------------------+
-| `FUNCTEST-450`_  |  Functest is Failing to get the token using   |
-|                  |  keystone client                              |
-+------------------+-----------------------------------------------+
 | `FUNCTEST-454`_  |  Cleanup failures when using HA networks in   |
 |                  |  Neutron                                      |
 +------------------+-----------------------------------------------+
-| `FUNCTEST-460`_  |  Wrong image format used in rally cases       |
-+------------------+-----------------------------------------------+
-| `FUNCTEST-462`_  |  OLD test fails after forcing the clone       |
-|                  |  release/beryllium-sr3 branch                 |
-+------------------+-----------------------------------------------+
-
 
 Useful links
 ============
@@ -541,18 +541,10 @@ Useful links
 
  - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 
-.. _`FUNCTEST-419` : https://jira.opnfv.org/browse/FUNCTEST-419
-
 .. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
 
-.. _`FUNCTEST-450` : https://jira.opnfv.org/browse/FUNCTEST-450
-
 .. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
 
-.. _`FUNCTEST-460` : https://jira.opnfv.org/browse/FUNCTEST-460
-
-.. _`FUNCTEST-462` : https://jira.opnfv.org/browse/FUNCTEST-462
-
 .. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
 
 .. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
@@ -561,6 +553,12 @@ Useful links
 
 .. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
 
+.. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
+
+.. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
+
+.. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423
+
 .. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
 
 .. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
@@ -571,49 +569,57 @@ Useful links
 
 .. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
 
-.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
+.. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
 
-.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
+.. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285
 
-.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
+.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
 
-.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
+.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
 
-.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
+.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
 
 .. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
 
-.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
+.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
 
-.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
+.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
 
-.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
+.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
 
-.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
+.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
 
-.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
+.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
+
+.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
+
+.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
+
+.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
+
+.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
 
 .. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
 
 .. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
 
-.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
+.. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213
 
-.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
+.. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219
 
-.. _`fuel-arm-res-29`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/29/console
+.. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376
 
-.. _`fuel-arm-res-24`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/24/console
+.. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492
 
-.. _`fuel-arm-res-30`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/30/console
+.. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console
 
-.. _`fuel-arm-res-28`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/28/console
+.. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console
 
-.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
+.. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console
 
-.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
+.. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console
 
-.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
+.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
 
 .. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
 
@@ -621,4 +627,11 @@ Useful links
 
 .. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
 
+.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
+
 .. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
+
+.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
+
+.. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345
+
Test suites and cases to verify OPNFV Platform functionality