docs/release/release-notes/functest-release.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. SPDX-License-Identifier: CC-BY-4.0
   3
   4 =======
   5 License
   6 =======
   7
   8 OPNFV Danube release note for Functest Docs
   9 (c) by Jose Lausuch (Ericsson)
  10
  11 OPNFV Danube release note for Functest Docs
  12 are licensed under a Creative Commons Attribution 4.0 International License.
  13 You should have received a copy of the license along with this.
  14 If not, see <http://creativecommons.org/licenses/by/4.0/>.
  15
  16 ===========================================
  17 OPNFV Danube1.0 release note for Functest
  18 ===========================================
  19
  20 Abstract
  21 ========
  22
  23 This document describes the release note of Functest project.
  24
  25
  26 Version history
  27 ===============
  28
  29 +------------+----------+------------------+------------------------+
  30 | **Date**   | **Ver.** | **Author**       | **Comment**            |
  31 |            |          |                  |                        |
  32 +------------+----------+------------------+------------------------+
  33 | 2016-08-17 | 1.0.0    | Morgan Richomme  | Functest for Colorado  |
  34 |            |          | (Orange)         | release                |
  35 +------------+----------+------------------+------------------------+
  36 | 2017-03-27 | 4.0.0    | Jose Lausuch     | Functest for Danube    |
  37 |            |          | (Ericsson)       | release                |
  38 +------------+----------+------------------+------------------------+
  39
  40 OPNFV Danube Release
  41 ======================
  42
  43 Functest deals with functional testing of the OPNFV solution.
  44 It includes test cases developed within the project and test cases developed in
  45 other OPNFV projects and other upstream communities.
  46
  47 The internal test cases are:
  48
  49  * connection_check
  50  * api_check
  51  * snaps_health_check
  52  * vping_ssh
  53  * vping_userdata
  54  * tempest_smoke_serial
  55  * refstack_defcore
  56  * snaps_smoke
  57  * rally_sanity
  58  * odl
  59  * tempest_full_parallel
  60  * rally_full
  61  * cloudify_ims
  62
  63 The OPNFV projects integrated into Functest framework for automation are:
  64
  65  * barometer
  66  * bgpvpn
  67  * doctor
  68  * domino
  69  * fds
  70  * multisite
  71  * netready
  72  * onos
  73  * odl-sfc
  74  * odl-netvirt
  75  * orchestra_ims
  76  * parser
  77  * promise
  78  * security scan
  79  * vyos_vrouter
  80
  81 The validation of a scenario requires a subset of these tests depending
  82 on the installer and the scenario.
  83
  84 The test cases from vnf (cloudify_ims, orchestra_ims, vyos_vrouter) and
  85 component categories (tempest full parallel, Rally full) are not considered for
  86 scenario validation.
  87
  88 Release Data
  89 ============
  90
  91 +--------------------------------------+--------------------------------------+
  92 | **Project**                          | functest                             |
  93 |                                      |                                      |
  94 +--------------------------------------+--------------------------------------+
  95 | **Repo/tag**                         | danube.1.0                           |
  96 |                                      |                                      |
  97 +--------------------------------------+--------------------------------------+
  98 | **Release designation**              | Danube base release                  |
  99 |                                      |                                      |
 100 +--------------------------------------+--------------------------------------+
 101 | **Release date**                     | April 3rd 2017                       |
 102 |                                      |                                      |
 103 +--------------------------------------+--------------------------------------+
 104 | **Purpose of the delivery**          | Danube base release                  |
 105 |                                      |                                      |
 106 +--------------------------------------+--------------------------------------+
 107
 108 Deliverables
 109 ============
 110
 111 Software
 112 --------
 113
 114  - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: danube.1.0)
 115
 116  - The TestAPI Docker image: https://hub.docker.com/r/opnfv/testapi (tag:danube.1.0)
 117
 118
 119 Documents
 120 ---------
 121
 122  - Installation/configuration guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
 123
 124  - User Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
 125
 126  - Developer Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/developer/devguide/index.html
 127
 128
 129 Version change
 130 ==============
 131
 132 Feature evolution
 133 -----------------
 134
 135 - Adoption of SNAPS as middleware in 4 new test cases (connection_check, api_check,
 136 snaps_health_check and snaps_smoke)
 137
 138 - Introduction of refstack suite
 139
 140 - Support new odl suites (odl-netvirt, fds)
 141
 142 - Introduction of VNF onboarding capabilities
 143
 144 - Support of new feature projects (fds, netready, barometer, orchestra, vyos_vrouter)
 145
 146
 147
 148 Framework
 149 ---------
 150
 151  - Harmonization of the naming, better adoption of OpenStack coding conventions
 152
 153  - Pythonization, rewritting of bash scripts
 154
 155  - Introduction of abstraction classes to ease and harmonize the integration of
 156  test cases (internal or from feature projects)
 157
 158  - New management of logger, env variables and configuration files
 159
 160  - Creation of unit tests on the whole framework to ensure stability
 161
 162  - Creation or ARM Functest docker
 163
 164
 165 Test API
 166 ---------
 167
 168 - automatic documentation (html & pdf)
 169
 170 - full dockerization and automation of the deployment on testresults.opnfv.org
 171
 172 - automation of test database backup on artifact
 173
 174
 175
 176
 177 Scenario Matrix
 178 ===============
 179
 180 For Danube 1.0, Functest was tested on the following HA scenarios (new
 181 dabube scenarios in bold):
 182
 183 +---------------------+---------+---------+---------+---------+
 184 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 185 +=====================+=========+=========+=========+=========+
 186 |   nosdn             |    X    |    X    |    X    |    X    |
 187 +---------------------+---------+---------+---------+---------+
 188 | **fdio**            |    X    |         |         |         |
 189 +---------------------+---------+---------+---------+---------+
 190 |   kvm               |    X    |         |    X    |         |
 191 +---------------------+---------+---------+---------+---------+
 192 | **kvm_ovs_dpdk**    |         |         |    X    |         |
 193 +---------------------+---------+---------+---------+---------+
 194 | **kvm_ovs_dpdk-bar**|         |         |    X    |         |
 195 +---------------------+---------+---------+---------+---------+
 196 |   lxd               |         |         |         |    X    |
 197 +---------------------+---------+---------+---------+---------+
 198 | **ovs**             |         |         |    X    |         |
 199 +---------------------+---------+---------+---------+---------+
 200 | **openo**           |         |    X    |         |         |
 201 +---------------------+---------+---------+---------+---------+
 202 |   odl_l2            |         |    X    |   X     |    X    |
 203 +---------------------+---------+---------+---------+---------+
 204 |   odl-bgpvpn        |   X     |         |         |         |
 205 +---------------------+---------+---------+---------+---------+
 206 |   odl_l2-bgpvpn     |         |         |   X     |         |
 207 +---------------------+---------+---------+---------+---------+
 208 | **odl_l2-fdio**     |    X    |         |         |         |
 209 +---------------------+---------+---------+---------+---------+
 210 |   odl_l2-sfc        |         |         |    X    |         |
 211 +---------------------+---------+---------+---------+---------+
 212 |   odl_l3            |    X    |    X    |    X    |         |
 213 +---------------------+---------+---------+---------+---------+
 214 | **ocl**             |         |   X     |         |         |
 215 +---------------------+---------+---------+---------+---------+
 216 |   onos              |         |   X     |         |         |
 217 +---------------------+---------+---------+---------+---------+
 218 |   multisite         |         |         |    X    |         |
 219 +---------------------+---------+---------+---------+---------+
 220
 221 Non HA scenarios:
 222
 223 +---------------------+---------+---------+---------+---------+
 224 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 225 +=====================+=========+=========+=========+=========+
 226 |   nosdn             |         |         |    X    |    X    |
 227 +---------------------+---------+---------+---------+---------+
 228 |   kvm               |         |         |    X    |         |
 229 +---------------------+---------+---------+---------+---------+
 230 | **kvm_ovs_dpdk**    |         |         |    X    |         |
 231 +---------------------+---------+---------+---------+---------+
 232 | **kvm_ovs_dpdk-bar**|         |         |    X    |         |
 233 +---------------------+---------+---------+---------+---------+
 234 |   lxd               |         |         |         |    X    |
 235 +---------------------+---------+---------+---------+---------+
 236 |   ovs               |         |         |    X    |         |
 237 +---------------------+---------+---------+---------+---------+
 238 |   odl_l2            |         |         |   X     |         |
 239 +---------------------+---------+---------+---------+---------+
 240 |   odl_l2-bgpvpn     |         |         |   X     |         |
 241 +---------------------+---------+---------+---------+---------+
 242 |   odl_l2-fdio       |    X    |         |         |         |
 243 +---------------------+---------+---------+---------+---------+
 244 | **odl_l3-fdio**     |    X    |         |         |         |
 245 +---------------------+---------+---------+---------+---------+
 246 |   odl_l2-sfc        |         |         |    X    |         |
 247 +---------------------+---------+---------+---------+---------+
 248 |   odl_l3            |         |         |    X    |         |
 249 +---------------------+---------+---------+---------+---------+
 250 | **odl_gluon**       |    X    |         |         |         |
 251 +---------------------+---------+---------+---------+---------+
 252
 253 Colorado deprecated scenarios:
 254
 255  * odl_l2-moon
 256  * onos-sfc
 257  * onos-noha
 258  * onos-sfc-noha
 259
 260 For the scenario validation, we consider only the healthcheck, smoke and
 261 features categories. These tests are run systematically in the CI daily loops.
 262
 263 Success criteria have been defined for these test cases, they can be
 264 PASS/FAIL or a success rate may be declared (100%, > 90%)
 265 All the criteria, as well as the test dependencies are declared in the
 266 ci/testcases.yaml file.
 267
 268 The scoring for the Danube release per installer can be described as
 269 follows.
 270
 271 The scoring is an indicator showing how many feature project test suites
 272 have been integrated on the scenario.
 273
 274 The scoring equals the number of tests * succesful iteration of each
 275 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
 276
 277  e.g.
 278  apex/odl_l2-nofeature-ha
 279  tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
 280  Scoring = 21/21 = 7 * 3
 281
 282 By default, if not specified, the scenarios are HA.
 283 HA means OpenStack High Availability (main services). Note that not
 284 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
 285 depends upon support of the specific service for HA deployment.
 286 For example, in the Danube release, the Congress service
 287 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
 288 support for HA operation has not yet been verified.
 289 See the release notes of the installers for more details.
 290
 291
 292 Apex
 293 ----
 294
 295 +------------------+---------+---------+-------------------+
 296 |  Scenario        | Scoring | Success |    Results        |
 297 |                  |         | rate    |                   |
 298 +==================+=========+=========+===================+
 299 ************* TODO *****************************************
 300 | nosdn            |  17/18  |   95%   | `apex-res-174`_   |
 301 +------------------+---------+---------+-------------------+
 302 | odl_l2           |  21/21  |   100%  | `apex-res-175`_   |
 303 +------------------+---------+---------+-------------------+
 304 | odl_l3           |  15/18  |    83%  | `apex-res-176`_   |
 305 +------------------+---------+---------+-------------------+
 306 | odl_l2-bgpvpn    |  14/18  |    78%  | `apex-res-235`_   |
 307 +------------------+---------+---------+-------------------+
 308 | odl_l2-fdio-noha |  12/15  |    80%  | `apex-res-6`_     |
 309 +------------------+---------+---------+-------------------+
 310 | odl_l2-sfc-noha  |  18/24  |    75%  | `apex-res-217`_   |
 311 +------------------+---------+---------+-------------------+
 312 | onos-nofeature-ha|  20/21  |    95%  | `apex-res-423`_   |
 313 +------------------+---------+---------+-------------------+
 314
 315
 316 Compass
 317 -------
 318
 319 +------------------+---------+---------+-------------------+
 320 |  Scenario        | Scoring | Success |  Results          |
 321 |                  |         | rate    |                   |
 322 +==================+=========+=========+===================+
 323 ************* TODO *****************************************
 324 | nosdn            |  12/12  |   100%  | `compass-res-55`_ |
 325 +------------------+---------+---------+-------------------+
 326 | odl_l2           |  15/15  |   100%  | `compass-res-59`_ |
 327 +------------------+---------+---------+-------------------+
 328 | odl_l3           |  9/12   |    75%  | `compass-res-73`_ |
 329 +------------------+---------+---------+-------------------+
 330 | odl_l2-moon      |  15/18  |    83%  | `compass-res-567`_|
 331 +------------------+---------+---------+-------------------+
 332 | onos-ha          |  15/15  |   100%  | `compass-res-285`_|
 333 +------------------+---------+---------+-------------------+
 334 | onos-sfc-ha      |  17/18  |    95%  | `compass-res-76`_ |
 335 +------------------+---------+---------+-------------------+
 336
 337 Note: all the Compass tests for Danube have been executed on virtual
 338 environment. Bare metal resources were used for Master branch.
 339
 340
 341 Fuel
 342 ----
 343
 344 +---------------------+---------+---------+-------------------+
 345 |  Scenario           | Scoring | Success |  Results          |
 346 |                     |         | rate    |                   |
 347 +=====================+=========+=========+===================+
 348 ************* TODO *****************************************
 349 | nosdn               |  18/18  |  100%   | `fuel-res-129`_   |
 350 +---------------------+---------+---------+-------------------+
 351 | nosdn-noha          |  15/15  |  100%   | `fuel-res-154`_   |
 352 +---------------------+---------+---------+-------------------+
 353 | nosdn-kvm           |  18/18  |  100%   | `fuel-res-128`_   |
 354 +---------------------+---------+---------+-------------------+
 355 | nosdn-kvm-noha      |  15/15  |  100%   | `fuel-res-161`_   |
 356 +---------------------+---------+---------+-------------------+
 357 | nosdn-ovs           |  12/18  |   67%*  |  `fuel-res-213`_  |
 358 +---------------------+---------+---------+-------------------+
 359 | nosdn-ovs-noha      |  15/15  |  100%   | `fuel-res-162`_   |
 360 +---------------------+---------+---------+-------------------+
 361 | odl_l2              |  21/21  |  100%   |  `fuel-res-123`_  |
 362 +---------------------+---------+---------+-------------------+
 363 | odl_l2-noha         |  17/18  |   94%   | `fuel-res-155`_   |
 364 +---------------------+---------+---------+-------------------+
 365 | odl_l2-bgpvpn       |  14/18  |   78%   | `fuel-res-119`_   |
 366 +---------------------+---------+---------+-------------------+
 367 | odl_l2-bgpvpn-noha  |  14/15  |   93%   | `fuel-res-160`_   |
 368 +---------------------+---------+---------+-------------------+
 369 | odl_l2-sfc-noha     |   6/21  |   29%   | `fuel-res-219`_   |
 370 +---------------------+---------+---------+-------------------+
 371 | odl_l2-sfc-ha       |  16/21  |   76%   | `fuel-res-376`_   |
 372 +---------------------+---------+---------+-------------------+
 373 | odl_l3              |  15/18  |   83%   | `fuel-res-115`_   |
 374 +---------------------+---------+---------+-------------------+
 375 | odl_l3-noha         |  12/15  |   80%   | `fuel-res-164`_   |
 376 +---------------------+---------+---------+-------------------+
 377 | onos                |  20/21  |   95%   | `fuel-res-492`_   |
 378 +---------------------+---------+---------+-------------------+
 379 | onos-noha           |  18/18  |  100%   | `fuel-res-166`_   |
 380 +---------------------+---------+---------+-------------------+
 381 | onos-sfc            |  24/24  |  100%   | `fuel-res-124`_   |
 382 +---------------------+---------+---------+-------------------+
 383 | onos-sfc-noha       |  21/21  |  100%   | `fuel-res-129`_   |
 384 +---------------------+---------+---------+-------------------+
 385 | multisite           |  N.R    |  100%   | `fuel-res-8`_     |
 386 +---------------------+---------+---------+-------------------+
 387
 388 *: all results passed, lacking iterations to reach the full score
 389
 390 Results of Functest on AArch64 Danube 3.0
 391
 392 +---------------------+---------+---------+----------------------+
 393 |  Scenario           | Scoring | Success |  Results             |
 394 |                     |         | rate    |                      |
 395 +=====================+=========+=========+======================+
 396 ************* TODO *****************************************
 397 | nosdn               |  18/18  |  100%   | `fuel-arm-res-128`_  |
 398 +---------------------+---------+---------+----------------------+
 399 | odl_l2              |  21/21  |  100%   | `fuel-arm-res-122`_  |
 400 +---------------------+---------+---------+----------------------+
 401 | odl_l2-noha         |  18/18  |  100%   | `fuel-arm-res-129`_  |
 402 +---------------------+---------+---------+----------------------+
 403 | odl_l3              |  17/18  |   95%   | `fuel-arm-res-135`_  |
 404 +---------------------+---------+---------+----------------------+
 405
 406
 407 Joid
 408 ----
 409
 410 +---------------------+---------+---------+-----------------+
 411 |  Scenario           | Scoring | Success |  Results        |
 412 |                     |         | rate    |                 |
 413 +=====================+=========+=========+=================+
 414 ************* TODO *****************************************
 415 | nosdn               |  18/18  |  100%   | `joid-res-102`_ |
 416 +---------------------+---------+---------+-----------------+
 417 | nosdn-noha          |  17/18  |   95%   | `joid-res-93`_  |
 418 +---------------------+---------+---------+-----------------+
 419 | nosdn-lxd           |  12/12  |  100%   | `joid-res-104`_ |
 420 +---------------------+---------+---------+-----------------+
 421 | nosdn-lxd-noha      |  12/12  |  100%   | `joid-res-91`_  |
 422 +---------------------+---------+---------+-----------------+
 423 | odl_l2              |  21/21  |  100%   | `joid-res-103`_ |
 424 +---------------------+---------+---------+-----------------+
 425 | onos                |  21/21  |  100%   | `joid-res-345`_ |
 426 +---------------------+---------+---------+-----------------+
 427 | onos-sfc            |  24/24  |  100%   | `joid-res-97`_  |
 428 +---------------------+---------+---------+-----------------+
 429
 430 It is highly recommended to install a json viewer in your browser
 431 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
 432
 433 You can get additional details through test logs on http://artifacts.opnfv.org/.
 434 As no search engine is available on the OPNFV artifact web site you must
 435 retrieve the pod identifier on which the tests have been executed (see
 436 field pod in any of the results) then click on the selected POD and look
 437 for the date of the test you are interested in.
 438
 439 The reporting pages can be found at:
 440
 441  * apex: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-apex.html
 442  * compass: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-compass.html
 443  * fuel: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html
 444  * joid: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-joid.html
 445
 446 Danube known restrictions/issues
 447 ==================================
 448
 449 ************* TODO *****************************************
 450
 451 +-----------+-----------+----------------------------------------------+
 452 | Installer | Scenario  |  Issue                                       |
 453 +===========+===========+==============================================+
 454 | any       | odl_l3-*  | Tempest cases related to using floating IP   |
 455 |           |           | addresses fail because of a known ODL bug.   |
 456 |           |           | vPing_ssh test case is excluded for the same |
 457 |           |           | reason.                                      |
 458 |           |           | https://jira.opnfv.org/browse/APEX-112       |
 459 |           |           | https://jira.opnfv.org/browse/FUNCTEST-445   |
 460 +-----------+-----------+----------------------------------------------+
 461 | apex/fuel | *-bgpvpn  | vPing_ssh (floating ips not supported) and   |
 462 |           |           | vIMS excluded. Some Tempest cases related to |
 463 |           |           | floating ips also excluded. Some performance |
 464 |           |           | issues have been detected in this scenario   |
 465 |           |           | (i.e. BGPVPN extension enabled) when running |
 466 |           |           | commands against the OpenStack APIs, thus    |
 467 |           |           | Rally sanity test case has been disabled.    |
 468 |           |           | Performance issues seem to be connected to   |
 469 |           |           | the ODL version. It is planned to reintroduce|
 470 |           |           | Rally sanity in Danube 2.0 with the          |
 471 |           |           | adoption of ODL Boron release.               |
 472 +-----------+-----------+----------------------------------------------+
 473 | fuel      | odl       | TestServerBasicOps test case skipped         |
 474 |           |           | Tempest ssh client is hanging on opendaylight|
 475 |           |           | enabled envs (getting deadlock in paramiko   |
 476 |           |           | recv_exit_status method) while trying to     |
 477 |           |           | execute a command on a vm.                   |
 478 +-----------+-----------+----------------------------------------------+
 479 | joid      | any       | Tempest cases related to object storage      |
 480 |           |           | excluded                                     |
 481 |           |           | https://gerrit.opnfv.org/gerrit/#/c/17871/   |
 482 +-----------+-----------+----------------------------------------------+
 483
 484
 485 Test and installer/scenario dependencies
 486 ========================================
 487
 488 It is not always possible to run all the test cases on all the scenarios.
 489 The following table details the dependencies of the test cases per
 490 scenario. The scenario dependencies (installer or scenario) are detailed
 491 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 492
 493 Test results
 494 ============
 495
 496 Test results are available in:
 497
 498  - test results document: http://artifacts.opnfv.org/functest
 499
 500  - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
 501
 502  - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
 503
 504
 505
 506 Open JIRA tickets
 507 =================
 508
 509 +------------------+-----------------------------------------------+
 510 |   JIRA           |         Description                           |
 511 +==================+===============================================+
 512 ************* TODO *****************************************
 513 |                  |                                               |
 514 |                  |                                               |
 515 +------------------+-----------------------------------------------+
 516
 517 Useful links
 518 ============
 519
 520  - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
 521
 522  - wiki Functest Danube page: https://wiki.opnfv.org/display/functest/Functest+Danube
 523
 524  - Functest repo: https://git.opnfv.org/cgit/functest
 525
 526  - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
 527
 528  - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
 529
 530  - Functest IRC chan: #opnfv-functest
 531
 532  - Reporting page: http://testresults.opnfv.org/reporting/danube.html
 533
 534  - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/functest/ci/testcases.yaml