docs/release/release-notes/functest-release.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. SPDX-License-Identifier: CC-BY-4.0
   3
   4 =======
   5 License
   6 =======
   7
   8 OPNFV Danube release note for Functest Docs
   9 (c) by Jose Lausuch (Ericsson)
  10
  11 OPNFV Danube release note for Functest Docs
  12 are licensed under a Creative Commons Attribution 4.0 International License.
  13 You should have received a copy of the license along with this.
  14 If not, see <http://creativecommons.org/licenses/by/4.0/>.
  15
  16 ===========================================
  17 OPNFV Danube1.0 release note for Functest
  18 ===========================================
  19
  20 Abstract
  21 ========
  22
  23 This document describes the release note of Functest project.
  24
  25
  26 Version history
  27 ===============
  28
  29 +------------+----------+------------------+------------------------+
  30 | **Date**   | **Ver.** | **Author**       | **Comment**            |
  31 |            |          |                  |                        |
  32 +------------+----------+------------------+------------------------+
  33 | 2016-08-17 | 1.0.0    | Morgan Richomme  | Functest for Colorado  |
  34 |            |          | (Orange)         | release                |
  35 +------------+----------+------------------+------------------------+
  36 | 2017-03-29 | 4.0.0    | Jose Lausuch     | Functest for Danube    |
  37 |            |          | (Ericsson)       | release                |
  38 +------------+----------+------------------+------------------------+
  39
  40
  41 OPNFV Danube Release
  42 ======================
  43
  44 Functest deals with functional testing of the OPNFV solution.
  45 It includes test cases developed within the project and test cases developed in
  46 other OPNFV projects and other upstream communities.
  47
  48 The internal test cases are:
  49
  50  * connection_check
  51  * api_check
  52  * snaps_health_check
  53  * vping_ssh
  54  * vping_userdata
  55  * tempest_smoke_serial
  56  * refstack_defcore
  57  * snaps_smoke
  58  * rally_sanity
  59  * odl
  60  * tempest_full_parallel
  61  * rally_full
  62  * cloudify_ims
  63
  64 The OPNFV projects integrated into Functest framework for automation are:
  65
  66  * barometer
  67  * bgpvpn
  68  * doctor
  69  * domino
  70  * fds
  71  * multisite
  72  * netready
  73  * onos
  74  * odl-sfc
  75  * odl-netvirt
  76  * parser
  77  * promise
  78  * security scan
  79  * orchestra_ims
  80  * vyos_vrouter
  81
  82 The validation of a scenario requires a subset of these tests depending
  83 on the installer and the scenario.
  84
  85 The test cases from vnf (cloudify_ims, orchestra_ims, vyos_vrouter) and
  86 component categories (tempest full parallel, Rally full) are not considered for
  87 scenario validation.
  88
  89 Release Data
  90 ============
  91
  92 +--------------------------------------+--------------------------------------+
  93 | **Project**                          | functest                             |
  94 |                                      |                                      |
  95 +--------------------------------------+--------------------------------------+
  96 | **Repo/tag**                         | danube.1.0                           |
  97 |                                      |                                      |
  98 +--------------------------------------+--------------------------------------+
  99 | **Release designation**              | Danube base release                  |
 100 |                                      |                                      |
 101 +--------------------------------------+--------------------------------------+
 102 | **Release date**                     | March 31st 2017                      |
 103 |                                      |                                      |
 104 +--------------------------------------+--------------------------------------+
 105 | **Purpose of the delivery**          | Danube base release                  |
 106 |                                      |                                      |
 107 +--------------------------------------+--------------------------------------+
 108
 109 Deliverables
 110 ============
 111
 112 Software
 113 --------
 114
 115  - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: danube.1.0)
 116
 117  - The TestAPI Docker image: https://hub.docker.com/r/opnfv/testapi (tag:danube.1.0)
 118
 119
 120 Documents
 121 ---------
 122
 123  - Installation/configuration guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
 124
 125  - User Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
 126
 127  - Developer Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/developer/devguide/index.html
 128
 129
 130 Version change
 131 ==============
 132
 133 Feature evolution
 134 -----------------
 135
 136 - Adoption of SNAPS as middleware in 4 new test cases (connection_check, api_check,
 137    snaps_health_check and snaps_smoke)
 138
 139 - Introduction of refstack suite
 140
 141 - Support new odl suites (odl-netvirt, fds)
 142
 143 - Introduction of VNF onboarding capabilities
 144
 145 - Support of new feature projects (fds, netready, barometer, orchestra, vyos_vrouter)
 146
 147
 148
 149 Framework
 150 ---------
 151
 152  - Harmonization of the naming, better adoption of OpenStack coding conventions
 153
 154  - Enhanced code to be more Object Oriented, removed bash scripts
 155
 156  - Introduction of abstraction classes to ease and harmonize the integration of
 157     test cases (internal or from feature projects)
 158
 159  - New management of logger, env variables and configuration files
 160
 161  - Creation of unit tests on the whole framework to ensure stability
 162
 163  - Creation or ARM Functest docker
 164
 165
 166 Test API
 167 ---------
 168
 169 - Automatic documentation (html & pdf)
 170
 171 - Full dockerization and automation of the deployment on testresults.opnfv.org
 172
 173 - Automation of test database backup on artifact
 174
 175
 176 New internal tests cases
 177 ------------------------
 178
 179 - connection_check
 180
 181 - api_check
 182
 183 - snaps_health_check (replacing shell script healtcheck)
 184
 185 - refstack_defcore
 186
 187 - snaps_smoke
 188
 189 - vyos_vrouter
 190
 191
 192 Scenario Matrix
 193 ===============
 194
 195 For Danube 1.0, Functest was tested on the following HA scenarios (new
 196 dabube scenarios in bold):
 197
 198 +---------------------+---------+---------+---------+---------+
 199 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 200 +=====================+=========+=========+=========+=========+
 201 |   nosdn             |    X    |    X    |    X    |    X    |
 202 +---------------------+---------+---------+---------+---------+
 203 | **fdio**            |    X    |         |         |         |
 204 +---------------------+---------+---------+---------+---------+
 205 |   kvm               |    X    |         |    X    |         |
 206 +---------------------+---------+---------+---------+---------+
 207 | **kvm_ovs_dpdk**    |         |         |    X    |         |
 208 +---------------------+---------+---------+---------+---------+
 209 | **kvm_ovs_dpdk-bar**|         |         |    X    |         |
 210 +---------------------+---------+---------+---------+---------+
 211 |   lxd               |         |         |         |    X    |
 212 +---------------------+---------+---------+---------+---------+
 213 | **ovs**             |         |         |    X    |         |
 214 +---------------------+---------+---------+---------+---------+
 215 | **openo**           |         |    X    |         |         |
 216 +---------------------+---------+---------+---------+---------+
 217 |   odl_l2            |         |    X    |   X     |    X    |
 218 +---------------------+---------+---------+---------+---------+
 219 |   odl-bgpvpn        |   X     |         |         |         |
 220 +---------------------+---------+---------+---------+---------+
 221 |   odl_l2-bgpvpn     |         |         |   X     |         |
 222 +---------------------+---------+---------+---------+---------+
 223 | **odl_l2-fdio**     |    X    |         |         |         |
 224 +---------------------+---------+---------+---------+---------+
 225 |   odl_l2-sfc        |         |         |    X    |         |
 226 +---------------------+---------+---------+---------+---------+
 227 |   odl_l3            |    X    |    X    |    X    |         |
 228 +---------------------+---------+---------+---------+---------+
 229 | **ocl**             |         |   X     |         |         |
 230 +---------------------+---------+---------+---------+---------+
 231 |   onos              |         |   X     |         |         |
 232 +---------------------+---------+---------+---------+---------+
 233 |   multisite         |         |         |    X    |         |
 234 +---------------------+---------+---------+---------+---------+
 235
 236 Non HA scenarios:
 237
 238 +---------------------+---------+---------+---------+---------+
 239 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 240 +=====================+=========+=========+=========+=========+
 241 |   nosdn             |         |         |    X    |    X    |
 242 +---------------------+---------+---------+---------+---------+
 243 |   kvm               |         |         |    X    |         |
 244 +---------------------+---------+---------+---------+---------+
 245 | **kvm_ovs_dpdk**    |         |         |    X    |         |
 246 +---------------------+---------+---------+---------+---------+
 247 | **kvm_ovs_dpdk-bar**|         |         |    X    |         |
 248 +---------------------+---------+---------+---------+---------+
 249 |   lxd               |         |         |         |    X    |
 250 +---------------------+---------+---------+---------+---------+
 251 |   ovs               |         |         |    X    |         |
 252 +---------------------+---------+---------+---------+---------+
 253 |   odl_l2            |         |         |   X     |         |
 254 +---------------------+---------+---------+---------+---------+
 255 |   odl_l2-bgpvpn     |         |         |   X     |         |
 256 +---------------------+---------+---------+---------+---------+
 257 |   odl_l2-fdio       |    X    |         |         |         |
 258 +---------------------+---------+---------+---------+---------+
 259 | **odl_l3-fdio**     |    X    |         |         |         |
 260 +---------------------+---------+---------+---------+---------+
 261 |   odl_l2-sfc        |         |         |    X    |         |
 262 +---------------------+---------+---------+---------+---------+
 263 |   odl_l3            |         |         |    X    |         |
 264 +---------------------+---------+---------+---------+---------+
 265 | **odl_gluon**       |    X    |         |         |         |
 266 +---------------------+---------+---------+---------+---------+
 267
 268 Colorado deprecated scenarios:
 269
 270  * odl_l2-moon
 271  * onos-sfc
 272  * onos-noha
 273  * onos-sfc-noha
 274
 275 For the scenario validation, we consider only the healthcheck, smoke and
 276 features categories. These tests are run systematically in the CI daily loops.
 277
 278 Success criterias have been defined for these test cases, they can be
 279 PASS/FAIL or a success rate may be declared (100%, > 90%).
 280 All the criteria, as well as the test dependencies are declared in the
 281 testcases.yaml file located in the ci directory of the repository.
 282
 283 The scoring for the Danube release per installer can be described as
 284 follows.
 285
 286 The scoring is an indicator showing how many feature project test suites
 287 have been integrated in the scenario.
 288
 289 The scoring equals the number of tests * successful iteration of each
 290 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
 291
 292  e.g.
 293  apex/odl_l2-nofeature-ha
 294  tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
 295  Scoring = 21/21 = 7 * 3
 296
 297 By default, if not specified, the scenarios are HA.
 298 HA means OpenStack High Availability (main services). Note that not
 299 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
 300 depends upon support of the specific service for HA deployment.
 301 For example, in the Danube release, the Congress service
 302 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
 303 support for HA operation has not yet been verified.
 304 See the release notes of the installers for more details.
 305
 306
 307 Apex
 308 ----
 309
 310 +------------------+---------+---------+-----------------+
 311 |  Scenario        | Scoring | Success |    Results      |
 312 |                  |         | rate    |                 |
 313 +==================+=========+=========+=================+
 314 | nosdn            |  33/33  |   100%  | `apex-res-1`_   |
 315 +------------------+---------+---------+-----------------+
 316 | odl_l3           |  27/33  |    82%  | `apex-res-2`_   |
 317 +------------------+---------+---------+-----------------+
 318 | odl-bgpvpn       |  26/30  |    87%  | `apex-res-3`_   |
 319 +------------------+---------+---------+-----------------+
 320 | odl-gluon        |  30/36  |    83%  | `apex-res-4`_   |
 321 +------------------+---------+---------+-----------------+
 322 | kvm              |  32/33  |    97%  | `apex-res-5`_   |
 323 +------------------+---------+---------+-----------------+
 324 | odl_l2-fdio      |  28/36  |    78%  | `apex-res-6`_   |
 325 +------------------+---------+---------+-----------------+
 326 | odl_l2-fdio-noha |  30/36  |    83%  | `apex-res-7`_   |
 327 +------------------+---------+---------+-----------------+
 328 | odl_l3-fdio-noha |  26/30  |    87%  | `apex-res-8`_   |
 329 +------------------+---------+---------+-----------------+
 330 | fdio             |   6/30  |    20%  | `apex-res-9`_   |
 331 +------------------+---------+---------+-----------------+
 332
 333 Compass
 334 -------
 335
 336 +------------------+---------+---------+------------------+
 337 |  Scenario        | Scoring | Success |  Results         |
 338 |                  |         | rate    |                  |
 339 +==================+=========+=========+==================+
 340 | nosdn            |  29/30  |    97%  | `compass-res-1`_ |
 341 +------------------+---------+---------+------------------+
 342 | odl_l2           |  28/33  |    84%  | `compass-res-2`_ |
 343 +------------------+---------+---------+------------------+
 344 | odl_l3           |  21/30  |    70%  | `compass-res-3`_ |
 345 +------------------+---------+---------+------------------+
 346 | onos             |  28/33  |    84%  | `compass-res-4`_ |
 347 +------------------+---------+---------+------------------+
 348 | openo            |  28/30  |    93%  | `compass-res-5`_ |
 349 +------------------+---------+---------+------------------+
 350 | ocl              |   4/30  |    13%  | `compass-res-6`_ |
 351 +------------------+---------+---------+------------------+
 352
 353 Note: all the Compass tests for Danube have been executed on virtual
 354 environment. Bare metal resources were used for Master branch.
 355
 356
 357 Fuel
 358 ----
 359
 360 +----------------------+---------+---------+----------------+
 361 |  Scenario            | Scoring | Success |  Results       |
 362 |                      |         | rate    |                |
 363 +======================+=========+=========+================+
 364 | nosdn                |  37/39  |   95%   | `fuel-res-1`_  |
 365 +----------------------+---------+---------+----------------+
 366 | nosdn-noha           |  36/36  |  100%   | `fuel-res-2`_  |
 367 +----------------------+---------+---------+----------------+
 368 | nosdn-kvm            |  37/39  |   95%   | `fuel-res-3`_  |
 369 +----------------------+---------+---------+----------------+
 370 | nosdn-kvm-noha       |  36/36  |  100%   | `fuel-res-4`_  |
 371 +----------------------+---------+---------+----------------+
 372 | nosdn-ovs            |  38/39  |   97%   | `fuel-res-5`_  |
 373 +----------------------+---------+---------+----------------+
 374 | nosdn-ovs-noha       |  36/36  |  100%   | `fuel-res-6`_  |
 375 +----------------------+---------+---------+----------------+
 376 | odl_l2               |  42/42  |  100%   | `fuel-res-7`_  |
 377 +----------------------+---------+---------+----------------+
 378 | odl_l2-noha          |  36/39  |   92%   | `fuel-res-8`_  |
 379 +----------------------+---------+---------+----------------+
 380 | odl_l2-sfc           |  40/45  |   89%   | `fuel-res-11`_ |
 381 +----------------------+---------+---------+----------------+
 382 | odl_l2-sfc-noha      |  36/42  |   86%   | `fuel-res-12`_ |
 383 +----------------------+---------+---------+----------------+
 384 | odl_l3               |  34/39  |   87%   | `fuel-res-13`_ |
 385 +----------------------+---------+---------+----------------+
 386 | odl_l3-noha          |  34/36  |   94%   | `fuel-res-14`_ |
 387 +----------------------+---------+---------+----------------+
 388 | kvm_ovs_dpdk         |   6/39  |   15%   | `fuel-res-15`_ |
 389 +----------------------+---------+---------+----------------+
 390 | kvm_ovs_dpdk_noha    |  36/36  |  100%   | `fuel-res-16`_ |
 391 +----------------------+---------+---------+----------------+
 392 | kvm_ovs_dpdk_bar     |   6/42  |   14%   | `fuel-res-17`_ |
 393 +----------------------+---------+---------+----------------+
 394 | kvm_ovs_dpdk_bar_noha|  38/39  |   97%   | `fuel-res-18`_ |
 395 +----------------------+---------+---------+----------------+
 396
 397
 398
 399
 400 Joid
 401 ----
 402
 403 +---------------------+---------+---------+---------------+
 404 |  Scenario           | Scoring | Success |  Results      |
 405 |                     |         | rate    |               |
 406 +=====================+=========+=========+===============+
 407 | nosdn               |  32/33  |   97%   | `joid-res-1`_ |
 408 +---------------------+---------+---------+---------------+
 409 | nosdn-noha          |  31/33  |   94%   | `joid-res-2`_ |
 410 +---------------------+---------+---------+---------------+
 411 | nosdn-lxd           |  18/24  |   75%   | `joid-res-3`_ |
 412 +---------------------+---------+---------+---------------+
 413 | nosdn-lxd-noha      |  17/24  |   71%   | `joid-res-4`_ |
 414 +---------------------+---------+---------+---------------+
 415 | odl_l2              |   9/36  |   25%   | `joid-res-5`_ |
 416 +---------------------+---------+---------+---------------+
 417
 418 It is highly recommended to install a json viewer in your browser
 419 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
 420
 421 You can get additional details through test logs on http://artifacts.opnfv.org/.
 422 As no search engine is available on the OPNFV artifact web site you must
 423 retrieve the pod identifier on which the tests have been executed (see
 424 field pod in any of the results) then click on the selected POD and look
 425 for the date of the test you are interested in.
 426
 427 The reporting pages can be found at:
 428
 429  * apex: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-apex.html
 430  * compass: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-compass.html
 431  * fuel: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html
 432  * joid: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-joid.html
 433
 434 Danube known restrictions/issues
 435 ==================================
 436
 437 ************* TODO *****************************************
 438
 439 +-----------+-----------+----------------------------------------------+
 440 | Installer | Scenario  |  Issue                                       |
 441 +===========+===========+==============================================+
 442 | fuel      | odl_-*    | Tempest test case "TestServerBasicOps"       |
 443 |           |           | disabled due to bug `tempest-bug`_           |
 444 +-----------+-----------+----------------------------------------------+
 445 | apex/fuel | *-bgpvpn  | Due to some instabilities in the bgpvpn      |
 446 |           |           | test case, the scenario has been postponed   |
 447 |           |           | to Danube 2.0                                |
 448 +-----------+-----------+----------------------------------------------+
 449 | apex      | *-gluon   | vPing_ssh disabled due to floating ips       |
 450 |           |           | not working 100% of the times.               |
 451 |           |           | Tempest test "test_reboot_server_hard"       |
 452 |           |           | disabled due to bug `gluon-bug`_             |
 453 +-----------+-----------+----------------------------------------------+
 454 | joid      | any       | Tempest cases related to object storage      |
 455 |           |           | excluded                                     |
 456 +-----------+-----------+----------------------------------------------+
 457 | any       | any       | The VNF tier has not been fully tested       |
 458 |           |           | since it has not been run in daily loops     |
 459 |           |           | in CI. Weekly jobs have been activated       |
 460 |           |           | a bit late in the process and have not been  |
 461 |           |           | used to validate the scenarios.              |
 462 +-----------+-----------+----------------------------------------------+
 463
 464 Test and installer/scenario dependencies
 465 ========================================
 466
 467 It is not always possible to run all the test cases on all the scenarios.
 468 The following table details the dependencies of the test cases per
 469 scenario. The scenario dependencies (installer or scenario) are detailed
 470 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 471
 472 Test results
 473 ============
 474
 475 Test results are available in:
 476
 477  - test results document: http://artifacts.opnfv.org/functest
 478
 479  - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
 480
 481  - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
 482
 483
 484
 485 Open JIRA tickets
 486 =================
 487
 488 +------------------+-----------------------------------------------+
 489 |   JIRA           |         Description                           |
 490 +==================+===============================================+
 491 |                  |                                               |
 492 |                  |                                               |
 493 +------------------+-----------------------------------------------+
 494
 495 All the tickets that are not blocking have been fixed or postponed
 496 the next release.
 497
 498 Functest Danube 1.0 is released without known bugs.
 499
 500
 501
 502 Useful links
 503 ============
 504
 505  - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
 506
 507  - wiki Functest Danube page: https://wiki.opnfv.org/display/functest/Functest+Danube
 508
 509  - Functest repo: https://git.opnfv.org/cgit/functest
 510
 511  - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
 512
 513  - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
 514
 515  - Functest IRC chan: #opnfv-functest
 516
 517  - Reporting page: http://testresults.opnfv.org/reporting/danube.html
 518
 519  - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/functest/ci/testcases.yaml
 520
 521 .. _`tempest-bug`: https://bugs.launchpad.net/tempest/+bug/1577632
 522
 523 .. _`gluon-bug`: https://bugs.opendaylight.org/show_bug.cgi?id=5586
 524
 525 .. _`apex-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-68
 526
 527 .. _`apex-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-69
 528
 529 .. _`apex-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-70
 530
 531 .. _`apex-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-66
 532
 533 .. _`apex-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-60
 534
 535 .. _`apex-res-6`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-73
 536
 537 .. _`apex-res-7`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-72
 538
 539 .. _`apex-res-8`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-69
 540
 541 .. _`apex-res-9`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-danube-daily-danube-62
 542
 543 .. _`compass-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-60
 544
 545 .. _`compass-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-59
 546
 547 .. _`compass-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-baremetal-daily-danube-69
 548
 549 .. _`compass-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-57
 550
 551 .. _`compass-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-baremetal-daily-danube-67
 552
 553 .. _`compass-res-6`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-baremetal-daily-danube-65
 554
 555 .. _`fuel-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-54
 556
 557 .. _`fuel-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-46
 558
 559 .. _`fuel-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-53
 560
 561 .. _`fuel-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-44
 562
 563 .. _`fuel-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-55
 564
 565 .. _`fuel-res-6`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-45
 566
 567 .. _`fuel-res-7`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-zte-pod1-daily-danube-4
 568
 569 .. _`fuel-res-8`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-48
 570
 571 .. _`fuel-res-9`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-52
 572
 573 .. _`fuel-res-10`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-43
 574
 575 .. _`fuel-res-11`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-50
 576
 577 .. _`fuel-res-12`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-42
 578
 579 .. _`fuel-res-13`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-48
 580
 581 .. _`fuel-res-14`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-50
 582
 583 .. _`fuel-res-15`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-51
 584
 585 .. _`fuel-res-16`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-49
 586
 587 .. _`fuel-res-17`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-49
 588
 589 .. _`fuel-res-18`: http://testresults.opnfv.org/test/api/v1/results?build_tag=  jenkins-functest-fuel-virtual-daily-danube-51
 590
 591 .. _`joid-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-54
 592
 593 .. _`joid-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-55
 594
 595 .. _`joid-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-56
 596
 597 .. _`joid-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-57
 598
 599 .. _`joid-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-46