docs/release-notes/functest-release.rst

   1 =======
   2 License
   3 =======
   4
   5 OPNFV Colorado release note for Functest Docs
   6 (c) by Morgan Richomme (Orange)
   7
   8 OPNFV Colorado release note for Functest Docs
   9 are licensed under a Creative Commons Attribution 4.0 International License.
  10 You should have received a copy of the license along with this.
  11 If not, see <http://creativecommons.org/licenses/by/4.0/>.
  12
  13 ===========================================
  14 OPNFV Colorado1.0 release note for Functest
  15 ===========================================
  16
  17 Abstract
  18 ========
  19
  20 This document describes the release note of Functest project.
  21
  22
  23 Version history
  24 ===============
  25
  26 +------------+----------+------------------+------------------------+
  27 | **Date**   | **Ver.** | **Author**       | **Comment**            |
  28 |            |          |                  |                        |
  29 +------------+----------+------------------+------------------------+
  30 | 2016-08-17 | 1.0.0    | Morgan Richomme  | Functest for Colorado  |
  31 |            |          | (Orange)         | release                |
  32 +------------+----------+------------------+------------------------+
  33
  34 OPNFV Colorado Release
  35 =========================
  36
  37 Functest deals with functional testing of the OPNFV solution.
  38 It includes test cases developed within the project and test cases developed in
  39 other OPNFV projects and other upstream communities.
  40
  41 The internal test cases are:
  42
  43  * healthcheck
  44  * vPing ssh
  45  * vPing userdata
  46  * Tempest Smoke Serial
  47  * Rally Sanity
  48  * ODL
  49  * Tempest full parallel
  50  * Rally full
  51  * vIMS
  52
  53 The OPNFV projects integrated into Functest framework for automation are:
  54
  55  * bgpvpn
  56  * Copper
  57  * Doctor
  58  * Domino
  59  * Moon
  60  * Multisite
  61  * ONOSFW
  62  * ONOS-sfc
  63  * ODL-sfc
  64  * Parser
  65  * Promise
  66  * Security scan
  67
  68 The validation of a scenario requires a subset of these tests depending
  69 on the installer and the scenario.
  70
  71 The 3 last internal test cases (tempest full parallel, Rally full and
  72 vIMS) are not considered for scenario validation.
  73
  74 Release Data
  75 ============
  76
  77 +--------------------------------------+--------------------------------------+
  78 | **Project**                          | functest                             |
  79 |                                      |                                      |
  80 +--------------------------------------+--------------------------------------+
  81 | **Repo/tag**                         | colorado.1.0                         |
  82 |                                      |                                      |
  83 +--------------------------------------+--------------------------------------+
  84 | **Release designation**              | Colorado base release                |
  85 |                                      |                                      |
  86 +--------------------------------------+--------------------------------------+
  87 | **Release date**                     | September 22 2016                    |
  88 |                                      |                                      |
  89 +--------------------------------------+--------------------------------------+
  90 | **Purpose of the delivery**          | Colorado base release                |
  91 |                                      |                                      |
  92 +--------------------------------------+--------------------------------------+
  93
  94 Deliverables
  95 ============
  96
  97 Software
  98 --------
  99
 100  - The Functest Docker image: * TODO link *
 101
 102  - The testapi Docker image: * TODO link *
 103
 104
 105 Documents
 106 ---------
 107
 108  - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
 109
 110  - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
 111
 112  - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 113
 114
 115 Version change
 116 ==============
 117
 118 Feature evolution
 119 -----------------
 120
 121  - refactoring of ODL functional tests (with upstream modifications)
 122
 123  - refactoring of testapi (update, swagger documentation, dockerization)
 124
 125  - jenkins logs improvement
 126
 127  - update integration of Doctor, Promise and SDNVPN  projects
 128
 129  - split Tempest and rally into 2 different tests: smoke and full
 130
 131  - vIMS test suite integration
 132
 133  - adoption of Kibana for dashboarding
 134
 135
 136 New features
 137 ------------
 138
 139  - Functest CLI to prepare and run the tests
 140
 141  - creation of the healthcheck test case
 142
 143  - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
 144
 145  - integration of new OPNFV feature projects (copper, domino, multisite,
 146  moon, parser, onos-sfc, odl-sfc, security scan)
 147
 148  - introduction of test tiers in functest framework
 149
 150  - automatic reporting
 151
 152  - introduction of a jenkins summary table
 153
 154  - support of ARM architecture
 155
 156
 157 Scenario Matrix
 158 ===============
 159
 160 For Colorado 1.0, Functest was tested on the following scenarios (if not
 161 precised, the scenario is a HA scenario):
 162
 163 +---------------------+---------+---------+---------+---------+
 164 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 165 +=====================+=========+=========+=========+=========+
 166 |   nosdn             |    X    |    X    |    X    |    X    |
 167 +---------------------+---------+---------+---------+---------+
 168 |   nosdn-noha        |         |         |    X    |    X    |
 169 +---------------------+---------+---------+---------+---------+
 170 |   odl_l2            |    X    |    X    |    X    |    X    |
 171 +---------------------+---------+---------+---------+---------+
 172 |   odl_l2-noha       |         |         |    X    |         |
 173 +---------------------+---------+---------+---------+---------+
 174 |   odl_l3            |    X    |    X    |    X    |         |
 175 +---------------------+---------+---------+---------+---------+
 176 |   odl_l3-noha       |         |         |    X    |         |
 177 +---------------------+---------+---------+---------+---------+
 178 |   odl_l2-bgpvpn     |    X    |         |    X    |         |
 179 +---------------------+---------+---------+---------+---------+
 180 |   odl_l2-bgpvpn-noha|         |         |    X    |         |
 181 +---------------------+---------+---------+---------+---------+
 182 |   odl_l2-fdio-noha  |    X    |         |         |         |
 183 +---------------------+---------+---------+---------+---------+
 184 |   odl_l2-moon       |         |    X    |         |         |
 185 +---------------------+---------+---------+---------+---------+
 186 |   odl_l2-sfc        |         |         |    X    |         |
 187 +---------------------+---------+---------+---------+---------+
 188 |   odl_l2-sfc-noha   |    X    |         |    X    |         |
 189 +---------------------+---------+---------+---------+---------+
 190 |   onos              |         |    X    |    X    |    X    |
 191 +---------------------+---------+---------+---------+---------+
 192 |   onos-noha         |         |         |    X    |         |
 193 +---------------------+---------+---------+---------+---------+
 194 |   onos-sfc          |         |    X    |    X    |    X    |
 195 +---------------------+---------+---------+---------+---------+
 196 |   onos-sfc-noha     |         |         |    X    |         |
 197 +---------------------+---------+---------+---------+---------+
 198 |   ovs-noha (dpdk)   |         |         |    X    |         |
 199 +---------------------+---------+---------+---------+---------+
 200 |   kvm               |         |         |    X    |         |
 201 +---------------------+---------+---------+---------+---------+
 202 |   kvm-noha          |         |         |    X    |         |
 203 +---------------------+---------+---------+---------+---------+
 204 |   multisite         |         |         |    X    |         |
 205 +---------------------+---------+---------+---------+---------+
 206 |   lxd               |         |         |         |    X    |
 207 +---------------------+---------+---------+---------+---------+
 208 |   lxd-noha          |         |         |         |    X    |
 209 +---------------------+---------+---------+---------+---------+
 210
 211 In Colorado, the functional tests have been sliced in 6 different
 212 categories:
 213
 214 +----------------+-----------------------------------------------+
 215 | Category       |  Description                                  |
 216 +================+===============================================+
 217 | healthcheck    | Basic OpenStack commands                      |
 218 +----------------+-----------------------------------------------+
 219 | smoke          | vPings, Tempest and rally smoke tests         |
 220 +----------------+-----------------------------------------------+
 221 | sdn_suites     | Specific SDN feature tests                    |
 222 +----------------+-----------------------------------------------+
 223 | features       | OPNFV feature project functional test suites  |
 224 +----------------+-----------------------------------------------+
 225 | openstack      | Advanced, long duration OpenStack tests       |
 226 |                | (Tempest and Rally full suite). Each test may |
 227 |                | last several hours                            |
 228 +----------------+-----------------------------------------------+
 229 | vnf            | Complex scenarios dealing with orchestration, |
 230 |                | VNF deployment and tests (e.g. vIMS)          |
 231 +----------------+-----------------------------------------------+
 232
 233 For the scenario validation, we consider only the healthcheck, smoke,
 234 sdn_suites and features categories. These tests are run systematically
 235 in the CI daily loops.
 236
 237 Success criteria have been defined for these test cases, they can be
 238 PASS/FAIL or a success rate may be declared (100%, > 90%)
 239 All the criteria, as well as the test dependencies are declared in the
 240 ci/testcases.yaml file.
 241
 242 The scoring for the Colorado release per installer can be described as
 243 follows.
 244
 245 The scoring is an indicator showing how many feature project test suites
 246 have been integrated on the scenario.
 247
 248 The scoring equals the number of tests * succesful iteration of each
 249 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
 250
 251  e.g.
 252  apex/odl_l2-nofeature-ha
 253  tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
 254  Scoring = 21/21 = 7 * 3
 255
 256 By default, if not specified, the scenarios are HA.
 257 HA means OpenStack High Availability (main services). Note that not
 258 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
 259 depends upon support of the specific service for HA deployment.
 260 For example, in the Colorado release, the Congress service
 261 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
 262 support for HA operation has not yet been verified.
 263 See the release notes of the installers for more details.
 264
 265
 266 Apex
 267 ----
 268
 269 +------------------+---------+---------+-------------------+
 270 |  Scenario        | Scoring | Success |    Results        |
 271 |                  |         | rate    |                   |
 272 +==================+=========+=========+===================+
 273 | nosdn            |  17/18  |   95%   | `apex-res-174`_   |
 274 +------------------+---------+---------+-------------------+
 275 | odl_l2           |  21/21  |   100%  | `apex-res-175`_   |
 276 +------------------+---------+---------+-------------------+
 277 | odl_l3           |  15/18  |    83%  | `apex-res-176`_   |
 278 +------------------+---------+---------+-------------------+
 279 | odl_l2-bgpvpn    |  14/18  |    78%  |     ??            |
 280 +------------------+---------+---------+-------------------+
 281 | odl_l2-fdio-noha |  12/15  |    80%  | `apex-res-6`_     |
 282 +------------------+---------+---------+-------------------+
 283 | odl_l2-sfc-noha  |  18/21  |    86%  |     ??            |
 284 +------------------+---------+---------+-------------------+
 285
 286 Compass
 287 -------
 288
 289 +------------------+---------+---------+-------------------+
 290 |  Scenario        | Scoring | Success |  Results          |
 291 |                  |         | rate    |                   |
 292 +==================+=========+=========+===================+
 293 | nosdn            |  12/12  |   100%  | `compass-res-55`_ |
 294 +------------------+---------+---------+-------------------+
 295 | odl_l2           |  15/15  |   100%  | `compass-res-59`_ |
 296 +------------------+---------+---------+-------------------+
 297 | odl_l3           |  9/12   |    75%  | `compass-res-73`_ |
 298 +------------------+---------+---------+-------------------+
 299 | odl_l2-moon      |  9/18   |      %  |    ??             |
 300 +------------------+---------+---------+-------------------+
 301 | onos-ha          |  15/15  |   100%  | `compass-res-77`_ |
 302 +------------------+---------+---------+-------------------+
 303 | onos-sfc-ha      |  16/18  |   100%  | `compass-res-76`_ |
 304 +------------------+---------+---------+-------------------+
 305
 306 Note: all the Compass tests for Colorado have been executed on virtual
 307 environment. Bare metal resources were used for Master branch.
 308
 309
 310 Fuel
 311 ----
 312
 313 +---------------------+---------+---------+-------------------+
 314 |  Scenario           | Scoring | Success |  Results          |
 315 |                     |         | rate    |                   |
 316 +=====================+=========+=========+===================+
 317 | nosdn               |  18/18  |  100%   | `fuel-res-129`_   |
 318 +---------------------+---------+---------+-------------------+
 319 | nosdn-noha          |  15/15  |  100%   | `fuel-res-154`_   |
 320 +---------------------+---------+---------+-------------------+
 321 | nosdn-kvm           |  18/18  |  100%   | `fuel-res-128`_   |
 322 +---------------------+---------+---------+-------------------+
 323 | nosdn-kvm-noha      |  15/15  |  100%   | `fuel-res-161`_   |
 324 +---------------------+---------+---------+-------------------+
 325 | nosdn-ovs-noha      |  15/15  |  100%   | `fuel-res-162`_   |
 326 +---------------------+---------+---------+-------------------+
 327 | odl_l2-sfc          |  21/21  |  100%   |                   |
 328 +---------------------+---------+---------+-------------------+
 329 | odl_l2-sfc-noha     |  16/18  |   88%   |                   |
 330 +---------------------+---------+---------+-------------------+
 331 | odl_l2              |  21/21  |  100%   |  `fuel-res-123`_  |
 332 +---------------------+---------+---------+-------------------+
 333 | odl_l2-noha         |  17/18  |   94%   | `fuel-res-155`_   |
 334 +---------------------+---------+---------+-------------------+
 335 | odl_l2-bgpvpn       |  17/18  |   94%   | `fuel-res-119`_   |
 336 +---------------------+---------+---------+-------------------+
 337 | odl_l2-bgpvpn-noha  |  15/15  |  100%   | `fuel-res-160`_   |
 338 +---------------------+---------+---------+-------------------+
 339 | odl_l3              |  15/18  |   67%   | `fuel-res-115`_   |
 340 +---------------------+---------+---------+-------------------+
 341 | odl_l3-noha         |  12/15  |   80%   | `fuel-res-164`_   |
 342 +---------------------+---------+---------+-------------------+
 343 | onos                |  20/21  |   95%   | `fuel-res-117`_   |
 344 +---------------------+---------+---------+-------------------+
 345 | onos-noha           |  18/18  |  100%   | `fuel-res-166`_   |
 346 +---------------------+---------+---------+-------------------+
 347 | onos-sfc            |  24/24  |  100%   | fuel-res-124`_    |
 348 +---------------------+---------+---------+-------------------+
 349 | onos-sfc-noha       |  21/21  |  100%   | `fuel-res-129`_   |
 350 +---------------------+---------+---------+-------------------+
 351 | multisite           |  N.R    |  100%   | `fuel-res-8`_     |
 352 +---------------------+---------+---------+-------------------+
 353
 354
 355 Results of Functest on AArch64 Colorado 1.0
 356
 357 +---------------------+---------+---------+----------------------+
 358 |  Scenario           | Scoring | Success |  Results             |
 359 |                     |         | rate    |                      |
 360 +=====================+=========+=========+======================+
 361 | nosdn               |  18/18  |  100%   | `fuel-arm-res-29`_   |
 362 +---------------------+---------+---------+----------------------+
 363 | odl_l2              |  21/21  |  100%   | `fuel-arm-res-24`_   |
 364 +---------------------+---------+---------+----------------------+
 365 | odl_l2-noha         |  17/18  |   94%   | `fuel-arm-res-30`_   |
 366 +---------------------+---------+---------+----------------------+
 367 | odl_l3              |  14/18  |   78%   | `fuel-arm-res-28`_   |
 368 +---------------------+---------+---------+----------------------+
 369
 370
 371 Joid
 372 ----
 373
 374 +---------------------+---------+---------+-----------------+
 375 |  Scenario           | Scoring | Success |  Results        |
 376 |                     |         | rate    |                 |
 377 +=====================+=========+=========+=================+
 378 | nosdn               |  18/18  |  100%   | `joid-res-102`_ |
 379 +---------------------+---------+---------+-----------------+
 380 | nosdn-noha          |  17/18  |   95%   | `joid-res-93`_  |
 381 +---------------------+---------+---------+-----------------+
 382 | nosdn-lxd           |  12/12  |  100%   | `joid-res-104`_ |
 383 +---------------------+---------+---------+-----------------+
 384 | nosdn-lxd-noha      |  12/12  |  100%   | `joid-res-91`_  |
 385 +---------------------+---------+---------+-----------------+
 386 | odl_l2              |  19/21  |   91%   | `joid-res-103`_ |
 387 +---------------------+---------+---------+-----------------+
 388 | onos                |  21/21  |  100%   | `joid-res-99`_  |
 389 +---------------------+---------+---------+-----------------+
 390 | onos-sfc            |  24/24  |  100%   | `joid-res-97`_  |
 391 +---------------------+---------+---------+-----------------+
 392
 393 It is highly recommended to install a json viewer in your browser
 394 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
 395
 396 You can get additional details through test logs on http://artifacts.opnfv.org/.
 397 As no search engine is available on the OPNFV artifact web site you must
 398 retrieve the pod identifier on which the tests have been executed (see
 399 field pod in any of the results) then click on the selected POD and look
 400 for the date of the test you are interested in.
 401
 402 The reporting pages can be found at:
 403
 404  * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
 405  * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
 406  * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
 407  * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html
 408
 409 Colorado known restrictions/issues
 410 ==================================
 411
 412 +-----------+-----------+----------------------------------------------+
 413 | Installer | Scenario  |  Issue                                       |
 414 +===========+===========+==============================================+
 415 | any       | odl_l3-*  | Tempest cases related to using floating IP   |
 416 |           |           | addresses fail because of a known ODL bug.   |
 417 |           |           | vPing_ssh test case is excluded for the same |
 418 |           |           | reason.                                      |
 419 |           |           | https://jira.opnfv.org/browse/APEX-112       |
 420 |           |           | https://jira.opnfv.org/browse/FUNCTEST-445   |
 421 +-----------+-----------+----------------------------------------------+
 422 | apex/fuel | *-bgpvpn  | vPing_ssh (floating ips not supported) and   |
 423 |           |           | vIMS excluded. Some Tempest cases related to |
 424 |           |           | floating ips also excluded. Some performance |
 425 |           |           | issues have been detected in this scenario   |
 426 |           |           | (i.e. BGPVPN extension enabled) when running |
 427 |           |           | commands against the OpenStack APIs, thus    |
 428 |           |           | Rally sanity test case has been disabled.    |
 429 |           |           | Performance issues seem to be connected to   |
 430 |           |           | the ODL version. It is planned to reintroduce|
 431 |           |           | Rally sanity in Colorado 2.0 with the        |
 432 |           |           | adoption of ODL Boron release.               |
 433 +-----------+-----------+----------------------------------------------+
 434 | apex      | *-fdio    | Due to late integration, fdio decided to     |
 435 |           |           | focus on mandatory tests and exclude feature |
 436 |           |           | tests (copper, doctor, security_scan) from   |
 437 |           |           | its scenarios                                |
 438 +-----------+-----------+----------------------------------------------+
 439 | compass   | moon      | First ODL test FAILS because ODL/Openstack   |
 440 |           |           | federation done in moon is partial. Only     |
 441 |           |           | MD-SAL is federated (not AD-SAL)             |
 442 +-----------+-----------+----------------------------------------------+
 443 | fuel      | any       | TestServerBasicOps test case skipped         |
 444 |           |           | https://gerrit.opnfv.org/gerrit/#/c/19635/   |
 445 +-----------+-----------+----------------------------------------------+
 446 | fuel      | kvm       | vPing_ssh and vIMS excluded (metadata related|
 447 |           |           | scenarios)                                   |
 448 +-----------+-----------+----------------------------------------------+
 449 | fuel      | multisite | random errors running multisite. A known bug |
 450 |           |           | in keystone mitaka, due to which memcache    |
 451 |           |           | raises exception and keystone becomes        |
 452 |           |           | unresponsive                                 |
 453 |           |           | bugs.launchpad.net/keystone/+bug/1600394     |
 454 |           |           | workaround consists in restarting memcache on|
 455 |           |           | server                                       |
 456 +-----------+-----------+----------------------------------------------+
 457 | joid      | any       | Tempest cases related to object storage      |
 458 |           |           | excluded                                     |
 459 |           |           | https://gerrit.opnfv.org/gerrit/#/c/17871/   |
 460 +-----------+-----------+----------------------------------------------+
 461 | joid      | domino    | Domino tests are skipped in CI. However the  |
 462 |           |           | test case can be run by manually setting     |
 463 |           |           | IS_IPandKEY_CONFIGURED=true after manually   |
 464 |           |           | setting properly the IP addresses of the 3   |
 465 |           |           | Controller nodes in the configuration file   |
 466 |           |           | /repos/domino/tests/run_multinode.sh         |
 467 +-----------+-----------+----------------------------------------------+
 468
 469
 470 Test and installer/scenario dependencies
 471 ========================================
 472
 473 It is not always possible to run all the test cases on all the scenarios.
 474 The following table details the dependencies of the test cases per
 475 scenario. The scenario dependencies (installer or scenario) are detailed
 476 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 477
 478 Test results
 479 ============
 480
 481 Test results are available in:
 482
 483  - test results document: http://artifacts.opnfv.org/functest
 484
 485  - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
 486
 487  - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
 488
 489
 490
 491 Open JIRA tickets
 492 =================
 493
 494 +------------------+-----------------------------------------------+
 495 |   JIRA           |         Description                           |
 496 +==================+===============================================+
 497 | `FUNCTEST-419`_  |  do not try to Remove docker                  |
 498 |                  |  image opnfv/functest:<none>                  |
 499 |                  |  reported by joid on Intel POD                |
 500 |                  |  may impact CI                                |
 501 |                  |  not reproducible                             |
 502 +------------------+-----------------------------------------------+
 503 | `FUNCTEST-446`_  |  Cleanup ODL-SFC output in Functest execution |
 504 |                  |  Impact on odl_l2-sfc scenarios               |
 505 +------------------+-----------------------------------------------+
 506 | `FUNCTEST-450`_  |  Functest is Failing to get the token using   |
 507 |                  |  keystone client                              |
 508 +------------------+-----------------------------------------------+
 509 | `FUNCTEST-454`_  |  Cleanup failures when using HA networks in   |
 510 |                  |  Neutron                                      |
 511 +------------------+-----------------------------------------------+
 512 | `FUNCTEST-460`_  |  Wrong image format used in rally cases       |
 513 +------------------+-----------------------------------------------+
 514 | `FUNCTEST-462`_  |  OLD test fails after forcing the clone       |
 515 |                  |  release/beryllium-sr3 branch                 |
 516 +------------------+-----------------------------------------------+
 517
 518
 519 Useful links
 520 ============
 521
 522  - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
 523
 524  - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado
 525
 526  - Functest repo: https://git.opnfv.org/cgit/functest
 527
 528  - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
 529
 530  - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
 531
 532  - Functest IRC chan: #opnfv-functest
 533
 534  - Functest reporting: http://testresults.opnfv.org/reporting
 535
 536  - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 537
 538  - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
 539
 540  - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
 541
 542  - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 543
 544 .. _`FUNCTEST-419` : https://jira.opnfv.org/browse/FUNCTEST-419
 545
 546 .. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
 547
 548 .. _`FUNCTEST-450` : https://jira.opnfv.org/browse/FUNCTEST-450
 549
 550 .. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
 551
 552 .. _`FUNCTEST-460` : https://jira.opnfv.org/browse/FUNCTEST-460
 553
 554 .. _`FUNCTEST-462` : https://jira.opnfv.org/browse/FUNCTEST-462
 555
 556 .. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
 557
 558 .. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
 559
 560 .. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
 561
 562 .. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
 563
 564 .. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
 565
 566 .. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
 567
 568 .. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
 569
 570 .. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
 571
 572 .. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
 573
 574 .. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
 575
 576 .. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
 577
 578 .. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
 579
 580 .. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
 581
 582 .. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
 583
 584 .. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
 585
 586 .. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
 587
 588 .. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
 589
 590 .. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
 591
 592 .. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
 593
 594 .. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
 595
 596 .. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
 597
 598 .. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
 599
 600 .. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
 601
 602 .. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
 603
 604 .. _`fuel-arm-res-29`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/29/console
 605
 606 .. _`fuel-arm-res-24`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/24/console
 607
 608 .. _`fuel-arm-res-30`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/30/console
 609
 610 .. _`fuel-arm-res-28`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/28/console
 611
 612 .. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
 613
 614 .. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
 615
 616 .. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
 617
 618 .. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
 619
 620 .. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
 621
 622 .. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
 623
 624 .. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103