docs/release-notes/functest-release.rst

   1 =======
   2 License
   3 =======
   4
   5 OPNFV Colorado release note for Functest Docs
   6 (c) by Morgan Richomme (Orange)
   7
   8 OPNFV Colorado release note for Functest Docs
   9 are licensed under a Creative Commons Attribution 4.0 International License.
  10 You should have received a copy of the license along with this.
  11 If not, see <http://creativecommons.org/licenses/by/4.0/>.
  12
  13 ===========================================
  14 OPNFV Colorado1.0 release note for Functest
  15 ===========================================
  16
  17 Abstract
  18 ========
  19
  20 This document describes the release note of Functest project.
  21
  22
  23 Version history
  24 ===============
  25
  26 +------------+----------+------------------+------------------------+
  27 | **Date**   | **Ver.** | **Author**       | **Comment**            |
  28 |            |          |                  |                        |
  29 +------------+----------+------------------+------------------------+
  30 | 2016-08-17 | 1.0.0    | Morgan Richomme  | Functest for Colorado  |
  31 |            |          | (Orange)         | release                |
  32 +------------+----------+------------------+------------------------+
  33
  34 OPNFV Colorado Release
  35 =========================
  36
  37 Functest deals with functional testing of the OPNFV solution.
  38 It includes test cases developed within the project and test cases developed in
  39 other OPNFV projects and other upstream communities.
  40
  41 The internal test cases are:
  42
  43  * healthcheck
  44  * vPing ssh
  45  * vPing userdata
  46  * Tempest Smoke Serial
  47  * Rally Sanity
  48  * ODL
  49  * Tempest full parallel
  50  * Rally full
  51  * vIMS
  52
  53 The OPNFV projects integrated into Functest framework for automation are:
  54
  55  * bgpvpn
  56  * Copper
  57  * Doctor
  58  * Domino
  59  * Moon
  60  * Multisite
  61  * ONOSFW
  62  * ONOS-sfc
  63  * ODL-sfc
  64  * Parser
  65  * Promise
  66  * Security scan
  67
  68 The validation of a scenario requires a subset of these tests depending
  69 on the installer and the scenario.
  70
  71 The 3 last internal test cases (tempest full parallel, Rally full and
  72 vIMS) are not considered for scenario validation.
  73
  74 Release Data
  75 ============
  76
  77 +--------------------------------------+--------------------------------------+
  78 | **Project**                          | functest                             |
  79 |                                      |                                      |
  80 +--------------------------------------+--------------------------------------+
  81 | **Repo/tag**                         | colorado.1.0                         |
  82 |                                      |                                      |
  83 +--------------------------------------+--------------------------------------+
  84 | **Release designation**              | Colorado base release                |
  85 |                                      |                                      |
  86 +--------------------------------------+--------------------------------------+
  87 | **Release date**                     | September 22 2016                    |
  88 |                                      |                                      |
  89 +--------------------------------------+--------------------------------------+
  90 | **Purpose of the delivery**          | Colorado base release                |
  91 |                                      |                                      |
  92 +--------------------------------------+--------------------------------------+
  93
  94 Deliverables
  95 ============
  96
  97 Software
  98 --------
  99
 100  - The Functest Docker image: * TODO link *
 101
 102  - The testapi Docker image: * TODO link *
 103
 104
 105 Documents
 106 ---------
 107
 108  - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
 109
 110  - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
 111
 112  - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 113
 114
 115 Version change
 116 ==============
 117
 118 Feature evolution
 119 -----------------
 120
 121  - refactoring of ODL functional tests (with upstream modifications)
 122
 123  - refactoring of testapi (update, swagger documentation, dockerization)
 124
 125  - jenkins logs improvement
 126
 127  - update integration of Doctor, Promise and SDNVPN  projects
 128
 129  - split Tempest and rally into 2 different tests: smoke and full
 130
 131  - vIMS test suite integration
 132
 133  - adoption of Kibana for dashboarding
 134
 135
 136 New features
 137 ------------
 138
 139  - Functest CLI to prepare and run the tests
 140
 141  - creation of the healthcheck test case
 142
 143  - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
 144
 145  - integration of new OPNFV feature projects (copper, domino, multisite,
 146  moon, parser, onos-sfc, odl-sfc, security scan)
 147
 148  - introduction of test tiers in functest framework
 149
 150  - automatic reporting
 151
 152  - introduction of a jenkins summary table
 153
 154  - support of ARM architecture
 155
 156
 157 Scenario Matrix
 158 ===============
 159
 160 For Colorado 1.0, Functest was tested on the following scenarios (if not
 161 precised, the scenario is a HA scenario):
 162
 163 +---------------------+---------+---------+---------+---------+
 164 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 165 +=====================+=========+=========+=========+=========+
 166 |   nosdn             |    X    |    X    |    X    |    X    |
 167 +---------------------+---------+---------+---------+---------+
 168 |   nosdn-noha        |         |         |    X    |    X    |
 169 +---------------------+---------+---------+---------+---------+
 170 |   odl_l2            |    X    |    X    |    X    |    X    |
 171 +---------------------+---------+---------+---------+---------+
 172 |   odl_l2-noha       |         |         |    X    |         |
 173 +---------------------+---------+---------+---------+---------+
 174 |   odl_l3            |    X    |    X    |    X    |         |
 175 +---------------------+---------+---------+---------+---------+
 176 |   odl_l3-noha       |         |         |    X    |         |
 177 +---------------------+---------+---------+---------+---------+
 178 |   odl_l2-bgpvpn     |    X    |         |    X    |         |
 179 +---------------------+---------+---------+---------+---------+
 180 |   odl_l2-bgpvpn-noha|         |         |    X    |         |
 181 +---------------------+---------+---------+---------+---------+
 182 |   odl_l2-fdio-noha  |    X    |         |         |         |
 183 +---------------------+---------+---------+---------+---------+
 184 |   odl_l2-moon       |         |    X    |         |         |
 185 +---------------------+---------+---------+---------+---------+
 186 |   odl_l2-sfc        |         |         |    X    |         |
 187 +---------------------+---------+---------+---------+---------+
 188 |   odl_l2-sfc-noha   |    X    |         |    X    |         |
 189 +---------------------+---------+---------+---------+---------+
 190 |   onos              |         |    X    |    X    |    X    |
 191 +---------------------+---------+---------+---------+---------+
 192 |   onos-noha         |         |         |    X    |         |
 193 +---------------------+---------+---------+---------+---------+
 194 |   onos-sfc          |         |    X    |    X    |    X    |
 195 +---------------------+---------+---------+---------+---------+
 196 |   onos-sfc-noha     |         |         |    X    |         |
 197 +---------------------+---------+---------+---------+---------+
 198 |   ovs-noha (dpdk)   |         |         |    X    |         |
 199 +---------------------+---------+---------+---------+---------+
 200 |   kvm               |         |         |    X    |         |
 201 +---------------------+---------+---------+---------+---------+
 202 |   kvm-noha          |         |         |    X    |         |
 203 +---------------------+---------+---------+---------+---------+
 204 |   multisite         |         |         |    X    |         |
 205 +---------------------+---------+---------+---------+---------+
 206 |   lxd               |         |         |         |    X    |
 207 +---------------------+---------+---------+---------+---------+
 208 |   lxd-noha          |         |         |         |    X    |
 209 +---------------------+---------+---------+---------+---------+
 210
 211 In Colorado, the functional tests have been sliced in 6 different
 212 categories:
 213
 214 +----------------+-----------------------------------------------+
 215 | Category       |  Description                                  |
 216 +================+===============================================+
 217 | healthcheck    | Basic OpenStack commands                      |
 218 +----------------+-----------------------------------------------+
 219 | smoke          | vPings, Tempest and rally smoke tests         |
 220 +----------------+-----------------------------------------------+
 221 | sdn_suites     | Specific SDN feature tests                    |
 222 +----------------+-----------------------------------------------+
 223 | features       | OPNFV feature project functional test suites  |
 224 +----------------+-----------------------------------------------+
 225 | openstack      | Advanced, long duration OpenStack tests       |
 226 |                | (Tempest and Rally full suite). Each test may |
 227 |                | last several hours                            |
 228 +----------------+-----------------------------------------------+
 229 | vnf            | Complex scenarios dealing with orchestration, |
 230 |                | VNF deployment and tests (e.g. vIMS)          |
 231 +----------------+-----------------------------------------------+
 232
 233 For the scenario validation, we consider only the healthcheck, smoke,
 234 sdn_suites and features categories. These tests are run systematically
 235 in the CI daily loops.
 236
 237 Success criteria have been defined for these test cases, they can be
 238 PASS/FAIL or a success rate may be declared (100%, > 90%)
 239 All the criteria, as well as the test dependencies are declared in the
 240 ci/testcases.yaml file.
 241
 242 The scoring for the Colorado release per installer can be described as
 243 follows.
 244
 245 The scoring is an indicator showing how many feature project test suites
 246 have been integrated on the scenario.
 247
 248 The scoring equals the number of tests * succesful iteration of each
 249 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
 250
 251  e.g.
 252  apex/odl_l2-nofeature-ha
 253  tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
 254  Scoring = 21/21 = 7 * 3
 255
 256 By default, if not specified, the scenarios are HA.
 257 HA means OpenStack High Availability (main services). Note that not
 258 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
 259 depends upon support of the specific service for HA deployment.
 260 For example, in the Colorado release, the Congress service
 261 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
 262 support for HA operation has not yet been verified.
 263 See the release notes of the installers for more details.
 264
 265
 266 Apex
 267 ----
 268
 269 +------------------+---------+---------+-------------------+
 270 |  Scenario        | Scoring | Success |    Results        |
 271 |                  |         | rate    |                   |
 272 +==================+=========+=========+===================+
 273 | nosdn            |  17/18  |   95%   | `apex-res-174`_   |
 274 +------------------+---------+---------+-------------------+
 275 | odl_l2           |  21/21  |   100%  | `apex-res-175`_   |
 276 +------------------+---------+---------+-------------------+
 277 | odl_l3           |  15/18  |    83%  | `apex-res-176`_   |
 278 +------------------+---------+---------+-------------------+
 279 | odl_l2-bgpvpn    |  14/18  |    78%  | `apex-res-235`_   |
 280 +------------------+---------+---------+-------------------+
 281 | odl_l2-fdio-noha |  12/15  |    80%  | `apex-res-6`_     |
 282 +------------------+---------+---------+-------------------+
 283 | odl_l2-sfc-noha  |  18/21  |    86%  | `apex-res-217`_   |
 284 +------------------+---------+---------+-------------------+
 285
 286 Compass
 287 -------
 288
 289 +------------------+---------+---------+-------------------+
 290 |  Scenario        | Scoring | Success |  Results          |
 291 |                  |         | rate    |                   |
 292 +==================+=========+=========+===================+
 293 | nosdn            |  12/12  |   100%  | `compass-res-55`_ |
 294 +------------------+---------+---------+-------------------+
 295 | odl_l2           |  15/15  |   100%  | `compass-res-59`_ |
 296 +------------------+---------+---------+-------------------+
 297 | odl_l3           |  9/12   |    75%  | `compass-res-73`_ |
 298 +------------------+---------+---------+-------------------+
 299 | odl_l2-moon      |  15/18  |    83%  | `compass-res-567`_|
 300 +------------------+---------+---------+-------------------+
 301 | onos-ha          |  15/15  |   100%  | `compass-res-77`_ |
 302 +------------------+---------+---------+-------------------+
 303 | onos-sfc-ha      |  16/18  |   100%  | `compass-res-76`_ |
 304 +------------------+---------+---------+-------------------+
 305
 306 Note: all the Compass tests for Colorado have been executed on virtual
 307 environment. Bare metal resources were used for Master branch.
 308
 309
 310 Fuel
 311 ----
 312
 313 +---------------------+---------+---------+-------------------+
 314 |  Scenario           | Scoring | Success |  Results          |
 315 |                     |         | rate    |                   |
 316 +=====================+=========+=========+===================+
 317 | nosdn               |  18/18  |  100%   | `fuel-res-129`_   |
 318 +---------------------+---------+---------+-------------------+
 319 | nosdn-noha          |  15/15  |  100%   | `fuel-res-154`_   |
 320 +---------------------+---------+---------+-------------------+
 321 | nosdn-kvm           |  18/18  |  100%   | `fuel-res-128`_   |
 322 +---------------------+---------+---------+-------------------+
 323 | nosdn-kvm-noha      |  15/15  |  100%   | `fuel-res-161`_   |
 324 +---------------------+---------+---------+-------------------+
 325 | nosdn-ovs-noha      |  15/15  |  100%   | `fuel-res-162`_   |
 326 +---------------------+---------+---------+-------------------+
 327 | odl_l2              |  21/21  |  100%   |  `fuel-res-123`_  |
 328 +---------------------+---------+---------+-------------------+
 329 | odl_l2-noha         |  17/18  |   94%   | `fuel-res-155`_   |
 330 +---------------------+---------+---------+-------------------+
 331 | odl_l2-bgpvpn       |  14/18  |   18%   | `fuel-res-119`_   |
 332 +---------------------+---------+---------+-------------------+
 333 | odl_l2-bgpvpn-noha  |  14/15  |   93%   | `fuel-res-160`_   |
 334 +---------------------+---------+---------+-------------------+
 335 | odl_l3              |  15/18  |   67%   | `fuel-res-115`_   |
 336 +---------------------+---------+---------+-------------------+
 337 | odl_l3-noha         |  12/15  |   80%   | `fuel-res-164`_   |
 338 +---------------------+---------+---------+-------------------+
 339 | onos                |  20/21  |   95%   | `fuel-res-117`_   |
 340 +---------------------+---------+---------+-------------------+
 341 | onos-noha           |  18/18  |  100%   | `fuel-res-166`_   |
 342 +---------------------+---------+---------+-------------------+
 343 | onos-sfc            |  24/24  |  100%   | `fuel-res-124`_   |
 344 +---------------------+---------+---------+-------------------+
 345 | onos-sfc-noha       |  21/21  |  100%   | `fuel-res-129`_   |
 346 +---------------------+---------+---------+-------------------+
 347 | multisite           |  N.R    |  100%   | `fuel-res-8`_     |
 348 +---------------------+---------+---------+-------------------+
 349
 350
 351 Results of Functest on AArch64 Colorado 1.0
 352
 353 +---------------------+---------+---------+----------------------+
 354 |  Scenario           | Scoring | Success |  Results             |
 355 |                     |         | rate    |                      |
 356 +=====================+=========+=========+======================+
 357 | nosdn               |  18/18  |  100%   | `fuel-arm-res-29`_   |
 358 +---------------------+---------+---------+----------------------+
 359 | odl_l2              |  21/21  |  100%   | `fuel-arm-res-24`_   |
 360 +---------------------+---------+---------+----------------------+
 361 | odl_l2-noha         |  17/18  |   94%   | `fuel-arm-res-30`_   |
 362 +---------------------+---------+---------+----------------------+
 363 | odl_l3              |  14/18  |   78%   | `fuel-arm-res-28`_   |
 364 +---------------------+---------+---------+----------------------+
 365
 366
 367 Joid
 368 ----
 369
 370 +---------------------+---------+---------+-----------------+
 371 |  Scenario           | Scoring | Success |  Results        |
 372 |                     |         | rate    |                 |
 373 +=====================+=========+=========+=================+
 374 | nosdn               |  18/18  |  100%   | `joid-res-102`_ |
 375 +---------------------+---------+---------+-----------------+
 376 | nosdn-noha          |  17/18  |   95%   | `joid-res-93`_  |
 377 +---------------------+---------+---------+-----------------+
 378 | nosdn-lxd           |  12/12  |  100%   | `joid-res-104`_ |
 379 +---------------------+---------+---------+-----------------+
 380 | nosdn-lxd-noha      |  12/12  |  100%   | `joid-res-91`_  |
 381 +---------------------+---------+---------+-----------------+
 382 | odl_l2              |  21/21  |  100%   | `joid-res-103`_ |
 383 +---------------------+---------+---------+-----------------+
 384 | onos                |  21/21  |  100%   | `joid-res-99`_  |
 385 +---------------------+---------+---------+-----------------+
 386 | onos-sfc            |  24/24  |  100%   | `joid-res-97`_  |
 387 +---------------------+---------+---------+-----------------+
 388
 389 It is highly recommended to install a json viewer in your browser
 390 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
 391
 392 You can get additional details through test logs on http://artifacts.opnfv.org/.
 393 As no search engine is available on the OPNFV artifact web site you must
 394 retrieve the pod identifier on which the tests have been executed (see
 395 field pod in any of the results) then click on the selected POD and look
 396 for the date of the test you are interested in.
 397
 398 The reporting pages can be found at:
 399
 400  * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
 401  * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
 402  * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
 403  * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html
 404
 405 Colorado known restrictions/issues
 406 ==================================
 407
 408 +-----------+-----------+----------------------------------------------+
 409 | Installer | Scenario  |  Issue                                       |
 410 +===========+===========+==============================================+
 411 | any       | odl_l3-*  | Tempest cases related to using floating IP   |
 412 |           |           | addresses fail because of a known ODL bug.   |
 413 |           |           | vPing_ssh test case is excluded for the same |
 414 |           |           | reason.                                      |
 415 |           |           | https://jira.opnfv.org/browse/APEX-112       |
 416 |           |           | https://jira.opnfv.org/browse/FUNCTEST-445   |
 417 +-----------+-----------+----------------------------------------------+
 418 | apex/fuel | *-bgpvpn  | vPing_ssh (floating ips not supported) and   |
 419 |           |           | vIMS excluded. Some Tempest cases related to |
 420 |           |           | floating ips also excluded. Some performance |
 421 |           |           | issues have been detected in this scenario   |
 422 |           |           | (i.e. BGPVPN extension enabled) when running |
 423 |           |           | commands against the OpenStack APIs, thus    |
 424 |           |           | Rally sanity test case has been disabled.    |
 425 |           |           | Performance issues seem to be connected to   |
 426 |           |           | the ODL version. It is planned to reintroduce|
 427 |           |           | Rally sanity in Colorado 2.0 with the        |
 428 |           |           | adoption of ODL Boron release.               |
 429 +-----------+-----------+----------------------------------------------+
 430 | apex      | *-fdio    | Due to late integration, fdio decided to     |
 431 |           |           | focus on mandatory tests and exclude feature |
 432 |           |           | tests (copper, doctor, security_scan) from   |
 433 |           |           | its scenarios                                |
 434 +-----------+-----------+----------------------------------------------+
 435 | compass   | moon      | First ODL test FAILS because ODL/Openstack   |
 436 |           |           | federation done in moon is partial. Only     |
 437 |           |           | MD-SAL is federated (not AD-SAL)             |
 438 +-----------+-----------+----------------------------------------------+
 439 | fuel      | any       | TestServerBasicOps test case skipped         |
 440 |           |           | https://gerrit.opnfv.org/gerrit/#/c/19635/   |
 441 +-----------+-----------+----------------------------------------------+
 442 | fuel      | kvm       | vPing_ssh and vIMS excluded (metadata related|
 443 |           |           | scenarios)                                   |
 444 +-----------+-----------+----------------------------------------------+
 445 | fuel      | multisite | random errors running multisite. A known bug |
 446 |           |           | in keystone mitaka, due to which memcache    |
 447 |           |           | raises exception and keystone becomes        |
 448 |           |           | unresponsive                                 |
 449 |           |           | bugs.launchpad.net/keystone/+bug/1600394     |
 450 |           |           | workaround consists in restarting memcache on|
 451 |           |           | server                                       |
 452 +-----------+-----------+----------------------------------------------+
 453 | joid      | any       | Tempest cases related to object storage      |
 454 |           |           | excluded                                     |
 455 |           |           | https://gerrit.opnfv.org/gerrit/#/c/17871/   |
 456 +-----------+-----------+----------------------------------------------+
 457 | joid      | domino    | Domino tests are skipped in CI. However the  |
 458 |           |           | test case can be run by manually setting     |
 459 |           |           | IS_IPandKEY_CONFIGURED=true after manually   |
 460 |           |           | setting properly the IP addresses of the 3   |
 461 |           |           | Controller nodes in the configuration file   |
 462 |           |           | /repos/domino/tests/run_multinode.sh         |
 463 +-----------+-----------+----------------------------------------------+
 464
 465
 466 Test and installer/scenario dependencies
 467 ========================================
 468
 469 It is not always possible to run all the test cases on all the scenarios.
 470 The following table details the dependencies of the test cases per
 471 scenario. The scenario dependencies (installer or scenario) are detailed
 472 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 473
 474 Test results
 475 ============
 476
 477 Test results are available in:
 478
 479  - test results document: http://artifacts.opnfv.org/functest
 480
 481  - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
 482
 483  - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
 484
 485
 486
 487 Open JIRA tickets
 488 =================
 489
 490 +------------------+-----------------------------------------------+
 491 |   JIRA           |         Description                           |
 492 +==================+===============================================+
 493 | `FUNCTEST-446`_  |  Cleanup ODL-SFC output in Functest execution |
 494 |                  |  Impact on odl_l2-sfc scenarios               |
 495 +------------------+-----------------------------------------------+
 496 | `FUNCTEST-454`_  |  Cleanup failures when using HA networks in   |
 497 |                  |  Neutron                                      |
 498 +------------------+-----------------------------------------------+
 499
 500 Useful links
 501 ============
 502
 503  - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
 504
 505  - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado
 506
 507  - Functest repo: https://git.opnfv.org/cgit/functest
 508
 509  - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
 510
 511  - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
 512
 513  - Functest IRC chan: #opnfv-functest
 514
 515  - Functest reporting: http://testresults.opnfv.org/reporting
 516
 517  - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 518
 519  - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
 520
 521  - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
 522
 523  - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 524
 525 .. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
 526
 527 .. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
 528
 529 .. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
 530
 531 .. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
 532
 533 .. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
 534
 535 .. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
 536
 537 .. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
 538
 539 .. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
 540
 541 .. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
 542
 543 .. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
 544
 545 .. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
 546
 547 .. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
 548
 549 .. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
 550
 551 .. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
 552
 553 .. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
 554
 555 .. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
 556
 557 .. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
 558
 559 .. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
 560
 561 .. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
 562
 563 .. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
 564
 565 .. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
 566
 567 .. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
 568
 569 .. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
 570
 571 .. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
 572
 573 .. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
 574
 575 .. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
 576
 577 .. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
 578
 579 .. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
 580
 581 .. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
 582
 583 .. _`fuel-arm-res-29`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/29/console
 584
 585 .. _`fuel-arm-res-24`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/24/console
 586
 587 .. _`fuel-arm-res-30`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/30/console
 588
 589 .. _`fuel-arm-res-28`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/28/console
 590
 591 .. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
 592
 593 .. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
 594
 595 .. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
 596
 597 .. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
 598
 599 .. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
 600
 601 .. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
 602
 603 .. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103