docs/release-notes/functest-release.rst

   1 =======
   2 License
   3 =======
   4
   5 OPNFV Colorado release note for Functest Docs
   6 (c) by Morgan Richomme (Orange)
   7
   8 OPNFV Colorado release note for Functest Docs
   9 are licensed under a Creative Commons Attribution 4.0 International License.
  10 You should have received a copy of the license along with this.
  11 If not, see <http://creativecommons.org/licenses/by/4.0/>.
  12
  13 ===========================================
  14 OPNFV Colorado1.0 release note for Functest
  15 ===========================================
  16
  17 Abstract
  18 ========
  19
  20 This document describes the release note of Functest project.
  21
  22
  23 Version history
  24 ===============
  25
  26 +------------+----------+------------------+------------------------+
  27 | **Date**   | **Ver.** | **Author**       | **Comment**            |
  28 |            |          |                  |                        |
  29 +------------+----------+------------------+------------------------+
  30 | 2016-08-17 | 1.0.0    | Morgan Richomme  | Functest for Colorado  |
  31 |            |          | (Orange)         | release                |
  32 +------------+----------+------------------+------------------------+
  33
  34 OPNFV Colorado Release
  35 =========================
  36
  37 Functest deals with functional testing of the OPNFV solution.
  38 It includes test cases developed within the project and test cases developed in
  39 other OPNFV projects and other upstream communities.
  40
  41 The internal test cases are:
  42
  43  * healthcheck
  44  * vPing ssh
  45  * vPing userdata
  46  * Tempest Smoke Serial
  47  * Rally Sanity
  48  * ODL
  49  * Tempest full parallel
  50  * Rally full
  51  * vIMS
  52
  53 The OPNFV projects integrated into Functest framework for automation are:
  54
  55  * bgpvpn
  56  * Copper
  57  * Doctor
  58  * Domino
  59  * Moon
  60  * Multisite
  61  * ONOSFW
  62  * ONOS-sfc
  63  * ODL-sfc
  64  * Parser
  65  * Promise
  66  * Security scan
  67
  68 The validation of a scenario requires a subset of these tests depending
  69 on the installer and the scenario.
  70
  71 The 3 last internal test cases (tempest full parallel, Rally full and
  72 vIMS) are not considered for scenario validation.
  73
  74 Release Data
  75 ============
  76
  77 +--------------------------------------+--------------------------------------+
  78 | **Project**                          | functest                             |
  79 |                                      |                                      |
  80 +--------------------------------------+--------------------------------------+
  81 | **Repo/tag**                         | colorado.1.0                         |
  82 |                                      |                                      |
  83 +--------------------------------------+--------------------------------------+
  84 | **Release designation**              | Colorado base release                |
  85 |                                      |                                      |
  86 +--------------------------------------+--------------------------------------+
  87 | **Release date**                     | September 22 2016                    |
  88 |                                      |                                      |
  89 +--------------------------------------+--------------------------------------+
  90 | **Purpose of the delivery**          | Colorado base release                |
  91 |                                      |                                      |
  92 +--------------------------------------+--------------------------------------+
  93
  94 Deliverables
  95 ============
  96
  97 Software
  98 --------
  99
 100  - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0)
 101
 102  - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0)
 103
 104
 105 Documents
 106 ---------
 107
 108  - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
 109
 110  - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
 111
 112  - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 113
 114
 115 Version change
 116 ==============
 117
 118 Feature evolution
 119 -----------------
 120
 121  - refactoring of ODL functional tests (with upstream modifications)
 122
 123  - refactoring of testapi (update, swagger documentation, dockerization)
 124
 125  - jenkins logs improvement
 126
 127  - update integration of Doctor, Promise and SDNVPN  projects
 128
 129  - split Tempest and rally into 2 different tests: smoke and full
 130
 131  - vIMS test suite integration
 132
 133  - adoption of Kibana for dashboarding
 134
 135
 136 New features
 137 ------------
 138
 139  - Functest CLI to prepare and run the tests
 140
 141  - creation of the healthcheck test case
 142
 143  - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
 144
 145  - integration of new OPNFV feature projects (copper, domino, multisite,
 146  moon, parser, onos-sfc, odl-sfc, security scan)
 147
 148  - introduction of test tiers in functest framework
 149
 150  - automatic reporting
 151
 152  - introduction of a jenkins summary table
 153
 154  - support of ARM architecture
 155
 156
 157 Scenario Matrix
 158 ===============
 159
 160 For Colorado 1.0, Functest was tested on the following scenarios (if not
 161 precised, the scenario is a HA scenario):
 162
 163 +---------------------+---------+---------+---------+---------+
 164 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 165 +=====================+=========+=========+=========+=========+
 166 |   nosdn             |    X    |    X    |    X    |    X    |
 167 +---------------------+---------+---------+---------+---------+
 168 |   nosdn-noha        |         |         |    X    |    X    |
 169 +---------------------+---------+---------+---------+---------+
 170 |   kvm               |         |         |    X    |         |
 171 +---------------------+---------+---------+---------+---------+
 172 |   kvm-noha          |         |         |    X    |         |
 173 +---------------------+---------+---------+---------+---------+
 174 |   lxd               |         |         |         |    X    |
 175 +---------------------+---------+---------+---------+---------+
 176 |   lxd-noha          |         |         |         |    X    |
 177 +---------------------+---------+---------+---------+---------+
 178 |   ovs-noha (dpdk)   |         |         |    X    |         |
 179 +---------------------+---------+---------+---------+---------+
 180 |   odl_l2            |    X    |    X    |    X    |    X    |
 181 +---------------------+---------+---------+---------+---------+
 182 |   odl_l2-noha       |         |         |    X    |         |
 183 +---------------------+---------+---------+---------+---------+
 184 |   odl_l2-bgpvpn     |    X    |         |    X    |         |
 185 +---------------------+---------+---------+---------+---------+
 186 |   odl_l2-bgpvpn-noha|         |         |    X    |         |
 187 +---------------------+---------+---------+---------+---------+
 188 |   odl_l2-fdio-noha  |    X    |         |         |         |
 189 +---------------------+---------+---------+---------+---------+
 190 |   odl_l2-moon       |         |    X    |         |         |
 191 +---------------------+---------+---------+---------+---------+
 192 |   odl_l2-sfc-noha   |    X    |         |    X    |         |
 193 +---------------------+---------+---------+---------+---------+
 194 |   odl_l3            |    X    |    X    |    X    |         |
 195 +---------------------+---------+---------+---------+---------+
 196 |   odl_l3-noha       |         |         |    X    |         |
 197 +---------------------+---------+---------+---------+---------+
 198 |   onos              |         |    X    |    X    |    X    |
 199 +---------------------+---------+---------+---------+---------+
 200 |   onos-noha         |         |         |    X    |         |
 201 +---------------------+---------+---------+---------+---------+
 202 |   onos-sfc          |         |    X    |    X    |    X    |
 203 +---------------------+---------+---------+---------+---------+
 204 |   onos-sfc-noha     |         |         |    X    |         |
 205 +---------------------+---------+---------+---------+---------+
 206 |   multisite         |         |         |    X    |         |
 207 +---------------------+---------+---------+---------+---------+
 208
 209 In Colorado, the functional tests have been sliced in 6 different
 210 categories:
 211
 212 +----------------+-----------------------------------------------+
 213 | Category       |  Description                                  |
 214 +================+===============================================+
 215 | healthcheck    | Basic OpenStack commands                      |
 216 +----------------+-----------------------------------------------+
 217 | smoke          | vPings, Tempest and rally smoke tests         |
 218 +----------------+-----------------------------------------------+
 219 | sdn_suites     | Specific SDN feature tests                    |
 220 +----------------+-----------------------------------------------+
 221 | features       | OPNFV feature project functional test suites  |
 222 +----------------+-----------------------------------------------+
 223 | openstack      | Advanced, long duration OpenStack tests       |
 224 |                | (Tempest and Rally full suite). Each test may |
 225 |                | last several hours                            |
 226 +----------------+-----------------------------------------------+
 227 | vnf            | Complex scenarios dealing with orchestration, |
 228 |                | VNF deployment and tests (e.g. vIMS)          |
 229 +----------------+-----------------------------------------------+
 230
 231 For the scenario validation, we consider only the healthcheck, smoke,
 232 sdn_suites and features categories. These tests are run systematically
 233 in the CI daily loops.
 234
 235 Success criteria have been defined for these test cases, they can be
 236 PASS/FAIL or a success rate may be declared (100%, > 90%)
 237 All the criteria, as well as the test dependencies are declared in the
 238 ci/testcases.yaml file.
 239
 240 The scoring for the Colorado release per installer can be described as
 241 follows.
 242
 243 The scoring is an indicator showing how many feature project test suites
 244 have been integrated on the scenario.
 245
 246 The scoring equals the number of tests * succesful iteration of each
 247 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
 248
 249  e.g.
 250  apex/odl_l2-nofeature-ha
 251  tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
 252  Scoring = 21/21 = 7 * 3
 253
 254 By default, if not specified, the scenarios are HA.
 255 HA means OpenStack High Availability (main services). Note that not
 256 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
 257 depends upon support of the specific service for HA deployment.
 258 For example, in the Colorado release, the Congress service
 259 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
 260 support for HA operation has not yet been verified.
 261 See the release notes of the installers for more details.
 262
 263
 264 Apex
 265 ----
 266
 267 +------------------+---------+---------+-------------------+
 268 |  Scenario        | Scoring | Success |    Results        |
 269 |                  |         | rate    |                   |
 270 +==================+=========+=========+===================+
 271 | nosdn            |  17/18  |   95%   | `apex-res-174`_   |
 272 +------------------+---------+---------+-------------------+
 273 | odl_l2           |  21/21  |   100%  | `apex-res-175`_   |
 274 +------------------+---------+---------+-------------------+
 275 | odl_l3           |  15/18  |    83%  | `apex-res-176`_   |
 276 +------------------+---------+---------+-------------------+
 277 | odl_l2-bgpvpn    |  14/18  |    78%  | `apex-res-235`_   |
 278 +------------------+---------+---------+-------------------+
 279 | odl_l2-fdio-noha |  12/15  |    80%  | `apex-res-6`_     |
 280 +------------------+---------+---------+-------------------+
 281 | odl_l2-sfc-noha  |  18/24  |    75%  | `apex-res-217`_   |
 282 +------------------+---------+---------+-------------------+
 283
 284 Compass
 285 -------
 286
 287 +------------------+---------+---------+-------------------+
 288 |  Scenario        | Scoring | Success |  Results          |
 289 |                  |         | rate    |                   |
 290 +==================+=========+=========+===================+
 291 | nosdn            |  12/12  |   100%  | `compass-res-55`_ |
 292 +------------------+---------+---------+-------------------+
 293 | odl_l2           |  15/15  |   100%  | `compass-res-59`_ |
 294 +------------------+---------+---------+-------------------+
 295 | odl_l3           |  9/12   |    75%  | `compass-res-73`_ |
 296 +------------------+---------+---------+-------------------+
 297 | odl_l2-moon      |  15/18  |    83%  | `compass-res-567`_|
 298 +------------------+---------+---------+-------------------+
 299 | onos-ha          |  15/15  |   100%  | `compass-res-77`_ |
 300 +------------------+---------+---------+-------------------+
 301 | onos-sfc-ha      |  17/18  |    95%  | `compass-res-76`_ |
 302 +------------------+---------+---------+-------------------+
 303
 304 Note: all the Compass tests for Colorado have been executed on virtual
 305 environment. Bare metal resources were used for Master branch.
 306
 307
 308 Fuel
 309 ----
 310
 311 +---------------------+---------+---------+-------------------+
 312 |  Scenario           | Scoring | Success |  Results          |
 313 |                     |         | rate    |                   |
 314 +=====================+=========+=========+===================+
 315 | nosdn               |  18/18  |  100%   | `fuel-res-129`_   |
 316 +---------------------+---------+---------+-------------------+
 317 | nosdn-noha          |  15/15  |  100%   | `fuel-res-154`_   |
 318 +---------------------+---------+---------+-------------------+
 319 | nosdn-kvm           |  18/18  |  100%   | `fuel-res-128`_   |
 320 +---------------------+---------+---------+-------------------+
 321 | nosdn-kvm-noha      |  15/15  |  100%   | `fuel-res-161`_   |
 322 +---------------------+---------+---------+-------------------+
 323 | nosdn-ovs           |  12/18  |   67%*  |  `fuel-res-213`_  |
 324 +---------------------+---------+---------+-------------------+
 325 | nosdn-ovs-noha      |  15/15  |  100%   | `fuel-res-162`_   |
 326 +---------------------+---------+---------+-------------------+
 327 | odl_l2              |  21/21  |  100%   |  `fuel-res-123`_  |
 328 +---------------------+---------+---------+-------------------+
 329 | odl_l2-noha         |  17/18  |   94%   | `fuel-res-155`_   |
 330 +---------------------+---------+---------+-------------------+
 331 | odl_l2-bgpvpn       |  14/18  |   78%   | `fuel-res-119`_   |
 332 +---------------------+---------+---------+-------------------+
 333 | odl_l2-bgpvpn-noha  |  14/15  |   93%   | `fuel-res-160`_   |
 334 +---------------------+---------+---------+-------------------+
 335 | odl_l2-sfc-noha     |   6/21  |   29%   | `fuel-res-219`_   |
 336 +---------------------+---------+---------+-------------------+
 337 | odl_l3              |  15/18  |   83%   | `fuel-res-115`_   |
 338 +---------------------+---------+---------+-------------------+
 339 | odl_l3-noha         |  12/15  |   80%   | `fuel-res-164`_   |
 340 +---------------------+---------+---------+-------------------+
 341 | onos                |  20/21  |   95%   | `fuel-res-117`_   |
 342 +---------------------+---------+---------+-------------------+
 343 | onos-noha           |  18/18  |  100%   | `fuel-res-166`_   |
 344 +---------------------+---------+---------+-------------------+
 345 | onos-sfc            |  24/24  |  100%   | `fuel-res-124`_   |
 346 +---------------------+---------+---------+-------------------+
 347 | onos-sfc-noha       |  21/21  |  100%   | `fuel-res-129`_   |
 348 +---------------------+---------+---------+-------------------+
 349 | multisite           |  N.R    |  100%   | `fuel-res-8`_     |
 350 +---------------------+---------+---------+-------------------+
 351
 352 *: all results passed, lacking iterations to reach the full score
 353
 354 Results of Functest on AArch64 Colorado 1.0
 355
 356 +---------------------+---------+---------+----------------------+
 357 |  Scenario           | Scoring | Success |  Results             |
 358 |                     |         | rate    |                      |
 359 +=====================+=========+=========+======================+
 360 | nosdn               |  18/18  |  100%   | `fuel-arm-res-29`_   |
 361 +---------------------+---------+---------+----------------------+
 362 | odl_l2              |  21/21  |  100%   | `fuel-arm-res-24`_   |
 363 +---------------------+---------+---------+----------------------+
 364 | odl_l2-noha         |  17/18  |   94%   | `fuel-arm-res-30`_   |
 365 +---------------------+---------+---------+----------------------+
 366 | odl_l3              |  14/18  |   78%   | `fuel-arm-res-28`_   |
 367 +---------------------+---------+---------+----------------------+
 368
 369
 370 Joid
 371 ----
 372
 373 +---------------------+---------+---------+-----------------+
 374 |  Scenario           | Scoring | Success |  Results        |
 375 |                     |         | rate    |                 |
 376 +=====================+=========+=========+=================+
 377 | nosdn               |  18/18  |  100%   | `joid-res-102`_ |
 378 +---------------------+---------+---------+-----------------+
 379 | nosdn-noha          |  17/18  |   95%   | `joid-res-93`_  |
 380 +---------------------+---------+---------+-----------------+
 381 | nosdn-lxd           |  12/12  |  100%   | `joid-res-104`_ |
 382 +---------------------+---------+---------+-----------------+
 383 | nosdn-lxd-noha      |  12/12  |  100%   | `joid-res-91`_  |
 384 +---------------------+---------+---------+-----------------+
 385 | odl_l2              |  21/21  |  100%   | `joid-res-103`_ |
 386 +---------------------+---------+---------+-----------------+
 387 | onos                |  21/21  |  100%   | `joid-res-99`_  |
 388 +---------------------+---------+---------+-----------------+
 389 | onos-sfc            |  24/24  |  100%   | `joid-res-97`_  |
 390 +---------------------+---------+---------+-----------------+
 391
 392 It is highly recommended to install a json viewer in your browser
 393 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
 394
 395 You can get additional details through test logs on http://artifacts.opnfv.org/.
 396 As no search engine is available on the OPNFV artifact web site you must
 397 retrieve the pod identifier on which the tests have been executed (see
 398 field pod in any of the results) then click on the selected POD and look
 399 for the date of the test you are interested in.
 400
 401 The reporting pages can be found at:
 402
 403  * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
 404  * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
 405  * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
 406  * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html
 407
 408 Colorado known restrictions/issues
 409 ==================================
 410
 411 +-----------+-----------+----------------------------------------------+
 412 | Installer | Scenario  |  Issue                                       |
 413 +===========+===========+==============================================+
 414 | any       | odl_l3-*  | Tempest cases related to using floating IP   |
 415 |           |           | addresses fail because of a known ODL bug.   |
 416 |           |           | vPing_ssh test case is excluded for the same |
 417 |           |           | reason.                                      |
 418 |           |           | https://jira.opnfv.org/browse/APEX-112       |
 419 |           |           | https://jira.opnfv.org/browse/FUNCTEST-445   |
 420 +-----------+-----------+----------------------------------------------+
 421 | apex/fuel | *-bgpvpn  | vPing_ssh (floating ips not supported) and   |
 422 |           |           | vIMS excluded. Some Tempest cases related to |
 423 |           |           | floating ips also excluded. Some performance |
 424 |           |           | issues have been detected in this scenario   |
 425 |           |           | (i.e. BGPVPN extension enabled) when running |
 426 |           |           | commands against the OpenStack APIs, thus    |
 427 |           |           | Rally sanity test case has been disabled.    |
 428 |           |           | Performance issues seem to be connected to   |
 429 |           |           | the ODL version. It is planned to reintroduce|
 430 |           |           | Rally sanity in Colorado 2.0 with the        |
 431 |           |           | adoption of ODL Boron release.               |
 432 +-----------+-----------+----------------------------------------------+
 433 | apex      | *-fdio    | Due to late integration, fdio decided to     |
 434 |           |           | focus on mandatory tests and exclude feature |
 435 |           |           | tests (copper, doctor, security_scan) from   |
 436 |           |           | its scenarios                                |
 437 +-----------+-----------+----------------------------------------------+
 438 | compass   | moon      | First ODL test FAILS because ODL/Openstack   |
 439 |           |           | federation done in moon is partial. Only     |
 440 |           |           | MD-SAL is federated (not AD-SAL)             |
 441 +-----------+-----------+----------------------------------------------+
 442 | fuel      | any       | TestServerBasicOps test case skipped         |
 443 |           |           | https://gerrit.opnfv.org/gerrit/#/c/19635/   |
 444 +-----------+-----------+----------------------------------------------+
 445 | fuel      | kvm       | vPing_ssh and vIMS excluded (metadata related|
 446 |           |           | scenarios)                                   |
 447 +-----------+-----------+----------------------------------------------+
 448 | fuel      | multisite | random errors running multisite. A known bug |
 449 |           |           | in keystone mitaka, due to which memcache    |
 450 |           |           | raises exception and keystone becomes        |
 451 |           |           | unresponsive                                 |
 452 |           |           | bugs.launchpad.net/keystone/+bug/1600394     |
 453 |           |           | workaround consists in restarting memcache on|
 454 |           |           | server                                       |
 455 +-----------+-----------+----------------------------------------------+
 456 | joid      | any       | Tempest cases related to object storage      |
 457 |           |           | excluded                                     |
 458 |           |           | https://gerrit.opnfv.org/gerrit/#/c/17871/   |
 459 +-----------+-----------+----------------------------------------------+
 460 | joid      | domino    | Domino tests are skipped in CI. However the  |
 461 |           |           | test case can be run by manually setting     |
 462 |           |           | IS_IPandKEY_CONFIGURED=true after manually   |
 463 |           |           | setting properly the IP addresses of the 3   |
 464 |           |           | Controller nodes in the configuration file   |
 465 |           |           | /repos/domino/tests/run_multinode.sh         |
 466 +-----------+-----------+----------------------------------------------+
 467
 468
 469 Test and installer/scenario dependencies
 470 ========================================
 471
 472 It is not always possible to run all the test cases on all the scenarios.
 473 The following table details the dependencies of the test cases per
 474 scenario. The scenario dependencies (installer or scenario) are detailed
 475 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 476
 477 Test results
 478 ============
 479
 480 Test results are available in:
 481
 482  - test results document: http://artifacts.opnfv.org/functest
 483
 484  - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
 485
 486  - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
 487
 488
 489
 490 Open JIRA tickets
 491 =================
 492
 493 +------------------+-----------------------------------------------+
 494 |   JIRA           |         Description                           |
 495 +==================+===============================================+
 496 | `FUNCTEST-446`_  |  Cleanup ODL-SFC output in Functest execution |
 497 |                  |  Impact on odl_l2-sfc scenarios               |
 498 +------------------+-----------------------------------------------+
 499 | `FUNCTEST-454`_  |  Cleanup failures when using HA networks in   |
 500 |                  |  Neutron                                      |
 501 +------------------+-----------------------------------------------+
 502
 503 Useful links
 504 ============
 505
 506  - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
 507
 508  - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado
 509
 510  - Functest repo: https://git.opnfv.org/cgit/functest
 511
 512  - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
 513
 514  - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
 515
 516  - Functest IRC chan: #opnfv-functest
 517
 518  - Functest reporting: http://testresults.opnfv.org/reporting
 519
 520  - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 521
 522  - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
 523
 524  - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
 525
 526  - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 527
 528 .. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
 529
 530 .. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
 531
 532 .. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
 533
 534 .. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
 535
 536 .. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
 537
 538 .. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
 539
 540 .. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
 541
 542 .. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
 543
 544 .. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
 545
 546 .. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
 547
 548 .. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
 549
 550 .. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
 551
 552 .. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
 553
 554 .. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
 555
 556 .. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
 557
 558 .. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
 559
 560 .. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
 561
 562 .. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
 563
 564 .. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
 565
 566 .. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
 567
 568 .. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
 569
 570 .. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
 571
 572 .. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
 573
 574 .. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
 575
 576 .. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
 577
 578 .. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
 579
 580 .. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
 581
 582 .. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
 583
 584 .. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
 585
 586 .. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213
 587
 588 .. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219
 589
 590 .. _`fuel-arm-res-29`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/29/console
 591
 592 .. _`fuel-arm-res-24`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/24/console
 593
 594 .. _`fuel-arm-res-30`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/30/console
 595
 596 .. _`fuel-arm-res-28`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/28/console
 597
 598 .. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
 599
 600 .. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
 601
 602 .. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
 603
 604 .. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
 605
 606 .. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
 607
 608 .. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
 609
 610 .. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104