docs/release/release-notes/functest-release.rst

   1 =======
   2 License
   3 =======
   4
   5 OPNFV Colorado release note for Functest Docs
   6 (c) by Morgan Richomme (Orange)
   7
   8 OPNFV Colorado release note for Functest Docs
   9 are licensed under a Creative Commons Attribution 4.0 International License.
  10 You should have received a copy of the license along with this.
  11 If not, see <http://creativecommons.org/licenses/by/4.0/>.
  12
  13 ===========================================
  14 OPNFV Colorado1.0 release note for Functest
  15 ===========================================
  16
  17 Abstract
  18 ========
  19
  20 This document describes the release note of Functest project.
  21
  22
  23 Version history
  24 ===============
  25
  26 +------------+----------+------------------+------------------------+
  27 | **Date**   | **Ver.** | **Author**       | **Comment**            |
  28 |            |          |                  |                        |
  29 +------------+----------+------------------+------------------------+
  30 | 2016-08-17 | 1.0.0    | Morgan Richomme  | Functest for Colorado  |
  31 |            |          | (Orange)         | release                |
  32 +------------+----------+------------------+------------------------+
  33
  34 OPNFV Colorado Release
  35 =========================
  36
  37 Functest deals with functional testing of the OPNFV solution.
  38 It includes test cases developed within the project and test cases developed in
  39 other OPNFV projects and other upstream communities.
  40
  41 The internal test cases are:
  42
  43  * healthcheck
  44  * vPing ssh
  45  * vPing userdata
  46  * Tempest Smoke Serial
  47  * Rally Sanity
  48  * ODL
  49  * Tempest full parallel
  50  * Rally full
  51  * vIMS
  52
  53 The OPNFV projects integrated into Functest framework for automation are:
  54
  55  * bgpvpn
  56  * Copper
  57  * Doctor
  58  * Domino
  59  * Moon
  60  * Multisite
  61  * ONOSFW
  62  * ONOS-sfc
  63  * ODL-sfc
  64  * Parser
  65  * Promise
  66  * Security scan
  67
  68 The validation of a scenario requires a subset of these tests depending
  69 on the installer and the scenario.
  70
  71 The 3 last internal test cases (tempest full parallel, Rally full and
  72 vIMS) are not considered for scenario validation.
  73
  74 Release Data
  75 ============
  76
  77 +--------------------------------------+--------------------------------------+
  78 | **Project**                          | functest                             |
  79 |                                      |                                      |
  80 +--------------------------------------+--------------------------------------+
  81 | **Repo/tag**                         | colorado.1.0                         |
  82 |                                      |                                      |
  83 +--------------------------------------+--------------------------------------+
  84 | **Release designation**              | Colorado base release                |
  85 |                                      |                                      |
  86 +--------------------------------------+--------------------------------------+
  87 | **Release date**                     | September 22 2016                    |
  88 |                                      |                                      |
  89 +--------------------------------------+--------------------------------------+
  90 | **Purpose of the delivery**          | Colorado base release                |
  91 |                                      |                                      |
  92 +--------------------------------------+--------------------------------------+
  93
  94 Deliverables
  95 ============
  96
  97 Software
  98 --------
  99
 100  - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0)
 101
 102  - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0)
 103
 104
 105 Documents
 106 ---------
 107
 108  - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
 109
 110  - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
 111
 112  - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 113
 114
 115 Version change
 116 ==============
 117
 118 Feature evolution
 119 -----------------
 120
 121  - refactoring of ODL functional tests (with upstream modifications)
 122
 123  - refactoring of testapi (update, swagger documentation, dockerization)
 124
 125  - jenkins logs improvement
 126
 127  - update integration of Doctor, Promise and SDNVPN  projects
 128
 129  - split Tempest and rally into 2 different tests: smoke and full
 130
 131  - vIMS test suite integration
 132
 133  - adoption of Kibana for dashboarding
 134
 135
 136 New features
 137 ------------
 138
 139  - Functest CLI to prepare and run the tests
 140
 141  - creation of the healthcheck test case
 142
 143  - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
 144
 145  - integration of new OPNFV feature projects (copper, domino, multisite,
 146  moon, parser, onos-sfc, odl-sfc, security scan)
 147
 148  - introduction of test tiers in functest framework
 149
 150  - automatic reporting
 151
 152  - introduction of a jenkins summary table
 153
 154  - support of ARM architecture
 155
 156
 157 Scenario Matrix
 158 ===============
 159
 160 For Colorado 1.0, Functest was tested on the following scenarios (if not
 161 precised, the scenario is a HA scenario):
 162
 163 +---------------------+---------+---------+---------+---------+
 164 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 165 +=====================+=========+=========+=========+=========+
 166 |   nosdn             |    X    |    X    |    X    |    X    |
 167 +---------------------+---------+---------+---------+---------+
 168 |   nosdn-noha        |         |         |    X    |    X    |
 169 +---------------------+---------+---------+---------+---------+
 170 |   kvm               |         |         |    X    |         |
 171 +---------------------+---------+---------+---------+---------+
 172 |   kvm-noha          |         |         |    X    |         |
 173 +---------------------+---------+---------+---------+---------+
 174 |   lxd               |         |         |         |    X    |
 175 +---------------------+---------+---------+---------+---------+
 176 |   lxd-noha          |         |         |         |    X    |
 177 +---------------------+---------+---------+---------+---------+
 178 |   ovs-noha (dpdk)   |         |         |    X    |         |
 179 +---------------------+---------+---------+---------+---------+
 180 |   odl_l2            |    X    |    X    |    X    |    X    |
 181 +---------------------+---------+---------+---------+---------+
 182 |   odl_l2-noha       |         |         |    X    |         |
 183 +---------------------+---------+---------+---------+---------+
 184 |   odl_l2-bgpvpn     |    X    |         |    X    |         |
 185 +---------------------+---------+---------+---------+---------+
 186 |   odl_l2-bgpvpn-noha|         |         |    X    |         |
 187 +---------------------+---------+---------+---------+---------+
 188 |   odl_l2-fdio-noha  |    X    |         |         |         |
 189 +---------------------+---------+---------+---------+---------+
 190 |   odl_l2-moon       |         |    X    |         |         |
 191 +---------------------+---------+---------+---------+---------+
 192 |   odl_l2-sfc-noha   |    X    |         |    X    |         |
 193 +---------------------+---------+---------+---------+---------+
 194 |   odl_l3            |    X    |    X    |    X    |         |
 195 +---------------------+---------+---------+---------+---------+
 196 |   odl_l3-noha       |         |         |    X    |         |
 197 +---------------------+---------+---------+---------+---------+
 198 |   onos              |         |    X    |    X    |    X    |
 199 +---------------------+---------+---------+---------+---------+
 200 |   onos-noha         |         |         |    X    |         |
 201 +---------------------+---------+---------+---------+---------+
 202 |   onos-sfc          |         |    X    |    X    |    X    |
 203 +---------------------+---------+---------+---------+---------+
 204 |   onos-sfc-noha     |         |         |    X    |         |
 205 +---------------------+---------+---------+---------+---------+
 206 |   multisite         |         |         |    X    |         |
 207 +---------------------+---------+---------+---------+---------+
 208
 209 For Colorado 2.0, the following scenarios have been re-released:
 210
 211 +---------------------+---------+---------+---------+---------+
 212 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 213 +=====================+=========+=========+=========+=========+
 214 |   onos              |    X    |    X    |    X    |    X    |
 215 +---------------------+---------+---------+---------+---------+
 216 |   odl_l2-sfc-ha     |         |         |    X    |         |
 217 +---------------------+---------+---------+---------+---------+
 218
 219
 220 In Colorado, the functional tests have been sliced in 6 different
 221 categories:
 222
 223 +----------------+-----------------------------------------------+
 224 | Category       |  Description                                  |
 225 +================+===============================================+
 226 | healthcheck    | Basic OpenStack commands                      |
 227 +----------------+-----------------------------------------------+
 228 | smoke          | vPings, Tempest and rally smoke tests         |
 229 +----------------+-----------------------------------------------+
 230 | sdn_suites     | Specific SDN feature tests                    |
 231 +----------------+-----------------------------------------------+
 232 | features       | OPNFV feature project functional test suites  |
 233 +----------------+-----------------------------------------------+
 234 | openstack      | Advanced, long duration OpenStack tests       |
 235 |                | (Tempest and Rally full suite). Each test may |
 236 |                | last several hours                            |
 237 +----------------+-----------------------------------------------+
 238 | vnf            | Complex scenarios dealing with orchestration, |
 239 |                | VNF deployment and tests (e.g. vIMS)          |
 240 +----------------+-----------------------------------------------+
 241
 242 For the scenario validation, we consider only the healthcheck, smoke,
 243 sdn_suites and features categories. These tests are run systematically
 244 in the CI daily loops.
 245
 246 Success criteria have been defined for these test cases, they can be
 247 PASS/FAIL or a success rate may be declared (100%, > 90%)
 248 All the criteria, as well as the test dependencies are declared in the
 249 ci/testcases.yaml file.
 250
 251 The scoring for the Colorado release per installer can be described as
 252 follows.
 253
 254 The scoring is an indicator showing how many feature project test suites
 255 have been integrated on the scenario.
 256
 257 The scoring equals the number of tests * succesful iteration of each
 258 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
 259
 260  e.g.
 261  apex/odl_l2-nofeature-ha
 262  tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
 263  Scoring = 21/21 = 7 * 3
 264
 265 By default, if not specified, the scenarios are HA.
 266 HA means OpenStack High Availability (main services). Note that not
 267 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
 268 depends upon support of the specific service for HA deployment.
 269 For example, in the Colorado release, the Congress service
 270 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
 271 support for HA operation has not yet been verified.
 272 See the release notes of the installers for more details.
 273
 274
 275 Apex
 276 ----
 277
 278 +------------------+---------+---------+-------------------+
 279 |  Scenario        | Scoring | Success |    Results        |
 280 |                  |         | rate    |                   |
 281 +==================+=========+=========+===================+
 282 | nosdn            |  17/18  |   95%   | `apex-res-174`_   |
 283 +------------------+---------+---------+-------------------+
 284 | odl_l2           |  21/21  |   100%  | `apex-res-175`_   |
 285 +------------------+---------+---------+-------------------+
 286 | odl_l3           |  15/18  |    83%  | `apex-res-176`_   |
 287 +------------------+---------+---------+-------------------+
 288 | odl_l2-bgpvpn    |  14/18  |    78%  | `apex-res-235`_   |
 289 +------------------+---------+---------+-------------------+
 290 | odl_l2-fdio-noha |  12/15  |    80%  | `apex-res-6`_     |
 291 +------------------+---------+---------+-------------------+
 292 | odl_l2-sfc-noha  |  18/24  |    75%  | `apex-res-217`_   |
 293 +------------------+---------+---------+-------------------+
 294 | onos-nofeature-ha|  20/21  |    95%  | `apex-res-423`_   |
 295 +------------------+---------+---------+-------------------+
 296
 297
 298 Compass
 299 -------
 300
 301 +------------------+---------+---------+-------------------+
 302 |  Scenario        | Scoring | Success |  Results          |
 303 |                  |         | rate    |                   |
 304 +==================+=========+=========+===================+
 305 | nosdn            |  12/12  |   100%  | `compass-res-55`_ |
 306 +------------------+---------+---------+-------------------+
 307 | odl_l2           |  15/15  |   100%  | `compass-res-59`_ |
 308 +------------------+---------+---------+-------------------+
 309 | odl_l3           |  9/12   |    75%  | `compass-res-73`_ |
 310 +------------------+---------+---------+-------------------+
 311 | odl_l2-moon      |  15/18  |    83%  | `compass-res-567`_|
 312 +------------------+---------+---------+-------------------+
 313 | onos-ha          |  15/15  |   100%  | `compass-res-285`_|
 314 +------------------+---------+---------+-------------------+
 315 | onos-sfc-ha      |  17/18  |    95%  | `compass-res-76`_ |
 316 +------------------+---------+---------+-------------------+
 317
 318 Note: all the Compass tests for Colorado have been executed on virtual
 319 environment. Bare metal resources were used for Master branch.
 320
 321
 322 Fuel
 323 ----
 324
 325 +---------------------+---------+---------+-------------------+
 326 |  Scenario           | Scoring | Success |  Results          |
 327 |                     |         | rate    |                   |
 328 +=====================+=========+=========+===================+
 329 | nosdn               |  18/18  |  100%   | `fuel-res-129`_   |
 330 +---------------------+---------+---------+-------------------+
 331 | nosdn-noha          |  15/15  |  100%   | `fuel-res-154`_   |
 332 +---------------------+---------+---------+-------------------+
 333 | nosdn-kvm           |  18/18  |  100%   | `fuel-res-128`_   |
 334 +---------------------+---------+---------+-------------------+
 335 | nosdn-kvm-noha      |  15/15  |  100%   | `fuel-res-161`_   |
 336 +---------------------+---------+---------+-------------------+
 337 | nosdn-ovs           |  12/18  |   67%*  |  `fuel-res-213`_  |
 338 +---------------------+---------+---------+-------------------+
 339 | nosdn-ovs-noha      |  15/15  |  100%   | `fuel-res-162`_   |
 340 +---------------------+---------+---------+-------------------+
 341 | odl_l2              |  21/21  |  100%   |  `fuel-res-123`_  |
 342 +---------------------+---------+---------+-------------------+
 343 | odl_l2-noha         |  17/18  |   94%   | `fuel-res-155`_   |
 344 +---------------------+---------+---------+-------------------+
 345 | odl_l2-bgpvpn       |  14/18  |   78%   | `fuel-res-119`_   |
 346 +---------------------+---------+---------+-------------------+
 347 | odl_l2-bgpvpn-noha  |  14/15  |   93%   | `fuel-res-160`_   |
 348 +---------------------+---------+---------+-------------------+
 349 | odl_l2-sfc-noha     |   6/21  |   29%   | `fuel-res-219`_   |
 350 +---------------------+---------+---------+-------------------+
 351 | odl_l2-sfc-ha       |  16/21  |   76%   | `fuel-res-376`_   |
 352 +---------------------+---------+---------+-------------------+
 353 | odl_l3              |  15/18  |   83%   | `fuel-res-115`_   |
 354 +---------------------+---------+---------+-------------------+
 355 | odl_l3-noha         |  12/15  |   80%   | `fuel-res-164`_   |
 356 +---------------------+---------+---------+-------------------+
 357 | onos                |  20/21  |   95%   | `fuel-res-492`_   |
 358 +---------------------+---------+---------+-------------------+
 359 | onos-noha           |  18/18  |  100%   | `fuel-res-166`_   |
 360 +---------------------+---------+---------+-------------------+
 361 | onos-sfc            |  24/24  |  100%   | `fuel-res-124`_   |
 362 +---------------------+---------+---------+-------------------+
 363 | onos-sfc-noha       |  21/21  |  100%   | `fuel-res-129`_   |
 364 +---------------------+---------+---------+-------------------+
 365 | multisite           |  N.R    |  100%   | `fuel-res-8`_     |
 366 +---------------------+---------+---------+-------------------+
 367
 368 *: all results passed, lacking iterations to reach the full score
 369
 370 Results of Functest on AArch64 Colorado 3.0
 371
 372 +---------------------+---------+---------+----------------------+
 373 |  Scenario           | Scoring | Success |  Results             |
 374 |                     |         | rate    |                      |
 375 +=====================+=========+=========+======================+
 376 | nosdn               |  18/18  |  100%   | `fuel-arm-res-128`_  |
 377 +---------------------+---------+---------+----------------------+
 378 | odl_l2              |  21/21  |  100%   | `fuel-arm-res-122`_  |
 379 +---------------------+---------+---------+----------------------+
 380 | odl_l2-noha         |  18/18  |  100%   | `fuel-arm-res-129`_  |
 381 +---------------------+---------+---------+----------------------+
 382 | odl_l3              |  17/18  |   95%   | `fuel-arm-res-135`_  |
 383 +---------------------+---------+---------+----------------------+
 384
 385
 386 Joid
 387 ----
 388
 389 +---------------------+---------+---------+-----------------+
 390 |  Scenario           | Scoring | Success |  Results        |
 391 |                     |         | rate    |                 |
 392 +=====================+=========+=========+=================+
 393 | nosdn               |  18/18  |  100%   | `joid-res-102`_ |
 394 +---------------------+---------+---------+-----------------+
 395 | nosdn-noha          |  17/18  |   95%   | `joid-res-93`_  |
 396 +---------------------+---------+---------+-----------------+
 397 | nosdn-lxd           |  12/12  |  100%   | `joid-res-104`_ |
 398 +---------------------+---------+---------+-----------------+
 399 | nosdn-lxd-noha      |  12/12  |  100%   | `joid-res-91`_  |
 400 +---------------------+---------+---------+-----------------+
 401 | odl_l2              |  21/21  |  100%   | `joid-res-103`_ |
 402 +---------------------+---------+---------+-----------------+
 403 | onos                |  21/21  |  100%   | `joid-res-345`_ |
 404 +---------------------+---------+---------+-----------------+
 405 | onos-sfc            |  24/24  |  100%   | `joid-res-97`_  |
 406 +---------------------+---------+---------+-----------------+
 407
 408 It is highly recommended to install a json viewer in your browser
 409 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
 410
 411 You can get additional details through test logs on http://artifacts.opnfv.org/.
 412 As no search engine is available on the OPNFV artifact web site you must
 413 retrieve the pod identifier on which the tests have been executed (see
 414 field pod in any of the results) then click on the selected POD and look
 415 for the date of the test you are interested in.
 416
 417 The reporting pages can be found at:
 418
 419  * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
 420  * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
 421  * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
 422  * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html
 423
 424 Colorado known restrictions/issues
 425 ==================================
 426
 427 +-----------+-----------+----------------------------------------------+
 428 | Installer | Scenario  |  Issue                                       |
 429 +===========+===========+==============================================+
 430 | any       | odl_l3-*  | Tempest cases related to using floating IP   |
 431 |           |           | addresses fail because of a known ODL bug.   |
 432 |           |           | vPing_ssh test case is excluded for the same |
 433 |           |           | reason.                                      |
 434 |           |           | https://jira.opnfv.org/browse/APEX-112       |
 435 |           |           | https://jira.opnfv.org/browse/FUNCTEST-445   |
 436 +-----------+-----------+----------------------------------------------+
 437 | apex/fuel | *-bgpvpn  | vPing_ssh (floating ips not supported) and   |
 438 |           |           | vIMS excluded. Some Tempest cases related to |
 439 |           |           | floating ips also excluded. Some performance |
 440 |           |           | issues have been detected in this scenario   |
 441 |           |           | (i.e. BGPVPN extension enabled) when running |
 442 |           |           | commands against the OpenStack APIs, thus    |
 443 |           |           | Rally sanity test case has been disabled.    |
 444 |           |           | Performance issues seem to be connected to   |
 445 |           |           | the ODL version. It is planned to reintroduce|
 446 |           |           | Rally sanity in Colorado 2.0 with the        |
 447 |           |           | adoption of ODL Boron release.               |
 448 +-----------+-----------+----------------------------------------------+
 449 | apex      | *-fdio    | Due to late integration, fdio decided to     |
 450 |           |           | focus on mandatory tests and exclude feature |
 451 |           |           | tests (copper, doctor, security_scan) from   |
 452 |           |           | its scenarios                                |
 453 +-----------+-----------+----------------------------------------------+
 454 | compass   | moon      | First ODL test FAILS because ODL/Openstack   |
 455 |           |           | federation done in moon is partial. Only     |
 456 |           |           | MD-SAL is federated (not AD-SAL)             |
 457 +-----------+-----------+----------------------------------------------+
 458 | fuel      | any       | TestServerBasicOps test case skipped         |
 459 |           |           | https://gerrit.opnfv.org/gerrit/#/c/19635/   |
 460 +-----------+-----------+----------------------------------------------+
 461 | fuel      | kvm       | vPing_ssh and vIMS excluded (metadata related|
 462 |           |           | scenarios)                                   |
 463 +-----------+-----------+----------------------------------------------+
 464 | fuel      | multisite | random errors running multisite. A known bug |
 465 |           |           | in keystone mitaka, due to which memcache    |
 466 |           |           | raises exception and keystone becomes        |
 467 |           |           | unresponsive                                 |
 468 |           |           | bugs.launchpad.net/keystone/+bug/1600394     |
 469 |           |           | workaround consists in restarting memcache on|
 470 |           |           | server                                       |
 471 +-----------+-----------+----------------------------------------------+
 472 | joid      | any       | Tempest cases related to object storage      |
 473 |           |           | excluded                                     |
 474 |           |           | https://gerrit.opnfv.org/gerrit/#/c/17871/   |
 475 +-----------+-----------+----------------------------------------------+
 476 | joid      | domino    | Domino tests are skipped in CI. However the  |
 477 |           |           | test case can be run by manually setting     |
 478 |           |           | IS_IPandKEY_CONFIGURED=true after manually   |
 479 |           |           | setting properly the IP addresses of the 3   |
 480 |           |           | Controller nodes in the configuration file   |
 481 |           |           | /repos/domino/tests/run_multinode.sh         |
 482 +-----------+-----------+----------------------------------------------+
 483
 484
 485 Test and installer/scenario dependencies
 486 ========================================
 487
 488 It is not always possible to run all the test cases on all the scenarios.
 489 The following table details the dependencies of the test cases per
 490 scenario. The scenario dependencies (installer or scenario) are detailed
 491 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 492
 493 Test results
 494 ============
 495
 496 Test results are available in:
 497
 498  - test results document: http://artifacts.opnfv.org/functest
 499
 500  - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
 501
 502  - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
 503
 504
 505
 506 Open JIRA tickets
 507 =================
 508
 509 +------------------+-----------------------------------------------+
 510 |   JIRA           |         Description                           |
 511 +==================+===============================================+
 512 | `FUNCTEST-446`_  |  Cleanup ODL-SFC output in Functest execution |
 513 |                  |  Impact on odl_l2-sfc scenarios               |
 514 +------------------+-----------------------------------------------+
 515 | `FUNCTEST-454`_  |  Cleanup failures when using HA networks in   |
 516 |                  |  Neutron                                      |
 517 +------------------+-----------------------------------------------+
 518
 519 Useful links
 520 ============
 521
 522  - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
 523
 524  - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado
 525
 526  - Functest repo: https://git.opnfv.org/cgit/functest
 527
 528  - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
 529
 530  - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
 531
 532  - Functest IRC chan: #opnfv-functest
 533
 534  - Functest reporting: http://testresults.opnfv.org/reporting
 535
 536  - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 537
 538  - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
 539
 540  - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
 541
 542  - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 543
 544 .. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
 545
 546 .. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
 547
 548 .. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
 549
 550 .. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
 551
 552 .. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
 553
 554 .. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
 555
 556 .. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
 557
 558 .. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
 559
 560 .. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423
 561
 562 .. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
 563
 564 .. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
 565
 566 .. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
 567
 568 .. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
 569
 570 .. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
 571
 572 .. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
 573
 574 .. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285
 575
 576 .. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
 577
 578 .. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
 579
 580 .. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
 581
 582 .. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
 583
 584 .. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
 585
 586 .. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
 587
 588 .. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
 589
 590 .. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
 591
 592 .. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
 593
 594 .. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
 595
 596 .. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
 597
 598 .. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
 599
 600 .. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
 601
 602 .. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
 603
 604 .. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
 605
 606 .. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213
 607
 608 .. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219
 609
 610 .. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376
 611
 612 .. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492
 613
 614 .. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console
 615
 616 .. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console
 617
 618 .. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console
 619
 620 .. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console
 621
 622 .. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
 623
 624 .. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
 625
 626 .. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
 627
 628 .. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
 629
 630 .. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
 631
 632 .. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
 633
 634 .. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
 635
 636 .. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345
 637