docs/release-notes/functest-release.rst

   1 =======
   2 License
   3 =======
   4
   5 OPNFV Colorado release note for Functest Docs
   6 (c) by Morgan Richomme (Orange)
   7
   8 OPNFV Colorado release note for Functest Docs
   9 are licensed under a Creative Commons Attribution 4.0 International License.
  10 You should have received a copy of the license along with this.
  11 If not, see <http://creativecommons.org/licenses/by/4.0/>.
  12
  13 ===========================================
  14 OPNFV Colorado1.0 release note for Functest
  15 ===========================================
  16
  17 Abstract
  18 ========
  19
  20 This document describes the release note of Functest project.
  21
  22
  23 Version history
  24 ===============
  25
  26 +------------+----------+------------------+------------------------+
  27 | **Date**   | **Ver.** | **Author**       | **Comment**            |
  28 |            |          |                  |                        |
  29 +------------+----------+------------------+------------------------+
  30 | 2016-08-17 | 1.0.0    | Morgan Richomme  | Functest for Colorado  |
  31 |            |          | (Orange)         | release                |
  32 +------------+----------+------------------+------------------------+
  33
  34 OPNFV Colorado Release
  35 =========================
  36
  37 Functest deals with functional testing of the OPNFV solution.
  38 It includes test cases developed within the project and test cases developed in
  39 other OPNFV projects and other upstream communities.
  40
  41 The internal test cases are:
  42
  43  * healthcheck
  44  * vPing ssh
  45  * vPing userdata
  46  * Tempest Smoke Serial
  47  * Rally Sanity
  48  * ODL
  49  * Tempest full parallel
  50  * Rally full
  51  * vIMS
  52
  53 The OPNFV projects integrated into Functest framework for automation are:
  54
  55  * bgpvpn
  56  * Copper
  57  * Doctor
  58  * Domino
  59  * Moon
  60  * Multisite
  61  * ONOSFW
  62  * ONOS-sfc
  63  * ODL-sfc
  64  * Parser
  65  * Promise
  66  * Security scan
  67
  68 The validation of a scenario requires a subset of these tests depending
  69 on the installer and the scenario.
  70
  71 The 3 last internal test cases (tempest full parallel, Rally full and
  72 vIMS) are not considered for scenario validation.
  73
  74 Release Data
  75 ============
  76
  77 +--------------------------------------+--------------------------------------+
  78 | **Project**                          | functest                             |
  79 |                                      |                                      |
  80 +--------------------------------------+--------------------------------------+
  81 | **Repo/tag**                         | colorado.1.0                         |
  82 |                                      |                                      |
  83 +--------------------------------------+--------------------------------------+
  84 | **Release designation**              | Colorado base release                |
  85 |                                      |                                      |
  86 +--------------------------------------+--------------------------------------+
  87 | **Release date**                     | September 22 2016                    |
  88 |                                      |                                      |
  89 +--------------------------------------+--------------------------------------+
  90 | **Purpose of the delivery**          | Colorado base release                |
  91 |                                      |                                      |
  92 +--------------------------------------+--------------------------------------+
  93
  94 Deliverables
  95 ============
  96
  97 Software
  98 --------
  99
 100  - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0)
 101
 102  - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0)
 103
 104
 105 Documents
 106 ---------
 107
 108  - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
 109
 110  - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
 111
 112  - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 113
 114
 115 Version change
 116 ==============
 117
 118 Feature evolution
 119 -----------------
 120
 121  - refactoring of ODL functional tests (with upstream modifications)
 122
 123  - refactoring of testapi (update, swagger documentation, dockerization)
 124
 125  - jenkins logs improvement
 126
 127  - update integration of Doctor, Promise and SDNVPN  projects
 128
 129  - split Tempest and rally into 2 different tests: smoke and full
 130
 131  - vIMS test suite integration
 132
 133  - adoption of Kibana for dashboarding
 134
 135
 136 New features
 137 ------------
 138
 139  - Functest CLI to prepare and run the tests
 140
 141  - creation of the healthcheck test case
 142
 143  - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
 144
 145  - integration of new OPNFV feature projects (copper, domino, multisite,
 146  moon, parser, onos-sfc, odl-sfc, security scan)
 147
 148  - introduction of test tiers in functest framework
 149
 150  - automatic reporting
 151
 152  - introduction of a jenkins summary table
 153
 154  - support of ARM architecture
 155
 156
 157 Scenario Matrix
 158 ===============
 159
 160 For Colorado 1.0, Functest was tested on the following scenarios (if not
 161 precised, the scenario is a HA scenario):
 162
 163 +---------------------+---------+---------+---------+---------+
 164 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 165 +=====================+=========+=========+=========+=========+
 166 |   nosdn             |    X    |    X    |    X    |    X    |
 167 +---------------------+---------+---------+---------+---------+
 168 |   nosdn-noha        |         |         |    X    |    X    |
 169 +---------------------+---------+---------+---------+---------+
 170 |   kvm               |         |         |    X    |         |
 171 +---------------------+---------+---------+---------+---------+
 172 |   kvm-noha          |         |         |    X    |         |
 173 +---------------------+---------+---------+---------+---------+
 174 |   lxd               |         |         |         |    X    |
 175 +---------------------+---------+---------+---------+---------+
 176 |   lxd-noha          |         |         |         |    X    |
 177 +---------------------+---------+---------+---------+---------+
 178 |   ovs-noha (dpdk)   |         |         |    X    |         |
 179 +---------------------+---------+---------+---------+---------+
 180 |   odl_l2            |    X    |    X    |    X    |    X    |
 181 +---------------------+---------+---------+---------+---------+
 182 |   odl_l2-noha       |         |         |    X    |         |
 183 +---------------------+---------+---------+---------+---------+
 184 |   odl_l2-bgpvpn     |    X    |         |    X    |         |
 185 +---------------------+---------+---------+---------+---------+
 186 |   odl_l2-bgpvpn-noha|         |         |    X    |         |
 187 +---------------------+---------+---------+---------+---------+
 188 |   odl_l2-fdio-noha  |    X    |         |         |         |
 189 +---------------------+---------+---------+---------+---------+
 190 |   odl_l2-moon       |         |    X    |         |         |
 191 +---------------------+---------+---------+---------+---------+
 192 |   odl_l2-sfc-noha   |    X    |         |    X    |         |
 193 +---------------------+---------+---------+---------+---------+
 194 |   odl_l3            |    X    |    X    |    X    |         |
 195 +---------------------+---------+---------+---------+---------+
 196 |   odl_l3-noha       |         |         |    X    |         |
 197 +---------------------+---------+---------+---------+---------+
 198 |   onos              |         |    X    |    X    |    X    |
 199 +---------------------+---------+---------+---------+---------+
 200 |   onos-noha         |         |         |    X    |         |
 201 +---------------------+---------+---------+---------+---------+
 202 |   onos-sfc          |         |    X    |    X    |    X    |
 203 +---------------------+---------+---------+---------+---------+
 204 |   onos-sfc-noha     |         |         |    X    |         |
 205 +---------------------+---------+---------+---------+---------+
 206 |   multisite         |         |         |    X    |         |
 207 +---------------------+---------+---------+---------+---------+
 208
 209 For Colorado 2.0, the following scenarios have been released:
 210
 211 +---------------------+---------+---------+---------+---------+
 212 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 213 +=====================+=========+=========+=========+=========+
 214 |   onos-nofeature    |    X    |    X    |    X    |    X    |
 215 +---------------------+---------+---------+---------+---------+
 216 |   odl_l2-sfc-ha     |         |         |    X    |         |
 217 +---------------------+---------+---------+---------+---------+
 218
 219
 220 In Colorado, the functional tests have been sliced in 6 different
 221 categories:
 222
 223 +----------------+-----------------------------------------------+
 224 | Category       |  Description                                  |
 225 +================+===============================================+
 226 | healthcheck    | Basic OpenStack commands                      |
 227 +----------------+-----------------------------------------------+
 228 | smoke          | vPings, Tempest and rally smoke tests         |
 229 +----------------+-----------------------------------------------+
 230 | sdn_suites     | Specific SDN feature tests                    |
 231 +----------------+-----------------------------------------------+
 232 | features       | OPNFV feature project functional test suites  |
 233 +----------------+-----------------------------------------------+
 234 | openstack      | Advanced, long duration OpenStack tests       |
 235 |                | (Tempest and Rally full suite). Each test may |
 236 |                | last several hours                            |
 237 +----------------+-----------------------------------------------+
 238 | vnf            | Complex scenarios dealing with orchestration, |
 239 |                | VNF deployment and tests (e.g. vIMS)          |
 240 +----------------+-----------------------------------------------+
 241
 242 For the scenario validation, we consider only the healthcheck, smoke,
 243 sdn_suites and features categories. These tests are run systematically
 244 in the CI daily loops.
 245
 246 Success criteria have been defined for these test cases, they can be
 247 PASS/FAIL or a success rate may be declared (100%, > 90%)
 248 All the criteria, as well as the test dependencies are declared in the
 249 ci/testcases.yaml file.
 250
 251 The scoring for the Colorado release per installer can be described as
 252 follows.
 253
 254 The scoring is an indicator showing how many feature project test suites
 255 have been integrated on the scenario.
 256
 257 The scoring equals the number of tests * succesful iteration of each
 258 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
 259
 260  e.g.
 261  apex/odl_l2-nofeature-ha
 262  tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
 263  Scoring = 21/21 = 7 * 3
 264
 265 By default, if not specified, the scenarios are HA.
 266 HA means OpenStack High Availability (main services). Note that not
 267 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
 268 depends upon support of the specific service for HA deployment.
 269 For example, in the Colorado release, the Congress service
 270 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
 271 support for HA operation has not yet been verified.
 272 See the release notes of the installers for more details.
 273
 274
 275 Apex
 276 ----
 277
 278 +------------------+---------+---------+-------------------+
 279 |  Scenario        | Scoring | Success |    Results        |
 280 |                  |         | rate    |                   |
 281 +==================+=========+=========+===================+
 282 | nosdn            |  17/18  |   95%   | `apex-res-174`_   |
 283 +------------------+---------+---------+-------------------+
 284 | odl_l2           |  21/21  |   100%  | `apex-res-175`_   |
 285 +------------------+---------+---------+-------------------+
 286 | odl_l3           |  15/18  |    83%  | `apex-res-176`_   |
 287 +------------------+---------+---------+-------------------+
 288 | odl_l2-bgpvpn    |  14/18  |    78%  | `apex-res-235`_   |
 289 +------------------+---------+---------+-------------------+
 290 | odl_l2-fdio-noha |  12/15  |    80%  | `apex-res-6`_     |
 291 +------------------+---------+---------+-------------------+
 292 | odl_l2-sfc-noha  |  18/24  |    75%  | `apex-res-217`_   |
 293 +------------------+---------+---------+-------------------+
 294 | onos-nofeature-ha|  20/21  |    95%  | `apex-res-423`_   |
 295 +------------------+---------+---------+-------------------+
 296
 297
 298 Compass
 299 -------
 300
 301 +------------------+---------+---------+-------------------+
 302 |  Scenario        | Scoring | Success |  Results          |
 303 |                  |         | rate    |                   |
 304 +==================+=========+=========+===================+
 305 | nosdn            |  12/12  |   100%  | `compass-res-55`_ |
 306 +------------------+---------+---------+-------------------+
 307 | odl_l2           |  15/15  |   100%  | `compass-res-59`_ |
 308 +------------------+---------+---------+-------------------+
 309 | odl_l3           |  9/12   |    75%  | `compass-res-73`_ |
 310 +------------------+---------+---------+-------------------+
 311 | odl_l2-moon      |  15/18  |    83%  | `compass-res-567`_|
 312 +------------------+---------+---------+-------------------+
 313 | onos-ha          |  15/15  |   100%  | `compass-res-77`_ |
 314 +------------------+---------+---------+-------------------+
 315 | onos-sfc-ha      |  17/18  |    95%  | `compass-res-76`_ |
 316 +------------------+---------+---------+-------------------+
 317 | onos-nofeature-ha|  15/15  |   100%  | `compass-res-285`_|
 318 +------------------+---------+---------+-------------------+
 319
 320 Note: all the Compass tests for Colorado have been executed on virtual
 321 environment. Bare metal resources were used for Master branch.
 322
 323
 324 Fuel
 325 ----
 326
 327 +---------------------+---------+---------+-------------------+
 328 |  Scenario           | Scoring | Success |  Results          |
 329 |                     |         | rate    |                   |
 330 +=====================+=========+=========+===================+
 331 | nosdn               |  18/18  |  100%   | `fuel-res-129`_   |
 332 +---------------------+---------+---------+-------------------+
 333 | nosdn-noha          |  15/15  |  100%   | `fuel-res-154`_   |
 334 +---------------------+---------+---------+-------------------+
 335 | nosdn-kvm           |  18/18  |  100%   | `fuel-res-128`_   |
 336 +---------------------+---------+---------+-------------------+
 337 | nosdn-kvm-noha      |  15/15  |  100%   | `fuel-res-161`_   |
 338 +---------------------+---------+---------+-------------------+
 339 | nosdn-ovs           |  12/18  |   67%*  |  `fuel-res-213`_  |
 340 +---------------------+---------+---------+-------------------+
 341 | nosdn-ovs-noha      |  15/15  |  100%   | `fuel-res-162`_   |
 342 +---------------------+---------+---------+-------------------+
 343 | odl_l2              |  21/21  |  100%   |  `fuel-res-123`_  |
 344 +---------------------+---------+---------+-------------------+
 345 | odl_l2-noha         |  17/18  |   94%   | `fuel-res-155`_   |
 346 +---------------------+---------+---------+-------------------+
 347 | odl_l2-bgpvpn       |  14/18  |   78%   | `fuel-res-119`_   |
 348 +---------------------+---------+---------+-------------------+
 349 | odl_l2-bgpvpn-noha  |  14/15  |   93%   | `fuel-res-160`_   |
 350 +---------------------+---------+---------+-------------------+
 351 | odl_l2-sfc-noha     |   6/21  |   29%   | `fuel-res-219`_   |
 352 +---------------------+---------+---------+-------------------+
 353 | odl_l2-sfc-ha       |  16/21  |   76%   | `fuel-res-376`_   |
 354 +---------------------+---------+---------+-------------------+
 355 | odl_l3              |  15/18  |   83%   | `fuel-res-115`_   |
 356 +---------------------+---------+---------+-------------------+
 357 | odl_l3-noha         |  12/15  |   80%   | `fuel-res-164`_   |
 358 +---------------------+---------+---------+-------------------+
 359 | onos                |  20/21  |   95%   | `fuel-res-117`_   |
 360 +---------------------+---------+---------+-------------------+
 361 | onos-noha           |  18/18  |  100%   | `fuel-res-166`_   |
 362 +---------------------+---------+---------+-------------------+
 363 | onos-sfc            |  24/24  |  100%   | `fuel-res-124`_   |
 364 +---------------------+---------+---------+-------------------+
 365 | onos-sfc-noha       |  21/21  |  100%   | `fuel-res-129`_   |
 366 +---------------------+---------+---------+-------------------+
 367 | onos-nofeature-ha   |  18/18  |  100%   | `fuel-res-492`_   |
 368 +---------------------+---------+---------+-------------------+
 369 | multisite           |  N.R    |  100%   | `fuel-res-8`_     |
 370 +---------------------+---------+---------+-------------------+
 371
 372 *: all results passed, lacking iterations to reach the full score
 373
 374 Results of Functest on AArch64 Colorado 3.0
 375
 376 +---------------------+---------+---------+----------------------+
 377 |  Scenario           | Scoring | Success |  Results             |
 378 |                     |         | rate    |                      |
 379 +=====================+=========+=========+======================+
 380 | nosdn               |  18/18  |  100%   | `fuel-arm-res-128`_  |
 381 +---------------------+---------+---------+----------------------+
 382 | odl_l2              |  21/21  |  100%   | `fuel-arm-res-122`_  |
 383 +---------------------+---------+---------+----------------------+
 384 | odl_l2-noha         |  18/18  |  100%   | `fuel-arm-res-129`_  |
 385 +---------------------+---------+---------+----------------------+
 386 | odl_l3              |  17/18  |   95%   | `fuel-arm-res-135`_  |
 387 +---------------------+---------+---------+----------------------+
 388
 389
 390 Joid
 391 ----
 392
 393 +---------------------+---------+---------+-----------------+
 394 |  Scenario           | Scoring | Success |  Results        |
 395 |                     |         | rate    |                 |
 396 +=====================+=========+=========+=================+
 397 | nosdn               |  18/18  |  100%   | `joid-res-102`_ |
 398 +---------------------+---------+---------+-----------------+
 399 | nosdn-noha          |  17/18  |   95%   | `joid-res-93`_  |
 400 +---------------------+---------+---------+-----------------+
 401 | nosdn-lxd           |  12/12  |  100%   | `joid-res-104`_ |
 402 +---------------------+---------+---------+-----------------+
 403 | nosdn-lxd-noha      |  12/12  |  100%   | `joid-res-91`_  |
 404 +---------------------+---------+---------+-----------------+
 405 | odl_l2              |  21/21  |  100%   | `joid-res-103`_ |
 406 +---------------------+---------+---------+-----------------+
 407 | onos                |  21/21  |  100%   | `joid-res-99`_  |
 408 +---------------------+---------+---------+-----------------+
 409 | onos-sfc            |  24/24  |  100%   | `joid-res-97`_  |
 410 +---------------------+---------+---------+-----------------+
 411 | onos-nofeature-ha   |  21/21  |  100%   | `joid-res-345`_ |
 412 +---------------------+---------+---------+-----------------+
 413
 414 It is highly recommended to install a json viewer in your browser
 415 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
 416
 417 You can get additional details through test logs on http://artifacts.opnfv.org/.
 418 As no search engine is available on the OPNFV artifact web site you must
 419 retrieve the pod identifier on which the tests have been executed (see
 420 field pod in any of the results) then click on the selected POD and look
 421 for the date of the test you are interested in.
 422
 423 The reporting pages can be found at:
 424
 425  * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
 426  * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
 427  * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
 428  * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html
 429
 430 Colorado known restrictions/issues
 431 ==================================
 432
 433 +-----------+-----------+----------------------------------------------+
 434 | Installer | Scenario  |  Issue                                       |
 435 +===========+===========+==============================================+
 436 | any       | odl_l3-*  | Tempest cases related to using floating IP   |
 437 |           |           | addresses fail because of a known ODL bug.   |
 438 |           |           | vPing_ssh test case is excluded for the same |
 439 |           |           | reason.                                      |
 440 |           |           | https://jira.opnfv.org/browse/APEX-112       |
 441 |           |           | https://jira.opnfv.org/browse/FUNCTEST-445   |
 442 +-----------+-----------+----------------------------------------------+
 443 | apex/fuel | *-bgpvpn  | vPing_ssh (floating ips not supported) and   |
 444 |           |           | vIMS excluded. Some Tempest cases related to |
 445 |           |           | floating ips also excluded. Some performance |
 446 |           |           | issues have been detected in this scenario   |
 447 |           |           | (i.e. BGPVPN extension enabled) when running |
 448 |           |           | commands against the OpenStack APIs, thus    |
 449 |           |           | Rally sanity test case has been disabled.    |
 450 |           |           | Performance issues seem to be connected to   |
 451 |           |           | the ODL version. It is planned to reintroduce|
 452 |           |           | Rally sanity in Colorado 2.0 with the        |
 453 |           |           | adoption of ODL Boron release.               |
 454 +-----------+-----------+----------------------------------------------+
 455 | apex      | *-fdio    | Due to late integration, fdio decided to     |
 456 |           |           | focus on mandatory tests and exclude feature |
 457 |           |           | tests (copper, doctor, security_scan) from   |
 458 |           |           | its scenarios                                |
 459 +-----------+-----------+----------------------------------------------+
 460 | compass   | moon      | First ODL test FAILS because ODL/Openstack   |
 461 |           |           | federation done in moon is partial. Only     |
 462 |           |           | MD-SAL is federated (not AD-SAL)             |
 463 +-----------+-----------+----------------------------------------------+
 464 | fuel      | any       | TestServerBasicOps test case skipped         |
 465 |           |           | https://gerrit.opnfv.org/gerrit/#/c/19635/   |
 466 +-----------+-----------+----------------------------------------------+
 467 | fuel      | kvm       | vPing_ssh and vIMS excluded (metadata related|
 468 |           |           | scenarios)                                   |
 469 +-----------+-----------+----------------------------------------------+
 470 | fuel      | multisite | random errors running multisite. A known bug |
 471 |           |           | in keystone mitaka, due to which memcache    |
 472 |           |           | raises exception and keystone becomes        |
 473 |           |           | unresponsive                                 |
 474 |           |           | bugs.launchpad.net/keystone/+bug/1600394     |
 475 |           |           | workaround consists in restarting memcache on|
 476 |           |           | server                                       |
 477 +-----------+-----------+----------------------------------------------+
 478 | joid      | any       | Tempest cases related to object storage      |
 479 |           |           | excluded                                     |
 480 |           |           | https://gerrit.opnfv.org/gerrit/#/c/17871/   |
 481 +-----------+-----------+----------------------------------------------+
 482 | joid      | domino    | Domino tests are skipped in CI. However the  |
 483 |           |           | test case can be run by manually setting     |
 484 |           |           | IS_IPandKEY_CONFIGURED=true after manually   |
 485 |           |           | setting properly the IP addresses of the 3   |
 486 |           |           | Controller nodes in the configuration file   |
 487 |           |           | /repos/domino/tests/run_multinode.sh         |
 488 +-----------+-----------+----------------------------------------------+
 489
 490
 491 Test and installer/scenario dependencies
 492 ========================================
 493
 494 It is not always possible to run all the test cases on all the scenarios.
 495 The following table details the dependencies of the test cases per
 496 scenario. The scenario dependencies (installer or scenario) are detailed
 497 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 498
 499 Test results
 500 ============
 501
 502 Test results are available in:
 503
 504  - test results document: http://artifacts.opnfv.org/functest
 505
 506  - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
 507
 508  - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
 509
 510
 511
 512 Open JIRA tickets
 513 =================
 514
 515 +------------------+-----------------------------------------------+
 516 |   JIRA           |         Description                           |
 517 +==================+===============================================+
 518 | `FUNCTEST-446`_  |  Cleanup ODL-SFC output in Functest execution |
 519 |                  |  Impact on odl_l2-sfc scenarios               |
 520 +------------------+-----------------------------------------------+
 521 | `FUNCTEST-454`_  |  Cleanup failures when using HA networks in   |
 522 |                  |  Neutron                                      |
 523 +------------------+-----------------------------------------------+
 524
 525 Useful links
 526 ============
 527
 528  - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
 529
 530  - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado
 531
 532  - Functest repo: https://git.opnfv.org/cgit/functest
 533
 534  - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
 535
 536  - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
 537
 538  - Functest IRC chan: #opnfv-functest
 539
 540  - Functest reporting: http://testresults.opnfv.org/reporting
 541
 542  - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 543
 544  - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
 545
 546  - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
 547
 548  - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 549
 550 .. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
 551
 552 .. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
 553
 554 .. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
 555
 556 .. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
 557
 558 .. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
 559
 560 .. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
 561
 562 .. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
 563
 564 .. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
 565
 566 .. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423
 567
 568 .. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
 569
 570 .. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
 571
 572 .. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
 573
 574 .. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
 575
 576 .. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
 577
 578 .. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
 579
 580 .. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285
 581
 582 .. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
 583
 584 .. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
 585
 586 .. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
 587
 588 .. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
 589
 590 .. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
 591
 592 .. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
 593
 594 .. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
 595
 596 .. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
 597
 598 .. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
 599
 600 .. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
 601
 602 .. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
 603
 604 .. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
 605
 606 .. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
 607
 608 .. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
 609
 610 .. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
 611
 612 .. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213
 613
 614 .. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219
 615
 616 .. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376
 617
 618 .. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492
 619
 620 .. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console
 621
 622 .. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console
 623
 624 .. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console
 625
 626 .. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console
 627
 628 .. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
 629
 630 .. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
 631
 632 .. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
 633
 634 .. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
 635
 636 .. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
 637
 638 .. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
 639
 640 .. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
 641
 642 .. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345
 643