5 OPNFV Colorado release note for Functest Docs
6 (c) by Morgan Richomme (Orange)
8 OPNFV Colorado release note for Functest Docs
9 are licensed under a Creative Commons Attribution 4.0 International License.
10 You should have received a copy of the license along with this.
11 If not, see <http://creativecommons.org/licenses/by/4.0/>.
13 ===========================================
14 OPNFV Colorado1.0 release note for Functest
15 ===========================================
20 This document describes the release note of Functest project.
26 +------------+----------+------------------+------------------------+
27 | **Date** | **Ver.** | **Author** | **Comment** |
29 +------------+----------+------------------+------------------------+
30 | 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado |
31 | | | (Orange) | release |
32 +------------+----------+------------------+------------------------+
34 OPNFV Colorado Release
35 =========================
37 Functest deals with functional testing of the OPNFV solution.
38 It includes test cases developed within the project and test cases developed in
39 other OPNFV projects and other upstream communities.
41 The internal test cases are:
46 * Tempest Smoke Serial
49 * Tempest full parallel
53 The OPNFV projects integrated into Functest framework for automation are:
68 The validation of a scenario requires a subset of these tests depending
69 on the installer and the scenario.
71 The 3 last internal test cases (tempest full parallel, Rally full and
72 vIMS) are not considered for scenario validation.
77 +--------------------------------------+--------------------------------------+
78 | **Project** | functest |
80 +--------------------------------------+--------------------------------------+
81 | **Repo/tag** | colorado.1.0 |
83 +--------------------------------------+--------------------------------------+
84 | **Release designation** | Colorado base release |
86 +--------------------------------------+--------------------------------------+
87 | **Release date** | September 22 2016 |
89 +--------------------------------------+--------------------------------------+
90 | **Purpose of the delivery** | Colorado base release |
92 +--------------------------------------+--------------------------------------+
100 - The Functest Docker image: * TODO link *
102 - The testapi Docker image: * TODO link *
108 - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
110 - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
112 - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
121 - refactoring of ODL functional tests (with upstream modifications)
123 - refactoring of testapi (update, swagger documentation, dockerization)
125 - jenkins logs improvement
127 - update integration of Doctor, Promise and SDNVPN projects
129 - split Tempest and rally into 2 different tests: smoke and full
131 - vIMS test suite integration
133 - adoption of Kibana for dashboarding
139 - Functest CLI to prepare and run the tests
141 - creation of the healthcheck test case
143 - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
145 - integration of new OPNFV feature projects (copper, domino, multisite,
146 moon, parser, onos-sfc, odl-sfc, security scan)
148 - introduction of test tiers in functest framework
150 - automatic reporting
152 - introduction of a jenkins summary table
154 - support of ARM architecture
160 For Colorado 1.0, Functest was tested on the following scenarios (if not
161 precised, the scenario is a HA scenario):
163 +---------------------+---------+---------+---------+---------+
164 | Scenario | Apex | Compass | Fuel | Joid |
165 +=====================+=========+=========+=========+=========+
166 | nosdn | X | X | X | X |
167 +---------------------+---------+---------+---------+---------+
168 | nosdn-noha | | | X | X |
169 +---------------------+---------+---------+---------+---------+
170 | odl_l2 | X | X | X | X |
171 +---------------------+---------+---------+---------+---------+
172 | odl_l2-noha | | | X | |
173 +---------------------+---------+---------+---------+---------+
174 | odl_l3 | X | X | X | |
175 +---------------------+---------+---------+---------+---------+
176 | odl_l3-noha | | | X | |
177 +---------------------+---------+---------+---------+---------+
178 | odl_l2-bgpvpn | X | | X | |
179 +---------------------+---------+---------+---------+---------+
180 | odl_l2-bgpvpn-noha| | | X | |
181 +---------------------+---------+---------+---------+---------+
182 | odl_l2-fdio-noha | X | | | |
183 +---------------------+---------+---------+---------+---------+
184 | odl_l2-moon | | X | | |
185 +---------------------+---------+---------+---------+---------+
186 | odl_l2-sfc | | | X | |
187 +---------------------+---------+---------+---------+---------+
188 | odl_l2-sfc-noha | X | | X | |
189 +---------------------+---------+---------+---------+---------+
190 | onos | | X | X | X |
191 +---------------------+---------+---------+---------+---------+
192 | onos-noha | | | X | |
193 +---------------------+---------+---------+---------+---------+
194 | onos-sfc | | X | X | X |
195 +---------------------+---------+---------+---------+---------+
196 | onos-sfc-noha | | | X | |
197 +---------------------+---------+---------+---------+---------+
198 | ovs-noha (dpdk) | | | X | |
199 +---------------------+---------+---------+---------+---------+
201 +---------------------+---------+---------+---------+---------+
202 | kvm-noha | | | X | |
203 +---------------------+---------+---------+---------+---------+
204 | multisite | | | X | |
205 +---------------------+---------+---------+---------+---------+
207 +---------------------+---------+---------+---------+---------+
208 | lxd-noha | | | | X |
209 +---------------------+---------+---------+---------+---------+
211 In Colorado, the functional tests have been sliced in 6 different
214 +----------------+-----------------------------------------------+
215 | Category | Description |
216 +================+===============================================+
217 | healthcheck | Basic OpenStack commands |
218 +----------------+-----------------------------------------------+
219 | smoke | vPings, Tempest and rally smoke tests |
220 +----------------+-----------------------------------------------+
221 | sdn_suites | Specific SDN feature tests |
222 +----------------+-----------------------------------------------+
223 | features | OPNFV feature project functional test suites |
224 +----------------+-----------------------------------------------+
225 | openstack | Advanced, long duration OpenStack tests |
226 | | (Tempest and Rally full suite). Each test may |
227 | | last several hours |
228 +----------------+-----------------------------------------------+
229 | vnf | Complex scenarios dealing with orchestration, |
230 | | VNF deployment and tests (e.g. vIMS) |
231 +----------------+-----------------------------------------------+
233 For the scenario validation, we consider only the healthcheck, smoke,
234 sdn_suites and features categories. These tests are run systematically
235 in the CI daily loops.
237 Success criteria have been defined for these test cases, they can be
238 PASS/FAIL or a success rate may be declared (100%, > 90%)
239 All the criteria, as well as the test dependencies are declared in the
240 ci/testcases.yaml file.
242 The scoring for the Colorado release per installer can be described as
245 The scoring is an indicator showing how many feature project test suites
246 have been integrated on the scenario.
248 The scoring equals the number of tests * succesful iteration of each
249 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
252 apex/odl_l2-nofeature-ha
253 tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
254 Scoring = 21/21 = 7 * 3
256 By default, if not specified, the scenarios are HA.
257 HA means OpenStack High Availability (main services). Note that not
258 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
259 depends upon support of the specific service for HA deployment.
260 For example, in the Colorado release, the Congress service
261 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
262 support for HA operation has not yet been verified.
263 See the release notes of the installers for more details.
269 +------------------+---------+---------+-------------------+
270 | Scenario | Scoring | Success | Results |
272 +==================+=========+=========+===================+
273 | nosdn | 17/18 | 95% | `apex-res-174`_ |
274 +------------------+---------+---------+-------------------+
275 | odl_l2 | 21/21 | 100% | `apex-res-175`_ |
276 +------------------+---------+---------+-------------------+
277 | odl_l3 | 15/18 | 83% | `apex-res-176`_ |
278 +------------------+---------+---------+-------------------+
279 | odl_l2-bgpvpn | 14/18 | 78% | ?? |
280 +------------------+---------+---------+-------------------+
281 | odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ |
282 +------------------+---------+---------+-------------------+
283 | odl_l2-sfc-noha | 18/21 | 86% | ?? |
284 +------------------+---------+---------+-------------------+
289 +------------------+---------+---------+-------------------+
290 | Scenario | Scoring | Success | Results |
292 +==================+=========+=========+===================+
293 | nosdn | 12/12 | 100% | `compass-res-55`_ |
294 +------------------+---------+---------+-------------------+
295 | odl_l2 | 15/15 | 100% | `compass-res-59`_ |
296 +------------------+---------+---------+-------------------+
297 | odl_l3 | 9/12 | 75% | `compass-res-73`_ |
298 +------------------+---------+---------+-------------------+
299 | odl_l2-moon | 9/18 | % | ?? |
300 +------------------+---------+---------+-------------------+
301 | onos-ha | 15/15 | 100% | `compass-res-77`_ |
302 +------------------+---------+---------+-------------------+
303 | onos-sfc-ha | 16/18 | 100% | `compass-res-76`_ |
304 +------------------+---------+---------+-------------------+
306 Note: all the Compass tests for Colorado have been executed on virtual
307 environment. Bare metal resources were used for Master branch.
313 +---------------------+---------+---------+-------------------+
314 | Scenario | Scoring | Success | Results |
316 +=====================+=========+=========+===================+
317 | nosdn | 18/18 | 100% | `fuel-res-129`_ |
318 +---------------------+---------+---------+-------------------+
319 | nosdn-noha | 15/15 | 100% | `fuel-res-154`_ |
320 +---------------------+---------+---------+-------------------+
321 | nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ |
322 +---------------------+---------+---------+-------------------+
323 | nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ |
324 +---------------------+---------+---------+-------------------+
325 | nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ |
326 +---------------------+---------+---------+-------------------+
327 | odl_l2-sfc | 21/21 | 100% | |
328 +---------------------+---------+---------+-------------------+
329 | odl_l2-sfc-noha | 16/18 | 88% | |
330 +---------------------+---------+---------+-------------------+
331 | odl_l2 | 21/21 | 100% | `fuel-res-123`_ |
332 +---------------------+---------+---------+-------------------+
333 | odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ |
334 +---------------------+---------+---------+-------------------+
335 | odl_l2-bgpvpn | 17/18 | 94% | `fuel-res-119`_ |
336 +---------------------+---------+---------+-------------------+
337 | odl_l2-bgpvpn-noha | 15/15 | 100% | `fuel-res-160`_ |
338 +---------------------+---------+---------+-------------------+
339 | odl_l3 | 15/18 | 67% | `fuel-res-115`_ |
340 +---------------------+---------+---------+-------------------+
341 | odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ |
342 +---------------------+---------+---------+-------------------+
343 | onos | 20/21 | 95% | `fuel-res-117`_ |
344 +---------------------+---------+---------+-------------------+
345 | onos-noha | 18/18 | 100% | `fuel-res-166`_ |
346 +---------------------+---------+---------+-------------------+
347 | onos-sfc | 24/24 | 100% | fuel-res-124`_ |
348 +---------------------+---------+---------+-------------------+
349 | onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ |
350 +---------------------+---------+---------+-------------------+
351 | multisite | N.R | 100% | `fuel-res-8`_ |
352 +---------------------+---------+---------+-------------------+
355 Results of Functest on AArch64 Colorado 1.0
357 +---------------------+---------+---------+----------------------+
358 | Scenario | Scoring | Success | Results |
360 +=====================+=========+=========+======================+
361 | nosdn | 18/18 | 100% | `fuel-arm-res-29`_ |
362 +---------------------+---------+---------+----------------------+
363 | odl_l2 | 21/21 | 100% | `fuel-arm-res-24`_ |
364 +---------------------+---------+---------+----------------------+
365 | odl_l2-noha | 17/18 | 94% | `fuel-arm-res-30`_ |
366 +---------------------+---------+---------+----------------------+
367 | odl_l3 | 14/18 | 78% | `fuel-arm-res-28`_ |
368 +---------------------+---------+---------+----------------------+
374 +---------------------+---------+---------+-----------------+
375 | Scenario | Scoring | Success | Results |
377 +=====================+=========+=========+=================+
378 | nosdn | 18/18 | 100% | `joid-res-102`_ |
379 +---------------------+---------+---------+-----------------+
380 | nosdn-noha | 17/18 | 95% | `joid-res-93`_ |
381 +---------------------+---------+---------+-----------------+
382 | nosdn-lxd | 12/12 | 100% | `joid-res-104`_ |
383 +---------------------+---------+---------+-----------------+
384 | nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ |
385 +---------------------+---------+---------+-----------------+
386 | odl_l2 | 19/21 | 91% | `joid-res-103`_ |
387 +---------------------+---------+---------+-----------------+
388 | onos | 21/21 | 100% | `joid-res-99`_ |
389 +---------------------+---------+---------+-----------------+
390 | onos-sfc | 24/24 | 100% | `joid-res-97`_ |
391 +---------------------+---------+---------+-----------------+
393 It is highly recommended to install a json viewer in your browser
394 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
396 You can get additional details through test logs on http://artifacts.opnfv.org/.
397 As no search engine is available on the OPNFV artifact web site you must
398 retrieve the pod identifier on which the tests have been executed (see
399 field pod in any of the results) then click on the selected POD and look
400 for the date of the test you are interested in.
402 The reporting pages can be found at:
404 * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
405 * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
406 * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
407 * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html
409 Colorado known restrictions/issues
410 ==================================
412 +-----------+-----------+----------------------------------------------+
413 | Installer | Scenario | Issue |
414 +===========+===========+==============================================+
415 | any | odl_l3-* | Tempest cases related to using floating IP |
416 | | | addresses fail because of a known ODL bug. |
417 | | | vPing_ssh test case is excluded for the same |
419 | | | https://jira.opnfv.org/browse/APEX-112 |
420 | | | https://jira.opnfv.org/browse/FUNCTEST-445 |
421 +-----------+-----------+----------------------------------------------+
422 | apex/fuel | *-bgpvpn | vPing_ssh (floating ips not supported) and |
423 | | | vIMS excluded. Some Tempest cases related to |
424 | | | floating ips also excluded. Some performance |
425 | | | issues have been detected in this scenario |
426 | | | (i.e. BGPVPN extension enabled) when running |
427 | | | commands against the OpenStack APIs, thus |
428 | | | Rally sanity test case has been disabled. |
429 | | | Performance issues seem to be connected to |
430 | | | the ODL version. It is planned to reintroduce|
431 | | | Rally sanity in Colorado 2.0 with the |
432 | | | adoption of ODL Boron release. |
433 +-----------+-----------+----------------------------------------------+
434 | apex | *-fdio | Due to late integration, fdio decided to |
435 | | | focus on mandatory tests and exclude feature |
436 | | | tests (copper, doctor, security_scan) from |
437 | | | its scenarios |
438 +-----------+-----------+----------------------------------------------+
439 | compass | moon | First ODL test FAILS because ODL/Openstack |
440 | | | federation done in moon is partial. Only |
441 | | | MD-SAL is federated (not AD-SAL) |
442 +-----------+-----------+----------------------------------------------+
443 | fuel | any | TestServerBasicOps test case skipped |
444 | | | https://gerrit.opnfv.org/gerrit/#/c/19635/ |
445 +-----------+-----------+----------------------------------------------+
446 | fuel | kvm | vPing_ssh and vIMS excluded (metadata related|
448 +-----------+-----------+----------------------------------------------+
449 | fuel | multisite | random errors running multisite. A known bug |
450 | | | in keystone mitaka, due to which memcache |
451 | | | raises exception and keystone becomes |
453 | | | bugs.launchpad.net/keystone/+bug/1600394 |
454 | | | workaround consists in restarting memcache on|
456 +-----------+-----------+----------------------------------------------+
457 | joid | any | Tempest cases related to object storage |
459 | | | https://gerrit.opnfv.org/gerrit/#/c/17871/ |
460 +-----------+-----------+----------------------------------------------+
461 | joid | domino | Domino tests are skipped in CI. However the |
462 | | | test case can be run by manually setting |
463 | | | IS_IPandKEY_CONFIGURED=true after manually |
464 | | | setting properly the IP addresses of the 3 |
465 | | | Controller nodes in the configuration file |
466 | | | /repos/domino/tests/run_multinode.sh |
467 +-----------+-----------+----------------------------------------------+
470 Test and installer/scenario dependencies
471 ========================================
473 It is not always possible to run all the test cases on all the scenarios.
474 The following table details the dependencies of the test cases per
475 scenario. The scenario dependencies (installer or scenario) are detailed
476 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
481 Test results are available in:
483 - test results document: http://artifacts.opnfv.org/functest
485 - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
487 - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
494 +------------------+-----------------------------------------------+
495 | JIRA | Description |
496 +==================+===============================================+
497 | `FUNCTEST-419`_ | do not try to Remove docker |
498 | | image opnfv/functest:<none> |
499 | | reported by joid on Intel POD |
501 | | not reproducible |
502 +------------------+-----------------------------------------------+
503 | `FUNCTEST-446`_ | Cleanup ODL-SFC output in Functest execution |
504 | | Impact on odl_l2-sfc scenarios |
505 +------------------+-----------------------------------------------+
506 | `FUNCTEST-450`_ | Functest is Failing to get the token using |
507 | | keystone client |
508 +------------------+-----------------------------------------------+
509 | `FUNCTEST-454`_ | Cleanup failures when using HA networks in |
511 +------------------+-----------------------------------------------+
512 | `FUNCTEST-460`_ | Wrong image format used in rally cases |
513 +------------------+-----------------------------------------------+
514 | `FUNCTEST-462`_ | OLD test fails after forcing the clone |
515 | | release/beryllium-sr3 branch |
516 +------------------+-----------------------------------------------+
522 - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
524 - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado
526 - Functest repo: https://git.opnfv.org/cgit/functest
528 - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
530 - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
532 - Functest IRC chan: #opnfv-functest
534 - Functest reporting: http://testresults.opnfv.org/reporting
536 - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
538 - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
540 - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
542 - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
544 .. _`FUNCTEST-419` : https://jira.opnfv.org/browse/FUNCTEST-419
546 .. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
548 .. _`FUNCTEST-450` : https://jira.opnfv.org/browse/FUNCTEST-450
550 .. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
552 .. _`FUNCTEST-460` : https://jira.opnfv.org/browse/FUNCTEST-460
554 .. _`FUNCTEST-462` : https://jira.opnfv.org/browse/FUNCTEST-462
556 .. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
558 .. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
560 .. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
562 .. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
564 .. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
566 .. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
568 .. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
570 .. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
572 .. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
574 .. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
576 .. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
578 .. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
580 .. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
582 .. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
584 .. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
586 .. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
588 .. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
590 .. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
592 .. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
594 .. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
596 .. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
598 .. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
600 .. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
602 .. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
604 .. _`fuel-arm-res-29`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/29/console
606 .. _`fuel-arm-res-24`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/24/console
608 .. _`fuel-arm-res-30`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/30/console
610 .. _`fuel-arm-res-28`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/28/console
612 .. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
614 .. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
616 .. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
618 .. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
620 .. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
622 .. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
624 .. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103