5 OPNFV Colorado release note for Functest Docs
6 (c) by Morgan Richomme (Orange)
8 OPNFV Colorado release note for Functest Docs
9 are licensed under a Creative Commons Attribution 4.0 International License.
10 You should have received a copy of the license along with this.
11 If not, see <http://creativecommons.org/licenses/by/4.0/>.
13 ===========================================
14 OPNFV Colorado1.0 release note for Functest
15 ===========================================
20 This document describes the release note of Functest project.
26 +------------+----------+------------------+------------------------+
27 | **Date** | **Ver.** | **Author** | **Comment** |
29 +------------+----------+------------------+------------------------+
30 | 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado |
31 | | | (Orange) | release |
32 +------------+----------+------------------+------------------------+
34 OPNFV Colorado Release
35 =========================
37 Functest deals with functional testing of the OPNFV solution.
38 It includes test cases developed within the project and test cases developed in
39 other OPNFV projects and other upstream communities.
41 The internal test cases are:
46 * Tempest Smoke Serial
49 * Tempest full parallel
53 The OPNFV projects integrated into Functest framework for automation are:
68 The validation of a scenario requires a subset of these tests depending
69 on the installer and the scenario.
71 The 3 last internal test cases (tempest full parallel, Rally full and
72 vIMS) are not considered for scenario validation.
77 +--------------------------------------+--------------------------------------+
78 | **Project** | functest |
80 +--------------------------------------+--------------------------------------+
81 | **Repo/tag** | colorado.1.0 |
83 +--------------------------------------+--------------------------------------+
84 | **Release designation** | Colorado base release |
86 +--------------------------------------+--------------------------------------+
87 | **Release date** | September 22 2016 |
89 +--------------------------------------+--------------------------------------+
90 | **Purpose of the delivery** | Colorado base release |
92 +--------------------------------------+--------------------------------------+
100 - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0)
102 - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0)
108 - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
110 - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
112 - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
121 - refactoring of ODL functional tests (with upstream modifications)
123 - refactoring of testapi (update, swagger documentation, dockerization)
125 - jenkins logs improvement
127 - update integration of Doctor, Promise and SDNVPN projects
129 - split Tempest and rally into 2 different tests: smoke and full
131 - vIMS test suite integration
133 - adoption of Kibana for dashboarding
139 - Functest CLI to prepare and run the tests
141 - creation of the healthcheck test case
143 - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
145 - integration of new OPNFV feature projects (copper, domino, multisite,
146 moon, parser, onos-sfc, odl-sfc, security scan)
148 - introduction of test tiers in functest framework
150 - automatic reporting
152 - introduction of a jenkins summary table
154 - support of ARM architecture
160 For Colorado 1.0, Functest was tested on the following scenarios (if not
161 precised, the scenario is a HA scenario):
163 +---------------------+---------+---------+---------+---------+
164 | Scenario | Apex | Compass | Fuel | Joid |
165 +=====================+=========+=========+=========+=========+
166 | nosdn | X | X | X | X |
167 +---------------------+---------+---------+---------+---------+
168 | nosdn-noha | | | X | X |
169 +---------------------+---------+---------+---------+---------+
171 +---------------------+---------+---------+---------+---------+
172 | kvm-noha | | | X | |
173 +---------------------+---------+---------+---------+---------+
175 +---------------------+---------+---------+---------+---------+
176 | lxd-noha | | | | X |
177 +---------------------+---------+---------+---------+---------+
178 | ovs-noha (dpdk) | | | X | |
179 +---------------------+---------+---------+---------+---------+
180 | odl_l2 | X | X | X | X |
181 +---------------------+---------+---------+---------+---------+
182 | odl_l2-noha | | | X | |
183 +---------------------+---------+---------+---------+---------+
184 | odl_l2-bgpvpn | X | | X | |
185 +---------------------+---------+---------+---------+---------+
186 | odl_l2-bgpvpn-noha| | | X | |
187 +---------------------+---------+---------+---------+---------+
188 | odl_l2-fdio-noha | X | | | |
189 +---------------------+---------+---------+---------+---------+
190 | odl_l2-moon | | X | | |
191 +---------------------+---------+---------+---------+---------+
192 | odl_l2-sfc-noha | X | | X | |
193 +---------------------+---------+---------+---------+---------+
194 | odl_l3 | X | X | X | |
195 +---------------------+---------+---------+---------+---------+
196 | odl_l3-noha | | | X | |
197 +---------------------+---------+---------+---------+---------+
198 | onos | | X | X | X |
199 +---------------------+---------+---------+---------+---------+
200 | onos-noha | | | X | |
201 +---------------------+---------+---------+---------+---------+
202 | onos-sfc | | X | X | X |
203 +---------------------+---------+---------+---------+---------+
204 | onos-sfc-noha | | | X | |
205 +---------------------+---------+---------+---------+---------+
206 | multisite | | | X | |
207 +---------------------+---------+---------+---------+---------+
209 For Colorado 2.0, the following scenarios have been re-released:
211 +---------------------+---------+---------+---------+---------+
212 | Scenario | Apex | Compass | Fuel | Joid |
213 +=====================+=========+=========+=========+=========+
214 | onos | X | X | X | X |
215 +---------------------+---------+---------+---------+---------+
216 | odl_l2-sfc-ha | | | X | |
217 +---------------------+---------+---------+---------+---------+
220 In Colorado, the functional tests have been sliced in 6 different
223 +----------------+-----------------------------------------------+
224 | Category | Description |
225 +================+===============================================+
226 | healthcheck | Basic OpenStack commands |
227 +----------------+-----------------------------------------------+
228 | smoke | vPings, Tempest and rally smoke tests |
229 +----------------+-----------------------------------------------+
230 | sdn_suites | Specific SDN feature tests |
231 +----------------+-----------------------------------------------+
232 | features | OPNFV feature project functional test suites |
233 +----------------+-----------------------------------------------+
234 | openstack | Advanced, long duration OpenStack tests |
235 | | (Tempest and Rally full suite). Each test may |
236 | | last several hours |
237 +----------------+-----------------------------------------------+
238 | vnf | Complex scenarios dealing with orchestration, |
239 | | VNF deployment and tests (e.g. vIMS) |
240 +----------------+-----------------------------------------------+
242 For the scenario validation, we consider only the healthcheck, smoke,
243 sdn_suites and features categories. These tests are run systematically
244 in the CI daily loops.
246 Success criteria have been defined for these test cases, they can be
247 PASS/FAIL or a success rate may be declared (100%, > 90%)
248 All the criteria, as well as the test dependencies are declared in the
249 ci/testcases.yaml file.
251 The scoring for the Colorado release per installer can be described as
254 The scoring is an indicator showing how many feature project test suites
255 have been integrated on the scenario.
257 The scoring equals the number of tests * succesful iteration of each
258 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
261 apex/odl_l2-nofeature-ha
262 tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
263 Scoring = 21/21 = 7 * 3
265 By default, if not specified, the scenarios are HA.
266 HA means OpenStack High Availability (main services). Note that not
267 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
268 depends upon support of the specific service for HA deployment.
269 For example, in the Colorado release, the Congress service
270 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
271 support for HA operation has not yet been verified.
272 See the release notes of the installers for more details.
278 +------------------+---------+---------+-------------------+
279 | Scenario | Scoring | Success | Results |
281 +==================+=========+=========+===================+
282 | nosdn | 17/18 | 95% | `apex-res-174`_ |
283 +------------------+---------+---------+-------------------+
284 | odl_l2 | 21/21 | 100% | `apex-res-175`_ |
285 +------------------+---------+---------+-------------------+
286 | odl_l3 | 15/18 | 83% | `apex-res-176`_ |
287 +------------------+---------+---------+-------------------+
288 | odl_l2-bgpvpn | 14/18 | 78% | `apex-res-235`_ |
289 +------------------+---------+---------+-------------------+
290 | odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ |
291 +------------------+---------+---------+-------------------+
292 | odl_l2-sfc-noha | 18/24 | 75% | `apex-res-217`_ |
293 +------------------+---------+---------+-------------------+
294 | onos-nofeature-ha| 20/21 | 95% | `apex-res-423`_ |
295 +------------------+---------+---------+-------------------+
301 +------------------+---------+---------+-------------------+
302 | Scenario | Scoring | Success | Results |
304 +==================+=========+=========+===================+
305 | nosdn | 12/12 | 100% | `compass-res-55`_ |
306 +------------------+---------+---------+-------------------+
307 | odl_l2 | 15/15 | 100% | `compass-res-59`_ |
308 +------------------+---------+---------+-------------------+
309 | odl_l3 | 9/12 | 75% | `compass-res-73`_ |
310 +------------------+---------+---------+-------------------+
311 | odl_l2-moon | 15/18 | 83% | `compass-res-567`_|
312 +------------------+---------+---------+-------------------+
313 | onos-ha | 15/15 | 100% | `compass-res-285`_|
314 +------------------+---------+---------+-------------------+
315 | onos-sfc-ha | 17/18 | 95% | `compass-res-76`_ |
316 +------------------+---------+---------+-------------------+
318 Note: all the Compass tests for Colorado have been executed on virtual
319 environment. Bare metal resources were used for Master branch.
325 +---------------------+---------+---------+-------------------+
326 | Scenario | Scoring | Success | Results |
328 +=====================+=========+=========+===================+
329 | nosdn | 18/18 | 100% | `fuel-res-129`_ |
330 +---------------------+---------+---------+-------------------+
331 | nosdn-noha | 15/15 | 100% | `fuel-res-154`_ |
332 +---------------------+---------+---------+-------------------+
333 | nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ |
334 +---------------------+---------+---------+-------------------+
335 | nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ |
336 +---------------------+---------+---------+-------------------+
337 | nosdn-ovs | 12/18 | 67%* | `fuel-res-213`_ |
338 +---------------------+---------+---------+-------------------+
339 | nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ |
340 +---------------------+---------+---------+-------------------+
341 | odl_l2 | 21/21 | 100% | `fuel-res-123`_ |
342 +---------------------+---------+---------+-------------------+
343 | odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ |
344 +---------------------+---------+---------+-------------------+
345 | odl_l2-bgpvpn | 14/18 | 78% | `fuel-res-119`_ |
346 +---------------------+---------+---------+-------------------+
347 | odl_l2-bgpvpn-noha | 14/15 | 93% | `fuel-res-160`_ |
348 +---------------------+---------+---------+-------------------+
349 | odl_l2-sfc-noha | 6/21 | 29% | `fuel-res-219`_ |
350 +---------------------+---------+---------+-------------------+
351 | odl_l2-sfc-ha | 16/21 | 76% | `fuel-res-376`_ |
352 +---------------------+---------+---------+-------------------+
353 | odl_l3 | 15/18 | 83% | `fuel-res-115`_ |
354 +---------------------+---------+---------+-------------------+
355 | odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ |
356 +---------------------+---------+---------+-------------------+
357 | onos | 20/21 | 95% | `fuel-res-492`_ |
358 +---------------------+---------+---------+-------------------+
359 | onos-noha | 18/18 | 100% | `fuel-res-166`_ |
360 +---------------------+---------+---------+-------------------+
361 | onos-sfc | 24/24 | 100% | `fuel-res-124`_ |
362 +---------------------+---------+---------+-------------------+
363 | onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ |
364 +---------------------+---------+---------+-------------------+
365 | multisite | N.R | 100% | `fuel-res-8`_ |
366 +---------------------+---------+---------+-------------------+
368 *: all results passed, lacking iterations to reach the full score
370 Results of Functest on AArch64 Colorado 3.0
372 +---------------------+---------+---------+----------------------+
373 | Scenario | Scoring | Success | Results |
375 +=====================+=========+=========+======================+
376 | nosdn | 18/18 | 100% | `fuel-arm-res-128`_ |
377 +---------------------+---------+---------+----------------------+
378 | odl_l2 | 21/21 | 100% | `fuel-arm-res-122`_ |
379 +---------------------+---------+---------+----------------------+
380 | odl_l2-noha | 18/18 | 100% | `fuel-arm-res-129`_ |
381 +---------------------+---------+---------+----------------------+
382 | odl_l3 | 17/18 | 95% | `fuel-arm-res-135`_ |
383 +---------------------+---------+---------+----------------------+
389 +---------------------+---------+---------+-----------------+
390 | Scenario | Scoring | Success | Results |
392 +=====================+=========+=========+=================+
393 | nosdn | 18/18 | 100% | `joid-res-102`_ |
394 +---------------------+---------+---------+-----------------+
395 | nosdn-noha | 17/18 | 95% | `joid-res-93`_ |
396 +---------------------+---------+---------+-----------------+
397 | nosdn-lxd | 12/12 | 100% | `joid-res-104`_ |
398 +---------------------+---------+---------+-----------------+
399 | nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ |
400 +---------------------+---------+---------+-----------------+
401 | odl_l2 | 21/21 | 100% | `joid-res-103`_ |
402 +---------------------+---------+---------+-----------------+
403 | onos | 21/21 | 100% | `joid-res-345`_ |
404 +---------------------+---------+---------+-----------------+
405 | onos-sfc | 24/24 | 100% | `joid-res-97`_ |
406 +---------------------+---------+---------+-----------------+
408 It is highly recommended to install a json viewer in your browser
409 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
411 You can get additional details through test logs on http://artifacts.opnfv.org/.
412 As no search engine is available on the OPNFV artifact web site you must
413 retrieve the pod identifier on which the tests have been executed (see
414 field pod in any of the results) then click on the selected POD and look
415 for the date of the test you are interested in.
417 The reporting pages can be found at:
419 * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
420 * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
421 * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
422 * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html
424 Colorado known restrictions/issues
425 ==================================
427 +-----------+-----------+----------------------------------------------+
428 | Installer | Scenario | Issue |
429 +===========+===========+==============================================+
430 | any | odl_l3-* | Tempest cases related to using floating IP |
431 | | | addresses fail because of a known ODL bug. |
432 | | | vPing_ssh test case is excluded for the same |
434 | | | https://jira.opnfv.org/browse/APEX-112 |
435 | | | https://jira.opnfv.org/browse/FUNCTEST-445 |
436 +-----------+-----------+----------------------------------------------+
437 | apex/fuel | *-bgpvpn | vPing_ssh (floating ips not supported) and |
438 | | | vIMS excluded. Some Tempest cases related to |
439 | | | floating ips also excluded. Some performance |
440 | | | issues have been detected in this scenario |
441 | | | (i.e. BGPVPN extension enabled) when running |
442 | | | commands against the OpenStack APIs, thus |
443 | | | Rally sanity test case has been disabled. |
444 | | | Performance issues seem to be connected to |
445 | | | the ODL version. It is planned to reintroduce|
446 | | | Rally sanity in Colorado 2.0 with the |
447 | | | adoption of ODL Boron release. |
448 +-----------+-----------+----------------------------------------------+
449 | apex | *-fdio | Due to late integration, fdio decided to |
450 | | | focus on mandatory tests and exclude feature |
451 | | | tests (copper, doctor, security_scan) from |
452 | | | its scenarios |
453 +-----------+-----------+----------------------------------------------+
454 | compass | moon | First ODL test FAILS because ODL/Openstack |
455 | | | federation done in moon is partial. Only |
456 | | | MD-SAL is federated (not AD-SAL) |
457 +-----------+-----------+----------------------------------------------+
458 | fuel | any | TestServerBasicOps test case skipped |
459 | | | https://gerrit.opnfv.org/gerrit/#/c/19635/ |
460 +-----------+-----------+----------------------------------------------+
461 | fuel | kvm | vPing_ssh and vIMS excluded (metadata related|
463 +-----------+-----------+----------------------------------------------+
464 | fuel | multisite | random errors running multisite. A known bug |
465 | | | in keystone mitaka, due to which memcache |
466 | | | raises exception and keystone becomes |
468 | | | bugs.launchpad.net/keystone/+bug/1600394 |
469 | | | workaround consists in restarting memcache on|
471 +-----------+-----------+----------------------------------------------+
472 | joid | any | Tempest cases related to object storage |
474 | | | https://gerrit.opnfv.org/gerrit/#/c/17871/ |
475 +-----------+-----------+----------------------------------------------+
476 | joid | domino | Domino tests are skipped in CI. However the |
477 | | | test case can be run by manually setting |
478 | | | IS_IPandKEY_CONFIGURED=true after manually |
479 | | | setting properly the IP addresses of the 3 |
480 | | | Controller nodes in the configuration file |
481 | | | /repos/domino/tests/run_multinode.sh |
482 +-----------+-----------+----------------------------------------------+
485 Test and installer/scenario dependencies
486 ========================================
488 It is not always possible to run all the test cases on all the scenarios.
489 The following table details the dependencies of the test cases per
490 scenario. The scenario dependencies (installer or scenario) are detailed
491 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
496 Test results are available in:
498 - test results document: http://artifacts.opnfv.org/functest
500 - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
502 - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
509 +------------------+-----------------------------------------------+
510 | JIRA | Description |
511 +==================+===============================================+
512 | `FUNCTEST-446`_ | Cleanup ODL-SFC output in Functest execution |
513 | | Impact on odl_l2-sfc scenarios |
514 +------------------+-----------------------------------------------+
515 | `FUNCTEST-454`_ | Cleanup failures when using HA networks in |
517 +------------------+-----------------------------------------------+
522 - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
524 - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado
526 - Functest repo: https://git.opnfv.org/cgit/functest
528 - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
530 - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
532 - Functest IRC chan: #opnfv-functest
534 - Functest reporting: http://testresults.opnfv.org/reporting
536 - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
538 - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
540 - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
542 - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
544 .. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
546 .. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
548 .. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
550 .. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
552 .. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
554 .. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
556 .. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
558 .. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
560 .. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423
562 .. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
564 .. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
566 .. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
568 .. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
570 .. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
572 .. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
574 .. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285
576 .. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
578 .. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
580 .. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
582 .. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
584 .. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
586 .. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
588 .. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
590 .. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
592 .. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
594 .. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
596 .. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
598 .. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
600 .. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
602 .. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
604 .. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
606 .. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213
608 .. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219
610 .. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376
612 .. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492
614 .. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console
616 .. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console
618 .. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console
620 .. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console
622 .. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
624 .. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
626 .. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
628 .. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
630 .. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
632 .. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
634 .. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
636 .. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345