5 OPNFV Colorado release note for Functest Docs
6 (c) by Morgan Richomme (Orange)
8 OPNFV Colorado release note for Functest Docs
9 are licensed under a Creative Commons Attribution 4.0 International License.
10 You should have received a copy of the license along with this.
11 If not, see <http://creativecommons.org/licenses/by/4.0/>.
13 ===========================================
14 OPNFV Colorado1.0 release note for Functest
15 ===========================================
20 This document describes the release note of Functest project.
26 +------------+----------+------------------+------------------------+
27 | **Date** | **Ver.** | **Author** | **Comment** |
29 +------------+----------+------------------+------------------------+
30 | 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado |
31 | | | (Orange) | release |
32 +------------+----------+------------------+------------------------+
34 OPNFV Colorado Release
35 =========================
37 Functest deals with functional testing of the OPNFV solution.
38 It includes test cases developed within the project and test cases developed in
39 other OPNFV projects and other upstream communities.
41 The internal test cases are:
46 * Tempest Smoke Serial
49 * Tempest full parallel
53 The OPNFV projects integrated into Functest framework for automation are:
68 The validation of a scenario requires a subset of these tests depending
69 on the installer and the scenario.
71 The 3 last internal test cases (tempest full parallel, Rally full and
72 vIMS) are not considered for scenario validation.
77 +--------------------------------------+--------------------------------------+
78 | **Project** | functest |
80 +--------------------------------------+--------------------------------------+
81 | **Repo/tag** | colorado.1.0 |
83 +--------------------------------------+--------------------------------------+
84 | **Release designation** | Colorado base release |
86 +--------------------------------------+--------------------------------------+
87 | **Release date** | September 22 2016 |
89 +--------------------------------------+--------------------------------------+
90 | **Purpose of the delivery** | Colorado base release |
92 +--------------------------------------+--------------------------------------+
100 - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0)
102 - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0)
108 - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
110 - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
112 - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
121 - refactoring of ODL functional tests (with upstream modifications)
123 - refactoring of testapi (update, swagger documentation, dockerization)
125 - jenkins logs improvement
127 - update integration of Doctor, Promise and SDNVPN projects
129 - split Tempest and rally into 2 different tests: smoke and full
131 - vIMS test suite integration
133 - adoption of Kibana for dashboarding
139 - Functest CLI to prepare and run the tests
141 - creation of the healthcheck test case
143 - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
145 - integration of new OPNFV feature projects (copper, domino, multisite,
146 moon, parser, onos-sfc, odl-sfc, security scan)
148 - introduction of test tiers in functest framework
150 - automatic reporting
152 - introduction of a jenkins summary table
154 - support of ARM architecture
160 For Colorado 1.0, Functest was tested on the following scenarios (if not
161 precised, the scenario is a HA scenario):
163 +---------------------+---------+---------+---------+---------+
164 | Scenario | Apex | Compass | Fuel | Joid |
165 +=====================+=========+=========+=========+=========+
166 | nosdn | X | X | X | X |
167 +---------------------+---------+---------+---------+---------+
168 | nosdn-noha | | | X | X |
169 +---------------------+---------+---------+---------+---------+
171 +---------------------+---------+---------+---------+---------+
172 | kvm-noha | | | X | |
173 +---------------------+---------+---------+---------+---------+
175 +---------------------+---------+---------+---------+---------+
176 | lxd-noha | | | | X |
177 +---------------------+---------+---------+---------+---------+
178 | ovs-noha (dpdk) | | | X | |
179 +---------------------+---------+---------+---------+---------+
180 | odl_l2 | X | X | X | X |
181 +---------------------+---------+---------+---------+---------+
182 | odl_l2-noha | | | X | |
183 +---------------------+---------+---------+---------+---------+
184 | odl_l2-bgpvpn | X | | X | |
185 +---------------------+---------+---------+---------+---------+
186 | odl_l2-bgpvpn-noha| | | X | |
187 +---------------------+---------+---------+---------+---------+
188 | odl_l2-fdio-noha | X | | | |
189 +---------------------+---------+---------+---------+---------+
190 | odl_l2-moon | | X | | |
191 +---------------------+---------+---------+---------+---------+
192 | odl_l2-sfc-noha | X | | X | |
193 +---------------------+---------+---------+---------+---------+
194 | odl_l3 | X | X | X | |
195 +---------------------+---------+---------+---------+---------+
196 | odl_l3-noha | | | X | |
197 +---------------------+---------+---------+---------+---------+
198 | onos | | X | X | X |
199 +---------------------+---------+---------+---------+---------+
200 | onos-noha | | | X | |
201 +---------------------+---------+---------+---------+---------+
202 | onos-sfc | | X | X | X |
203 +---------------------+---------+---------+---------+---------+
204 | onos-sfc-noha | | | X | |
205 +---------------------+---------+---------+---------+---------+
206 | multisite | | | X | |
207 +---------------------+---------+---------+---------+---------+
209 For Colorado 2.0, the following scenarios have been released:
211 +---------------------+---------+---------+---------+---------+
212 | Scenario | Apex | Compass | Fuel | Joid |
213 +=====================+=========+=========+=========+=========+
214 | onos-nofeature | X | X | X | X |
215 +---------------------+---------+---------+---------+---------+
216 | odl_l2-sfc-ha | | | X | |
217 +---------------------+---------+---------+---------+---------+
220 In Colorado, the functional tests have been sliced in 6 different
223 +----------------+-----------------------------------------------+
224 | Category | Description |
225 +================+===============================================+
226 | healthcheck | Basic OpenStack commands |
227 +----------------+-----------------------------------------------+
228 | smoke | vPings, Tempest and rally smoke tests |
229 +----------------+-----------------------------------------------+
230 | sdn_suites | Specific SDN feature tests |
231 +----------------+-----------------------------------------------+
232 | features | OPNFV feature project functional test suites |
233 +----------------+-----------------------------------------------+
234 | openstack | Advanced, long duration OpenStack tests |
235 | | (Tempest and Rally full suite). Each test may |
236 | | last several hours |
237 +----------------+-----------------------------------------------+
238 | vnf | Complex scenarios dealing with orchestration, |
239 | | VNF deployment and tests (e.g. vIMS) |
240 +----------------+-----------------------------------------------+
242 For the scenario validation, we consider only the healthcheck, smoke,
243 sdn_suites and features categories. These tests are run systematically
244 in the CI daily loops.
246 Success criteria have been defined for these test cases, they can be
247 PASS/FAIL or a success rate may be declared (100%, > 90%)
248 All the criteria, as well as the test dependencies are declared in the
249 ci/testcases.yaml file.
251 The scoring for the Colorado release per installer can be described as
254 The scoring is an indicator showing how many feature project test suites
255 have been integrated on the scenario.
257 The scoring equals the number of tests * succesful iteration of each
258 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
261 apex/odl_l2-nofeature-ha
262 tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
263 Scoring = 21/21 = 7 * 3
265 By default, if not specified, the scenarios are HA.
266 HA means OpenStack High Availability (main services). Note that not
267 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
268 depends upon support of the specific service for HA deployment.
269 For example, in the Colorado release, the Congress service
270 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
271 support for HA operation has not yet been verified.
272 See the release notes of the installers for more details.
278 +------------------+---------+---------+-------------------+
279 | Scenario | Scoring | Success | Results |
281 +==================+=========+=========+===================+
282 | nosdn | 17/18 | 95% | `apex-res-174`_ |
283 +------------------+---------+---------+-------------------+
284 | odl_l2 | 21/21 | 100% | `apex-res-175`_ |
285 +------------------+---------+---------+-------------------+
286 | odl_l3 | 15/18 | 83% | `apex-res-176`_ |
287 +------------------+---------+---------+-------------------+
288 | odl_l2-bgpvpn | 14/18 | 78% | `apex-res-235`_ |
289 +------------------+---------+---------+-------------------+
290 | odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ |
291 +------------------+---------+---------+-------------------+
292 | odl_l2-sfc-noha | 18/24 | 75% | `apex-res-217`_ |
293 +------------------+---------+---------+-------------------+
294 | onos-nofeature-ha| 20/21 | 95% | `apex-res-423`_ |
295 +------------------+---------+---------+-------------------+
301 +------------------+---------+---------+-------------------+
302 | Scenario | Scoring | Success | Results |
304 +==================+=========+=========+===================+
305 | nosdn | 12/12 | 100% | `compass-res-55`_ |
306 +------------------+---------+---------+-------------------+
307 | odl_l2 | 15/15 | 100% | `compass-res-59`_ |
308 +------------------+---------+---------+-------------------+
309 | odl_l3 | 9/12 | 75% | `compass-res-73`_ |
310 +------------------+---------+---------+-------------------+
311 | odl_l2-moon | 15/18 | 83% | `compass-res-567`_|
312 +------------------+---------+---------+-------------------+
313 | onos-ha | 15/15 | 100% | `compass-res-77`_ |
314 +------------------+---------+---------+-------------------+
315 | onos-sfc-ha | 17/18 | 95% | `compass-res-76`_ |
316 +------------------+---------+---------+-------------------+
317 | onos-nofeature-ha| 15/15 | 100% | `compass-res-285`_|
318 +------------------+---------+---------+-------------------+
320 Note: all the Compass tests for Colorado have been executed on virtual
321 environment. Bare metal resources were used for Master branch.
327 +---------------------+---------+---------+-------------------+
328 | Scenario | Scoring | Success | Results |
330 +=====================+=========+=========+===================+
331 | nosdn | 18/18 | 100% | `fuel-res-129`_ |
332 +---------------------+---------+---------+-------------------+
333 | nosdn-noha | 15/15 | 100% | `fuel-res-154`_ |
334 +---------------------+---------+---------+-------------------+
335 | nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ |
336 +---------------------+---------+---------+-------------------+
337 | nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ |
338 +---------------------+---------+---------+-------------------+
339 | nosdn-ovs | 12/18 | 67%* | `fuel-res-213`_ |
340 +---------------------+---------+---------+-------------------+
341 | nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ |
342 +---------------------+---------+---------+-------------------+
343 | odl_l2 | 21/21 | 100% | `fuel-res-123`_ |
344 +---------------------+---------+---------+-------------------+
345 | odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ |
346 +---------------------+---------+---------+-------------------+
347 | odl_l2-bgpvpn | 14/18 | 78% | `fuel-res-119`_ |
348 +---------------------+---------+---------+-------------------+
349 | odl_l2-bgpvpn-noha | 14/15 | 93% | `fuel-res-160`_ |
350 +---------------------+---------+---------+-------------------+
351 | odl_l2-sfc-noha | 6/21 | 29% | `fuel-res-219`_ |
352 +---------------------+---------+---------+-------------------+
353 | odl_l2-sfc-ha | 16/21 | 76% | `fuel-res-376`_ |
354 +---------------------+---------+---------+-------------------+
355 | odl_l3 | 15/18 | 83% | `fuel-res-115`_ |
356 +---------------------+---------+---------+-------------------+
357 | odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ |
358 +---------------------+---------+---------+-------------------+
359 | onos | 20/21 | 95% | `fuel-res-117`_ |
360 +---------------------+---------+---------+-------------------+
361 | onos-noha | 18/18 | 100% | `fuel-res-166`_ |
362 +---------------------+---------+---------+-------------------+
363 | onos-sfc | 24/24 | 100% | `fuel-res-124`_ |
364 +---------------------+---------+---------+-------------------+
365 | onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ |
366 +---------------------+---------+---------+-------------------+
367 | onos-nofeature-ha | 18/18 | 100% | `fuel-res-492`_ |
368 +---------------------+---------+---------+-------------------+
369 | multisite | N.R | 100% | `fuel-res-8`_ |
370 +---------------------+---------+---------+-------------------+
372 *: all results passed, lacking iterations to reach the full score
374 Results of Functest on AArch64 Colorado 1.0
376 +---------------------+---------+---------+----------------------+
377 | Scenario | Scoring | Success | Results |
379 +=====================+=========+=========+======================+
380 | nosdn | 18/18 | 100% | `fuel-arm-res-29`_ |
381 +---------------------+---------+---------+----------------------+
382 | odl_l2 | 21/21 | 100% | `fuel-arm-res-24`_ |
383 +---------------------+---------+---------+----------------------+
384 | odl_l2-noha | 17/18 | 94% | `fuel-arm-res-30`_ |
385 +---------------------+---------+---------+----------------------+
386 | odl_l3 | 14/18 | 78% | `fuel-arm-res-28`_ |
387 +---------------------+---------+---------+----------------------+
393 +---------------------+---------+---------+-----------------+
394 | Scenario | Scoring | Success | Results |
396 +=====================+=========+=========+=================+
397 | nosdn | 18/18 | 100% | `joid-res-102`_ |
398 +---------------------+---------+---------+-----------------+
399 | nosdn-noha | 17/18 | 95% | `joid-res-93`_ |
400 +---------------------+---------+---------+-----------------+
401 | nosdn-lxd | 12/12 | 100% | `joid-res-104`_ |
402 +---------------------+---------+---------+-----------------+
403 | nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ |
404 +---------------------+---------+---------+-----------------+
405 | odl_l2 | 21/21 | 100% | `joid-res-103`_ |
406 +---------------------+---------+---------+-----------------+
407 | onos | 21/21 | 100% | `joid-res-99`_ |
408 +---------------------+---------+---------+-----------------+
409 | onos-sfc | 24/24 | 100% | `joid-res-97`_ |
410 +---------------------+---------+---------+-----------------+
411 | onos-nofeature-ha | 21/21 | 100% | `joid-res-345`_ |
412 +---------------------+---------+---------+-----------------+
414 It is highly recommended to install a json viewer in your browser
415 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
417 You can get additional details through test logs on http://artifacts.opnfv.org/.
418 As no search engine is available on the OPNFV artifact web site you must
419 retrieve the pod identifier on which the tests have been executed (see
420 field pod in any of the results) then click on the selected POD and look
421 for the date of the test you are interested in.
423 The reporting pages can be found at:
425 * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
426 * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
427 * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
428 * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html
430 Colorado known restrictions/issues
431 ==================================
433 +-----------+-----------+----------------------------------------------+
434 | Installer | Scenario | Issue |
435 +===========+===========+==============================================+
436 | any | odl_l3-* | Tempest cases related to using floating IP |
437 | | | addresses fail because of a known ODL bug. |
438 | | | vPing_ssh test case is excluded for the same |
440 | | | https://jira.opnfv.org/browse/APEX-112 |
441 | | | https://jira.opnfv.org/browse/FUNCTEST-445 |
442 +-----------+-----------+----------------------------------------------+
443 | apex/fuel | *-bgpvpn | vPing_ssh (floating ips not supported) and |
444 | | | vIMS excluded. Some Tempest cases related to |
445 | | | floating ips also excluded. Some performance |
446 | | | issues have been detected in this scenario |
447 | | | (i.e. BGPVPN extension enabled) when running |
448 | | | commands against the OpenStack APIs, thus |
449 | | | Rally sanity test case has been disabled. |
450 | | | Performance issues seem to be connected to |
451 | | | the ODL version. It is planned to reintroduce|
452 | | | Rally sanity in Colorado 2.0 with the |
453 | | | adoption of ODL Boron release. |
454 +-----------+-----------+----------------------------------------------+
455 | apex | *-fdio | Due to late integration, fdio decided to |
456 | | | focus on mandatory tests and exclude feature |
457 | | | tests (copper, doctor, security_scan) from |
458 | | | its scenarios |
459 +-----------+-----------+----------------------------------------------+
460 | compass | moon | First ODL test FAILS because ODL/Openstack |
461 | | | federation done in moon is partial. Only |
462 | | | MD-SAL is federated (not AD-SAL) |
463 +-----------+-----------+----------------------------------------------+
464 | fuel | any | TestServerBasicOps test case skipped |
465 | | | https://gerrit.opnfv.org/gerrit/#/c/19635/ |
466 +-----------+-----------+----------------------------------------------+
467 | fuel | kvm | vPing_ssh and vIMS excluded (metadata related|
469 +-----------+-----------+----------------------------------------------+
470 | fuel | multisite | random errors running multisite. A known bug |
471 | | | in keystone mitaka, due to which memcache |
472 | | | raises exception and keystone becomes |
474 | | | bugs.launchpad.net/keystone/+bug/1600394 |
475 | | | workaround consists in restarting memcache on|
477 +-----------+-----------+----------------------------------------------+
478 | joid | any | Tempest cases related to object storage |
480 | | | https://gerrit.opnfv.org/gerrit/#/c/17871/ |
481 +-----------+-----------+----------------------------------------------+
482 | joid | domino | Domino tests are skipped in CI. However the |
483 | | | test case can be run by manually setting |
484 | | | IS_IPandKEY_CONFIGURED=true after manually |
485 | | | setting properly the IP addresses of the 3 |
486 | | | Controller nodes in the configuration file |
487 | | | /repos/domino/tests/run_multinode.sh |
488 +-----------+-----------+----------------------------------------------+
491 Test and installer/scenario dependencies
492 ========================================
494 It is not always possible to run all the test cases on all the scenarios.
495 The following table details the dependencies of the test cases per
496 scenario. The scenario dependencies (installer or scenario) are detailed
497 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
502 Test results are available in:
504 - test results document: http://artifacts.opnfv.org/functest
506 - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
508 - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
515 +------------------+-----------------------------------------------+
516 | JIRA | Description |
517 +==================+===============================================+
518 | `FUNCTEST-446`_ | Cleanup ODL-SFC output in Functest execution |
519 | | Impact on odl_l2-sfc scenarios |
520 +------------------+-----------------------------------------------+
521 | `FUNCTEST-454`_ | Cleanup failures when using HA networks in |
523 +------------------+-----------------------------------------------+
528 - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
530 - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado
532 - Functest repo: https://git.opnfv.org/cgit/functest
534 - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
536 - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
538 - Functest IRC chan: #opnfv-functest
540 - Functest reporting: http://testresults.opnfv.org/reporting
542 - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
544 - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
546 - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
548 - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
550 .. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
552 .. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
554 .. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
556 .. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
558 .. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
560 .. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
562 .. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
564 .. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
566 .. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423
568 .. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
570 .. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
572 .. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
574 .. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
576 .. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
578 .. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
580 .. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285
582 .. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
584 .. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
586 .. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
588 .. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
590 .. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
592 .. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
594 .. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
596 .. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
598 .. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
600 .. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
602 .. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
604 .. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
606 .. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
608 .. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
610 .. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
612 .. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213
614 .. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219
616 .. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376
618 .. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492
620 .. _`fuel-arm-res-29`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/29/console
622 .. _`fuel-arm-res-24`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/24/console
624 .. _`fuel-arm-res-30`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/30/console
626 .. _`fuel-arm-res-28`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/28/console
628 .. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
630 .. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
632 .. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
634 .. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
636 .. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
638 .. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
640 .. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
642 .. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345