1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
8 OPNFV Danube release note for Functest Docs
9 (c) by Jose Lausuch (Ericsson)
11 OPNFV Danube release note for Functest Docs
12 are licensed under a Creative Commons Attribution 4.0 International License.
13 You should have received a copy of the license along with this.
14 If not, see <http://creativecommons.org/licenses/by/4.0/>.
16 ===========================================
17 OPNFV Danube2.0 release note for Functest
18 ===========================================
23 This document describes the release note of Functest project.
29 +------------+----------+------------------+------------------------+
30 | **Date** | **Ver.** | **Author** | **Comment** |
32 +------------+----------+------------------+------------------------+
33 | 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for |
34 | | | (Orange) | Colorado.1.0 release |
35 +------------+----------+------------------+------------------------+
36 | 2016-10-24 | 2.0.0 | Morgan Richomme | Functest for |
37 | | | (Orange) | Colorado.2.0 release |
38 +------------+----------+------------------+------------------------+
39 | 2016-08-17 | 3.0.0 | Morgan Richomme | Functest for |
40 | | | (Orange) | Colorado.3.0 release |
41 +------------+----------+------------------+------------------------+
42 | 2017-03-29 | 4.0.0 | Jose Lausuch | Functest for |
43 | | | (Ericsson) | Danube.1.0 release |
44 +------------+----------+------------------+------------------------+
45 | 2017-05-04 | 5.0.0 | Jose Lausuch | Functest for |
46 | | | (Ericsson) | Danube.2.0 release |
47 +------------+----------+------------------+------------------------+
48 | 2017-07-12 | 6.0.0 | Jose Lausuch | Functest for |
49 | | | (Ericsson) | Danube.3.0 release |
50 +------------+----------+------------------+------------------------+
53 ======================
55 Functest deals with functional testing of the OPNFV solution.
56 It includes test cases developed within the project and test cases developed in
57 other OPNFV projects and other upstream communities.
59 The internal test cases are:
66 * tempest_smoke_serial
71 * tempest_full_parallel
75 The OPNFV projects integrated into Functest framework for automation are:
92 The validation of a scenario requires a subset of these tests depending
93 on the installer and the scenario.
95 The test cases from vnf (cloudify_ims, orchestra_ims, vyos_vrouter) and
96 component categories (tempest full parallel, Rally full) are not considered for
102 +--------------------------------------+--------------------------------------+
103 | **Project** | functest |
105 +--------------------------------------+--------------------------------------+
106 | **Repo/tag** | danube.3.0 |
108 +--------------------------------------+--------------------------------------+
109 | **Release designation** | Danube service release |
111 +--------------------------------------+--------------------------------------+
112 | **Release date** | July 14th 2017 |
114 +--------------------------------------+--------------------------------------+
115 | **Purpose of the delivery** | Danube third release |
117 +--------------------------------------+--------------------------------------+
125 - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: danube.3.0)
127 - The TestAPI Docker image: https://hub.docker.com/r/opnfv/testapi (tag:danube.3.0)
133 - Installation/configuration guide: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/configguide/index.html
135 - User Guide: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/userguide/index.html
137 - Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/developer/devguide/index.html
144 - Bugfixes in refstack client
146 - Fixed vIMS onboarding and ims deployment
148 - Increased timeout of OpenBaton installation
150 - Switched to mitaka-eol in openstack repositories, as stable/mitake is not longer existing
152 - Minor fixes in logging and test cases
154 - 3 scenarios have been successfully executed on aarch64 architectures on Fuel deployments (nosdn, odl_l2 and odl_l3)
160 For Danube 3.0, Functest was tested on the following HA scenarios (new
161 dabube scenarios in bold):
163 +---------------------+---------+---------+---------+---------+
164 | Scenario | Apex | Compass | Fuel | Joid |
165 +=====================+=========+=========+=========+=========+
166 | nosdn | X | X | X | X |
167 +---------------------+---------+---------+---------+---------+
168 | **fdio** | X | | | |
169 +---------------------+---------+---------+---------+---------+
171 +---------------------+---------+---------+---------+---------+
172 | **kvm_ovs_dpdk** | | | X | |
173 +---------------------+---------+---------+---------+---------+
174 | **kvm_ovs_dpdk-bar**| | | X | |
175 +---------------------+---------+---------+---------+---------+
177 +---------------------+---------+---------+---------+---------+
178 | **ovs** | | | X | |
179 +---------------------+---------+---------+---------+---------+
180 | **openo** | | X | | |
181 +---------------------+---------+---------+---------+---------+
182 | odl_l2 | | X | X | X |
183 +---------------------+---------+---------+---------+---------+
184 | odl-bgpvpn | X | | | |
185 +---------------------+---------+---------+---------+---------+
186 | odl_l2-bgpvpn | | | X | |
187 +---------------------+---------+---------+---------+---------+
188 | **odl_l2-fdio** | X | | | |
189 +---------------------+---------+---------+---------+---------+
190 | odl_l2-sfc | | | X | |
191 +---------------------+---------+---------+---------+---------+
192 | odl_l3 | X | X | X | |
193 +---------------------+---------+---------+---------+---------+
194 | **ocl** | | X | | |
195 +---------------------+---------+---------+---------+---------+
197 +---------------------+---------+---------+---------+---------+
198 | multisite | | | X | |
199 +---------------------+---------+---------+---------+---------+
203 +---------------------+---------+---------+---------+---------+
204 | Scenario | Apex | Compass | Fuel | Joid |
205 +=====================+=========+=========+=========+=========+
206 | nosdn | | | X | X |
207 +---------------------+---------+---------+---------+---------+
209 +---------------------+---------+---------+---------+---------+
210 | **kvm_ovs_dpdk** | | | X | |
211 +---------------------+---------+---------+---------+---------+
212 | **kvm_ovs_dpdk-bar**| | | X | |
213 +---------------------+---------+---------+---------+---------+
215 +---------------------+---------+---------+---------+---------+
217 +---------------------+---------+---------+---------+---------+
219 +---------------------+---------+---------+---------+---------+
220 | odl_l2-bgpvpn | | | X | |
221 +---------------------+---------+---------+---------+---------+
222 | odl_l2-fdio | X | | | |
223 +---------------------+---------+---------+---------+---------+
224 | **odl_l3-fdio** | X | | | |
225 +---------------------+---------+---------+---------+---------+
226 | odl_l2-sfc | | | X | |
227 +---------------------+---------+---------+---------+---------+
229 +---------------------+---------+---------+---------+---------+
230 | **odl_gluon** | X | | | |
231 +---------------------+---------+---------+---------+---------+
233 Colorado deprecated scenarios:
240 For the scenario validation, we consider only the healthcheck, smoke and
241 features categories. These tests are run systematically in the CI daily loops.
243 Success criterias have been defined for these test cases, they can be
244 PASS/FAIL or a success rate may be declared (100%, > 90%).
245 All the criteria, as well as the test dependencies are declared in the
246 testcases.yaml file located in the ci directory of the repository.
248 The scoring for the Danube release per installer can be described as
251 The scoring is an indicator showing how many feature project test suites
252 have been integrated in the scenario.
254 The scoring equals the number of tests * successful iteration of each
255 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
258 apex/odl_l2-nofeature-ha
259 tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
260 Scoring = 21/21 = 7 * 3
262 By default, if not specified, the scenarios are HA.
263 HA means OpenStack High Availability (main services). Note that not
264 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
265 depends upon support of the specific service for HA deployment.
266 For example, in the Danube release, the Congress service
267 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
268 support for HA operation has not yet been verified.
269 See the release notes of the installers for more details.
275 +------------------+---------+---------+-----------------+
276 | Scenario | Scoring | Success | Results |
278 +==================+=========+=========+=================+
279 | nosdn | 20/30 | 67% | `apex-res-1`_ |
280 +------------------+---------+---------+-----------------+
281 | odl_l3 | 16/30 | 53% | `apex-res-2`_ |
282 +------------------+---------+---------+-----------------+
283 | odl-bgpvpn | 18/33 | 56% | `apex-res-3`_ |
284 +------------------+---------+---------+-----------------+
285 | odl-gluon | 20/33 | 61% | `apex-res-4`_ |
286 +------------------+---------+---------+-----------------+
287 | kvm | 20/30 | 67% | `apex-res-5`_ |
288 +------------------+---------+---------+-----------------+
289 | odl_l2-fdio | 25/33 | 76% | `apex-res-6`_ |
290 +------------------+---------+---------+-----------------+
291 | odl_l2-fdio-noha | 24/33 | 73% | local push |
292 +------------------+---------+---------+-----------------+
293 | odl_l3-fdio-noha | 18/30 | 60% | local push |
294 +------------------+---------+---------+-----------------+
295 | fdio | 25/27 | 93% | local push |
296 +------------------+---------+---------+-----------------+
301 +------------------+---------+---------+------------------+
302 | Scenario | Scoring | Success | Results |
304 +==================+=========+=========+==================+
305 | nosdn | 29/30 | 97% | `compass-res-1`_ |
306 +------------------+---------+---------+------------------+
307 | odl_l2 | 29/33 | 88% | `compass-res-2`_ |
308 +------------------+---------+---------+------------------+
309 | odl_l3 | 23/30 | 77% | `compass-res-3`_ |
310 +------------------+---------+---------+------------------+
311 | onos | 29/30 | 97% | `compass-res-4`_ |
312 +------------------+---------+---------+------------------+
313 | onos-sfc | 20/36 | 56% | `compass-res-5`_ |
314 +------------------+---------+---------+------------------+
316 Note: all the Compass tests for Danube have been executed on virtual
317 environment. Bare metal resources were used for Master branch.
323 +----------------------+---------+---------+----------------+
324 | Scenario | Scoring | Success | Results |
326 +======================+=========+=========+================+
327 | nosdn | 39/39 | 100% | `fuel-res-1`_ |
328 +----------------------+---------+---------+----------------+
329 | nosdn-noha | 36/36 | 100% | `fuel-res-2`_ |
330 +----------------------+---------+---------+----------------+
331 | nosdn-kvm | 39/39 | 100% | `fuel-res-3`_ |
332 +----------------------+---------+---------+----------------+
333 | nosdn-kvm-noha | 36/36 | 100% | `fuel-res-4`_ |
334 +----------------------+---------+---------+----------------+
335 | nosdn-ovs | 39/39 | 100% | `fuel-res-5`_ |
336 +----------------------+---------+---------+----------------+
337 | nosdn-ovs-noha | 36/36 | 100% | `fuel-res-6`_ |
338 +----------------------+---------+---------+----------------+
339 | odl_l2 | 42/42 | 100% | `fuel-res-7`_ |
340 +----------------------+---------+---------+----------------+
341 | odl_l2-noha | 39/39 | 100% | `fuel-res-8`_ |
342 +----------------------+---------+---------+----------------+
343 | odl_l2-sfc | 45/45 | 100% | `fuel-res-9`_ |
344 +----------------------+---------+---------+----------------+
345 | odl_l2-sfc-noha | 35/42 | 83% | `fuel-res-10`_ |
346 +----------------------+---------+---------+----------------+
347 | odl_l3 | 36/39 | 92% | `fuel-res-11`_ |
348 +----------------------+---------+---------+----------------+
349 | odl_l3-noha | 36/36 | 100% | `fuel-res-12`_ |
350 +----------------------+---------+---------+----------------+
351 | kvm_ovs_dpdk | 39/39 | 100% | `fuel-res-13`_ |
352 +----------------------+---------+---------+----------------+
353 | kvm_ovs_dpdk_noha | 35/36 | 97% | `fuel-res-14`_ |
354 +----------------------+---------+---------+----------------+
355 | kvm_ovs_dpdk_bar | 42/42 | 100% | `fuel-res-15`_ |
356 +----------------------+---------+---------+----------------+
357 | kvm_ovs_dpdk_bar_noha| 36/39 | 92% | `fuel-res-16`_ |
358 +----------------------+---------+---------+----------------+
366 +---------------------+---------+---------+---------------+
367 | Scenario | Scoring | Success | Results |
369 +=====================+=========+=========+===============+
370 | nosdn | 30/30 | 100% | `joid-res-1`_ |
371 +---------------------+---------+---------+---------------+
372 | nosdn-noha | 10/30 | 33% | `joid-res-2`_ |
373 +---------------------+---------+---------+---------------+
374 | nosdn-lxd | 19/21 | 90% | `joid-res-3`_ |
375 +---------------------+---------+---------+---------------+
376 | nosdn-lxd-noha | 15/21 | 71% | `joid-res-4`_ |
377 +---------------------+---------+---------+---------------+
378 | odl_l2 | 6/33 | 18% | `joid-res-5`_ |
379 +---------------------+---------+---------+---------------+
381 It is highly recommended to install a json viewer in your browser
382 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
384 You can get additional details through test logs on http://artifacts.opnfv.org/.
385 As no search engine is available on the OPNFV artifact web site you must
386 retrieve the pod identifier on which the tests have been executed (see
387 field pod in any of the results) then click on the selected POD and look
388 for the date of the test you are interested in.
390 The reporting pages can be found at:
392 * apex: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-apex.html
393 * compass: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-compass.html
394 * fuel: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html
395 * joid: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-joid.html
397 Danube known restrictions/issues
398 ==================================
400 ************* TODO *****************************************
402 +-----------+-----------+----------------------------------------------+
403 | Installer | Scenario | Issue |
404 +===========+===========+==============================================+
405 | fuel | odl_-* | Tempest test case "TestServerBasicOps" |
406 | | | disabled due to bug `tempest-bug`_ |
407 +-----------+-----------+----------------------------------------------+
408 | apex/fuel | *-bgpvpn | Due to some instabilities in the bgpvpn |
409 | | | test case, the scenario has been postponed |
410 | | | to Danube 2.0 |
411 +-----------+-----------+----------------------------------------------+
412 | apex | *-gluon | vPing_ssh disabled due to floating ips |
413 | | | not working 100% of the times. |
414 | | | Tempest test "test_reboot_server_hard" |
415 | | | disabled due to bug `gluon-bug`_ |
416 +-----------+-----------+----------------------------------------------+
417 | joid | any | Tempest cases related to object storage |
419 +-----------+-----------+----------------------------------------------+
420 | any | any | The VNF tier has not been fully tested |
421 | | | since it has not been run in daily loops |
422 | | | in CI. Weekly jobs have been activated |
423 | | | a bit late in the process and have not been |
424 | | | used to validate the scenarios. |
425 +-----------+-----------+----------------------------------------------+
427 Test and installer/scenario dependencies
428 ========================================
430 It is not always possible to run all the test cases on all the scenarios.
431 The following table details the dependencies of the test cases per
432 scenario. The scenario dependencies (installer or scenario) are detailed
433 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
438 Test results are available in:
440 - test results document: http://artifacts.opnfv.org/functest
442 - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
444 - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
451 +------------------+-----------------------------------------------+
452 | JIRA | Description |
453 +==================+===============================================+
456 +------------------+-----------------------------------------------+
458 All the tickets that are not blocking have been fixed or postponed
461 Functest Danube 2.0 is released without known bugs.
468 - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
470 - wiki Functest Danube page: https://wiki.opnfv.org/display/functest/Functest+Danube
472 - Functest repo: https://git.opnfv.org/cgit/functest
474 - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
476 - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
478 - Functest IRC chan: #opnfv-functest
480 - Reporting page: http://testresults.opnfv.org/reporting/danube.html
482 - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/functest/ci/testcases.yaml
484 .. _`tempest-bug`: https://bugs.launchpad.net/tempest/+bug/1577632
486 .. _`gluon-bug`: https://bugs.opendaylight.org/show_bug.cgi?id=5586
489 .. _`apex-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-455
491 .. _`apex-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-466
493 .. _`apex-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-449
495 .. _`apex-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-450
497 .. _`apex-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-461
499 .. _`apex-res-6`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-485
502 .. _`compass-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-484
504 .. _`compass-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-454
506 .. _`compass-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-baremetal-daily-danube-482
508 .. _`compass-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-481
510 .. _`compass-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-baremetal-daily-danube-453
513 .. _`fuel-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-54
515 .. _`fuel-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-937
517 .. _`fuel-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-873
519 .. _`fuel-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-935
521 .. _`fuel-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-875
523 .. _`fuel-res-6`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-936
525 .. _`fuel-res-7`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-867
527 .. _`fuel-res-8`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-939
529 .. _`fuel-res-9`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-870
531 .. _`fuel-res-10`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-933
533 .. _`fuel-res-11`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-868
535 .. _`fuel-res-12`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-940
537 .. _`fuel-res-13`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-871
539 .. _`fuel-res-14`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-938
541 .. _`fuel-res-15`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-869
543 .. _`fuel-res-16`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-941
546 .. _`joid-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-298
548 .. _`joid-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-260
550 .. _`joid-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-281
552 .. _`joid-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-301
554 .. _`joid-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-291