1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
8 OPNFV Danube release note for Functest Docs
9 (c) by Jose Lausuch (Ericsson)
11 OPNFV Danube release note for Functest Docs
12 are licensed under a Creative Commons Attribution 4.0 International License.
13 You should have received a copy of the license along with this.
14 If not, see <http://creativecommons.org/licenses/by/4.0/>.
16 ===========================================
17 OPNFV Danube2.0 release note for Functest
18 ===========================================
23 This document describes the release note of Functest project.
29 +------------+----------+------------------+------------------------+
30 | **Date** | **Ver.** | **Author** | **Comment** |
32 +------------+----------+------------------+------------------------+
33 | 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for |
34 | | | (Orange) | Colorado.1.0 release |
35 +------------+----------+------------------+------------------------+
36 | 2016-10-24 | 2.0.0 | Morgan Richomme | Functest for |
37 | | | (Orange) | Colorado.2.0 release |
38 +------------+----------+------------------+------------------------+
39 | 2016-08-17 | 3.0.0 | Morgan Richomme | Functest for |
40 | | | (Orange) | Colorado.3.0 release |
41 +------------+----------+------------------+------------------------+
42 | 2017-03-29 | 4.0.0 | Jose Lausuch | Functest for |
43 | | | (Ericsson) | Danube.1.0 release |
44 +------------+----------+------------------+------------------------+
45 | 2017-05-04 | 5.0.0 | Jose Lausuch | Functest for |
46 | | | (Ericsson) | Danube.2.0 release |
47 +------------+----------+------------------+------------------------+
48 | 2017-07-12 | 6.0.0 | Jose Lausuch | Functest for |
49 | | | (Ericsson) | Danube.3.0 release |
50 +------------+----------+------------------+------------------------+
53 ======================
55 Functest deals with functional testing of the OPNFV solution.
56 It includes test cases developed within the project and test cases developed in
57 other OPNFV projects and other upstream communities.
59 The internal test cases are:
66 * tempest_smoke_serial
71 * tempest_full_parallel
75 The OPNFV projects integrated into Functest framework for automation are:
93 The validation of a scenario requires a subset of these tests depending
94 on the installer and the scenario.
96 The test cases from vnf (cloudify_ims, orchestra_ims, vyos_vrouter) and
97 component categories (tempest full parallel, Rally full) are not considered for
103 +--------------------------------------+--------------------------------------+
104 | **Project** | functest |
106 +--------------------------------------+--------------------------------------+
107 | **Repo/tag** | danube.3.0 |
109 +--------------------------------------+--------------------------------------+
110 | **Release designation** | Danube service release |
112 +--------------------------------------+--------------------------------------+
113 | **Release date** | July 14th 2017 |
115 +--------------------------------------+--------------------------------------+
116 | **Purpose of the delivery** | Danube third release |
118 +--------------------------------------+--------------------------------------+
126 - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: danube.3.0)
128 - The TestAPI Docker image: https://hub.docker.com/r/opnfv/testapi (tag:danube.3.0)
134 - Installation/configuration guide: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/configguide/index.html
136 - User Guide: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/userguide/index.html
138 - Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/developer/devguide/index.html
145 - Bugfixes in refstack client
147 - Fixed vIMS onboarding and ims deployment
149 - Increased timeout of OpenBaton installation
151 - Switched to mitaka-eol in openstack repositories, as stable/mitake is not longer existing
153 - Minor fixes in logging and test cases
155 - 3 scenarios have been successfully executed on aarch64 architectures on Fuel deployments (nosdn, odl_l2 and odl_l3)
161 For Danube 3.0, Functest was tested on the following HA scenarios (new
162 dabube scenarios in bold):
164 +---------------------+---------+---------+---------+---------+
165 | Scenario | Apex | Compass | Fuel | Joid |
166 +=====================+=========+=========+=========+=========+
167 | nosdn | X | X | X | X |
168 +---------------------+---------+---------+---------+---------+
169 | **fdio** | X | | | |
170 +---------------------+---------+---------+---------+---------+
172 +---------------------+---------+---------+---------+---------+
173 | **kvm_ovs_dpdk** | | | X | |
174 +---------------------+---------+---------+---------+---------+
175 | **kvm_ovs_dpdk-bar**| | | X | |
176 +---------------------+---------+---------+---------+---------+
178 +---------------------+---------+---------+---------+---------+
179 | **ovs** | | | X | |
180 +---------------------+---------+---------+---------+---------+
181 | **openo** | | X | | |
182 +---------------------+---------+---------+---------+---------+
183 | odl_l2 | | X | X | X |
184 +---------------------+---------+---------+---------+---------+
185 | odl-bgpvpn | X | | | |
186 +---------------------+---------+---------+---------+---------+
187 | odl_l2-bgpvpn | | | X | |
188 +---------------------+---------+---------+---------+---------+
189 | **odl_l2-fdio** | X | | | |
190 +---------------------+---------+---------+---------+---------+
191 | odl_l2-sfc | | | X | |
192 +---------------------+---------+---------+---------+---------+
193 | odl_l3 | X | X | X | |
194 +---------------------+---------+---------+---------+---------+
195 | **ocl** | | X | | |
196 +---------------------+---------+---------+---------+---------+
198 +---------------------+---------+---------+---------+---------+
199 | multisite | | | X | |
200 +---------------------+---------+---------+---------+---------+
204 +---------------------+---------+---------+---------+---------+
205 | Scenario | Apex | Compass | Fuel | Joid |
206 +=====================+=========+=========+=========+=========+
207 | nosdn | | | X | X |
208 +---------------------+---------+---------+---------+---------+
210 +---------------------+---------+---------+---------+---------+
211 | **kvm_ovs_dpdk** | | | X | |
212 +---------------------+---------+---------+---------+---------+
213 | **kvm_ovs_dpdk-bar**| | | X | |
214 +---------------------+---------+---------+---------+---------+
216 +---------------------+---------+---------+---------+---------+
218 +---------------------+---------+---------+---------+---------+
220 +---------------------+---------+---------+---------+---------+
221 | odl_l2-bgpvpn | | | X | |
222 +---------------------+---------+---------+---------+---------+
223 | odl_l2-fdio | X | | | |
224 +---------------------+---------+---------+---------+---------+
225 | **odl_l3-fdio** | X | | | |
226 +---------------------+---------+---------+---------+---------+
227 | odl_l2-sfc | | | X | |
228 +---------------------+---------+---------+---------+---------+
230 +---------------------+---------+---------+---------+---------+
231 | **odl_gluon** | X | | | |
232 +---------------------+---------+---------+---------+---------+
234 Colorado deprecated scenarios:
241 For the scenario validation, we consider only the healthcheck, smoke and
242 features categories. These tests are run systematically in the CI daily loops.
244 Success criterias have been defined for these test cases, they can be
245 PASS/FAIL or a success rate may be declared (100%, > 90%).
246 All the criteria, as well as the test dependencies are declared in the
247 testcases.yaml file located in the ci directory of the repository.
249 The scoring for the Danube release per installer can be described as
252 The scoring is an indicator showing how many feature project test suites
253 have been integrated in the scenario.
255 The scoring equals the number of tests * successful iteration of each
256 test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
259 apex/odl_l2-nofeature-ha
260 tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
261 Scoring = 21/21 = 7 * 3
263 By default, if not specified, the scenarios are HA.
264 HA means OpenStack High Availability (main services). Note that not
265 all VIM (e.g. OpenStack) services are deployed in HA mode, as that
266 depends upon support of the specific service for HA deployment.
267 For example, in the Danube release, the Congress service
268 is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
269 support for HA operation has not yet been verified.
270 See the release notes of the installers for more details.
276 +------------------+---------+---------+-----------------+
277 | Scenario | Scoring | Success | Results |
279 +==================+=========+=========+=================+
280 | nosdn | 20/30 | 67% | `apex-res-1`_ |
281 +------------------+---------+---------+-----------------+
282 | odl_l3 | 16/30 | 53% | `apex-res-2`_ |
283 +------------------+---------+---------+-----------------+
284 | odl-bgpvpn | 18/33 | 56% | `apex-res-3`_ |
285 +------------------+---------+---------+-----------------+
286 | odl-gluon | 20/33 | 61% | `apex-res-4`_ |
287 +------------------+---------+---------+-----------------+
288 | kvm | 20/30 | 67% | `apex-res-5`_ |
289 +------------------+---------+---------+-----------------+
290 | odl_l2-fdio | 25/33 | 76% | `apex-res-6`_ |
291 +------------------+---------+---------+-----------------+
292 | odl_l2-fdio-noha | 24/33 | 73% | local push |
293 +------------------+---------+---------+-----------------+
294 | odl_l3-fdio-noha | 18/30 | 60% | local push |
295 +------------------+---------+---------+-----------------+
296 | fdio | 25/27 | 93% | local push |
297 +------------------+---------+---------+-----------------+
302 +------------------+---------+---------+------------------+
303 | Scenario | Scoring | Success | Results |
305 +==================+=========+=========+==================+
306 | nosdn | 29/30 | 97% | `compass-res-1`_ |
307 +------------------+---------+---------+------------------+
308 | odl_l2 | 29/33 | 88% | `compass-res-2`_ |
309 +------------------+---------+---------+------------------+
310 | odl_l3 | 23/30 | 77% | `compass-res-3`_ |
311 +------------------+---------+---------+------------------+
312 | onos | 29/30 | 97% | `compass-res-4`_ |
313 +------------------+---------+---------+------------------+
314 | onos-sfc | 20/36 | 56% | `compass-res-5`_ |
315 +------------------+---------+---------+------------------+
317 Note: all the Compass tests for Danube have been executed on virtual
318 environment. Bare metal resources were used for Master branch.
324 +----------------------+---------+---------+----------------+
325 | Scenario | Scoring | Success | Results |
327 +======================+=========+=========+================+
328 | nosdn | 39/39 | 100% | `fuel-res-1`_ |
329 +----------------------+---------+---------+----------------+
330 | nosdn-noha | 36/36 | 100% | `fuel-res-2`_ |
331 +----------------------+---------+---------+----------------+
332 | nosdn-kvm | 39/39 | 100% | `fuel-res-3`_ |
333 +----------------------+---------+---------+----------------+
334 | nosdn-kvm-noha | 36/36 | 100% | `fuel-res-4`_ |
335 +----------------------+---------+---------+----------------+
336 | nosdn-ovs | 39/39 | 100% | `fuel-res-5`_ |
337 +----------------------+---------+---------+----------------+
338 | nosdn-ovs-noha | 36/36 | 100% | `fuel-res-6`_ |
339 +----------------------+---------+---------+----------------+
340 | odl_l2 | 42/42 | 100% | `fuel-res-7`_ |
341 +----------------------+---------+---------+----------------+
342 | odl_l2-noha | 39/39 | 100% | `fuel-res-8`_ |
343 +----------------------+---------+---------+----------------+
344 | odl_l2-sfc | 45/45 | 100% | `fuel-res-9`_ |
345 +----------------------+---------+---------+----------------+
346 | odl_l2-sfc-noha | 35/42 | 83% | `fuel-res-10`_ |
347 +----------------------+---------+---------+----------------+
348 | odl_l3 | 36/39 | 92% | `fuel-res-11`_ |
349 +----------------------+---------+---------+----------------+
350 | odl_l3-noha | 36/36 | 100% | `fuel-res-12`_ |
351 +----------------------+---------+---------+----------------+
352 | kvm_ovs_dpdk | 39/39 | 100% | `fuel-res-13`_ |
353 +----------------------+---------+---------+----------------+
354 | kvm_ovs_dpdk_noha | 35/36 | 97% | `fuel-res-14`_ |
355 +----------------------+---------+---------+----------------+
356 | kvm_ovs_dpdk_bar | 42/42 | 100% | `fuel-res-15`_ |
357 +----------------------+---------+---------+----------------+
358 | kvm_ovs_dpdk_bar_noha| 36/39 | 92% | `fuel-res-16`_ |
359 +----------------------+---------+---------+----------------+
367 +---------------------+---------+---------+---------------+
368 | Scenario | Scoring | Success | Results |
370 +=====================+=========+=========+===============+
371 | nosdn | 30/30 | 100% | `joid-res-1`_ |
372 +---------------------+---------+---------+---------------+
373 | nosdn-noha | 10/30 | 33% | `joid-res-2`_ |
374 +---------------------+---------+---------+---------------+
375 | nosdn-lxd | 19/21 | 90% | `joid-res-3`_ |
376 +---------------------+---------+---------+---------------+
377 | nosdn-lxd-noha | 15/21 | 71% | `joid-res-4`_ |
378 +---------------------+---------+---------+---------------+
379 | odl_l2 | 6/33 | 18% | `joid-res-5`_ |
380 +---------------------+---------+---------+---------------+
382 It is highly recommended to install a json viewer in your browser
383 (e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
385 You can get additional details through test logs on http://artifacts.opnfv.org/.
386 As no search engine is available on the OPNFV artifact web site you must
387 retrieve the pod identifier on which the tests have been executed (see
388 field pod in any of the results) then click on the selected POD and look
389 for the date of the test you are interested in.
391 The reporting pages can be found at:
393 * apex: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-apex.html
394 * compass: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-compass.html
395 * fuel: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html
396 * joid: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-joid.html
398 Danube known restrictions/issues
399 ==================================
401 ************* TODO *****************************************
403 +-----------+-----------+----------------------------------------------+
404 | Installer | Scenario | Issue |
405 +===========+===========+==============================================+
406 | fuel | odl_-* | Tempest test case "TestServerBasicOps" |
407 | | | disabled due to bug `tempest-bug`_ |
408 +-----------+-----------+----------------------------------------------+
409 | apex/fuel | *-bgpvpn | Due to some instabilities in the bgpvpn |
410 | | | test case, the scenario has been postponed |
411 | | | to Danube 2.0 |
412 +-----------+-----------+----------------------------------------------+
413 | apex | *-gluon | vPing_ssh disabled due to floating ips |
414 | | | not working 100% of the times. |
415 | | | Tempest test "test_reboot_server_hard" |
416 | | | disabled due to bug `gluon-bug`_ |
417 +-----------+-----------+----------------------------------------------+
418 | joid | any | Tempest cases related to object storage |
420 +-----------+-----------+----------------------------------------------+
421 | any | any | The VNF tier has not been fully tested |
422 | | | since it has not been run in daily loops |
423 | | | in CI. Weekly jobs have been activated |
424 | | | a bit late in the process and have not been |
425 | | | used to validate the scenarios. |
426 +-----------+-----------+----------------------------------------------+
428 Test and installer/scenario dependencies
429 ========================================
431 It is not always possible to run all the test cases on all the scenarios.
432 The following table details the dependencies of the test cases per
433 scenario. The scenario dependencies (installer or scenario) are detailed
434 in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
439 Test results are available in:
441 - test results document: http://artifacts.opnfv.org/functest
443 - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
445 - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
452 +------------------+-----------------------------------------------+
453 | JIRA | Description |
454 +==================+===============================================+
457 +------------------+-----------------------------------------------+
459 All the tickets that are not blocking have been fixed or postponed
462 Functest Danube 2.0 is released without known bugs.
469 - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
471 - wiki Functest Danube page: https://wiki.opnfv.org/display/functest/Functest+Danube
473 - Functest repo: https://git.opnfv.org/cgit/functest
475 - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
477 - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
479 - Functest IRC chan: #opnfv-functest
481 - Reporting page: http://testresults.opnfv.org/reporting/danube.html
483 - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/functest/ci/testcases.yaml
485 .. _`tempest-bug`: https://bugs.launchpad.net/tempest/+bug/1577632
487 .. _`gluon-bug`: https://bugs.opendaylight.org/show_bug.cgi?id=5586
490 .. _`apex-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-455
492 .. _`apex-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-466
494 .. _`apex-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-449
496 .. _`apex-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-450
498 .. _`apex-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-461
500 .. _`apex-res-6`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-baremetal-daily-danube-485
503 .. _`compass-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-484
505 .. _`compass-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-454
507 .. _`compass-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-baremetal-daily-danube-482
509 .. _`compass-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-danube-481
511 .. _`compass-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-baremetal-daily-danube-453
514 .. _`fuel-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-54
516 .. _`fuel-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-937
518 .. _`fuel-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-873
520 .. _`fuel-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-935
522 .. _`fuel-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-875
524 .. _`fuel-res-6`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-936
526 .. _`fuel-res-7`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-867
528 .. _`fuel-res-8`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-939
530 .. _`fuel-res-9`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-870
532 .. _`fuel-res-10`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-933
534 .. _`fuel-res-11`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-868
536 .. _`fuel-res-12`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-940
538 .. _`fuel-res-13`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-871
540 .. _`fuel-res-14`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-938
542 .. _`fuel-res-15`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-danube-869
544 .. _`fuel-res-16`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-danube-941
547 .. _`joid-res-1`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-298
549 .. _`joid-res-2`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-260
551 .. _`joid-res-3`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-281
553 .. _`joid-res-4`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-301
555 .. _`joid-res-5`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-danube-291