From: Cedric Ollivier Date: Fri, 20 Apr 2018 07:55:40 +0000 (+0000) Subject: Merge "Update test results in userguide" X-Git-Tag: opnfv-7.0.0~328 X-Git-Url: https://gerrit.opnfv.org/gerrit/gitweb?a=commitdiff_plain;h=62a8c30ec6ddeff54bb91d54428caa0831bfd38e;p=functest.git Merge "Update test results in userguide" --- 62a8c30ec6ddeff54bb91d54428caa0831bfd38e diff --cc docs/testing/user/userguide/test_results.rst index c215e05ec,40a7af33c..bb28989ae --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@@ -15,74 -15,75 +15,111 @@@ If you want additional logs, you may co Automated testing ----------------- - In automated mode, test results are displayed in jenkins logs, a summary is - provided at the end of the job and can be described as follow:: - - +-------------------------+----------------------------------------------------------+ - | ENV VAR | VALUE | - +-------------------------+----------------------------------------------------------+ - | INSTALLER_TYPE | daisy | - | DEPLOY_SCENARIO | os-nosdn-nofeature-ha | - | BUILD_TAG | jenkins-functest-daisy-baremetal-daily-master-67 | - | CI_LOOP | daily | - +-------------------------+----------------------------------------------------------+ - - :: - - +------------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:08 | PASS | - | api_check | functest | healthcheck | 04:22 | PASS | - | snaps_health_check | functest | healthcheck | 00:35 | PASS | - | vping_ssh | functest | smoke | 00:54 | PASS | - | vping_userdata | functest | smoke | 00:27 | PASS | - | tempest_smoke_serial | functest | smoke | 19:39 | FAIL | - | rally_sanity | functest | smoke | 15:16 | PASS | - | refstack_defcore | functest | smoke | 15:55 | PASS | - | snaps_smoke | functest | smoke | 26:45 | FAIL | - | cloudify_ims | functest | vnf | 23:56 | PASS | - | orchestra_openims | orchestra | vnf | 15:07 | PASS | - | orchestra_clearwaterims | orchestra | vnf | 19:10 | PASS | - | vyos_vrouter | functest | vnf | 00:00 | SKIP | - +------------------------------+------------------+---------------------+------------------+----------------+ + In automated mode, tests are run within split Alpine containers, and test + results are displayed in jenkins logs. The result summary is provided at the + end of each suite and can be described as follow. + + Healthcheck suite:: + + +----------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:07 | PASS | + | api_check | functest | healthcheck | 07:46 | PASS | + | snaps_health_check | functest | healthcheck | 00:36 | PASS | + +----------------------------+------------------+---------------------+------------------+----------------+ + + Smoke suite:: + + +------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +------------------------------+------------------+---------------+------------------+----------------+ + | vping_ssh | functest | smoke | 00:57 | PASS | + | vping_userdata | functest | smoke | 00:33 | PASS | + | tempest_smoke_serial | functest | smoke | 13:22 | PASS | + | rally_sanity | functest | smoke | 24:07 | PASS | + | refstack_defcore | functest | smoke | 05:21 | PASS | + | patrole | functest | smoke | 04:29 | PASS | + | snaps_smoke | functest | smoke | 46:54 | PASS | + | odl | functest | smoke | 00:00 | SKIP | + | neutron_trunk | functest | smoke | 00:00 | SKIP | + +------------------------------+------------------+---------------+------------------+----------------+ + + Features suite:: + + +-----------------------------+------------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-----------------------------+------------------------+------------------+------------------+----------------+ + | doctor-notification | doctor | features | 00:00 | SKIP | + | bgpvpn | sdnvpn | features | 00:00 | SKIP | + | functest-odl-sfc | sfc | features | 00:00 | SKIP | + | barometercollectd | barometer | features | 00:00 | SKIP | + | fds | fastdatastacks | features | 00:00 | SKIP | + +-----------------------------+------------------------+------------------+------------------+----------------+ + + Components suite:: + + +-------------------------------+------------------+--------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------------+------------------+--------------------+------------------+----------------+ + | tempest_full_parallel | functest | components | 48:28 | PASS | + | rally_full | functest | components | 126:02 | PASS | + +-------------------------------+------------------+--------------------+------------------+----------------+ + + Vnf suite:: + + +----------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+--------------+------------------+----------------+ + | cloudify_ims | functest | vnf | 28:15 | PASS | + | vyos_vrouter | functest | vnf | 17:59 | PASS | + | juju_epc | functest | vnf | 46:44 | PASS | + +----------------------+------------------+--------------+------------------+----------------+ + + Parser testcase:: + + +-----------------------+-----------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-----------------------+-----------------+------------------+------------------+----------------+ + | parser-basics | parser | features | 00:00 | SKIP | + +-----------------------+-----------------+------------------+------------------+----------------+ +Functest Kubernetes test result:: + + +--------------------------------------+------------------------------------------------------------+ + | ENV VAR | VALUE | + +--------------------------------------+------------------------------------------------------------+ + | INSTALLER_TYPE | compass | + | DEPLOY_SCENARIO | k8-nosdn-nofeature-ha | + | BUILD_TAG | jenkins-functest-compass-baremetal-daily-master-75 | + | CI_LOOP | daily | + +--------------------------------------+------------------------------------------------------------+ + +Kubernetes healthcheck suite:: + + +-------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------+------------------+---------------------+------------------+----------------+ + | k8s_smoke | functest | healthcheck | 01:54 | PASS | + +-------------------+------------------+---------------------+------------------+----------------+ + +Kubernetes smoke suite:: + + +-------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+---------------+------------------+----------------+ + | k8s_conformance | functest | smoke | 57:47 | PASS | + +-------------------------+------------------+---------------+------------------+----------------+ + +Kubernetes features suite:: + + +----------------------+------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+------------------+------------------+----------------+ + | stor4nfv_k8s | stor4nfv | stor4nfv | 00:00 | SKIP | + | clover_k8s | clover | clover | 00:00 | SKIP | + +----------------------+------------------+------------------+------------------+----------------+ + Results are automatically pushed to the test results database, some additional result files are pushed to OPNFV artifact web sites.