.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-Executing the functest suites
-=============================
+Executing the functest suites (Ubuntu)
+======================================
Manual testing
--------------
root@22e436918db0:~/repos/functest/ci# functest tier list
- 0. healthcheck:
- ['healthcheck', 'connection_check', 'api_check',]
+ ['connection_check', 'api_check', 'snaps_health_check',]
- 1. smoke:
- ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity', 'snaps_smoke', 'odl']
+ ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'odl', 'rally_sanity', 'refstack_defcore', 'snaps_smoke']
- 2. features:
- ['doctor', 'security_scan']
+ ['doctor', 'domino', 'promise', security_scan']
- 3. components:
['tempest_full_parallel', 'rally_full']
- 4. vnf:
- ['cloudify_ims']
+ ['cloudify_ims', 'orchestra_ims', 'vyos_vrouter']
and
root@22e436918db0:~/repos/functest/ci# functest testcase list
- healthcheck
api_check
connection_check
+ snaps_health_check
vping_ssh
vping_userdata
snaps_smoke
+ refstack_defcore
tempest_smoke_serial
rally_sanity
odl
- doctor
- security_scan
tempest_full_parallel
rally_full
- cloudify_ims
+ vyos_vrouter
+
+Note the list of test cases depend on the installer and the scenario.
More specific details on specific Tiers or Test Cases can be seen wih the
'show' command::
To execute a Test Tier or Test Case, the 'run' command is used::
root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck
- Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck'
- 2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file...
- 2016-06-30 11:44:56,937 - run_tests - INFO - ############################################
- 2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck'
- 2016-06-30 11:44:56,938 - run_tests - INFO - ############################################
- 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
- 2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'...
- 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
- 2016-06-30 11:44:56,953 - healtcheck - INFO - Testing Keystone API...
- 2016-06-30 11:45:05,351 - healtcheck - INFO - ...Keystone OK!
- 2016-06-30 11:45:05,354 - healtcheck - INFO - Testing Glance API...
- 2016-06-30 11:45:29,746 - healtcheck - INFO - ... Glance OK!
- 2016-06-30 11:45:29,749 - healtcheck - INFO - Testing Cinder API...
- 2016-06-30 11:45:37,502 - healtcheck - INFO - ...Cinder OK!
- 2016-06-30 11:45:37,505 - healtcheck - INFO - Testing Neutron API...
- 2016-06-30 11:45:39,664 - healtcheck - INFO - External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca
- 2016-06-30 11:45:39,667 - healtcheck - INFO - 1. Create Networks...
- 2016-06-30 11:45:44,227 - healtcheck - INFO - 2. Create subnets...
- 2016-06-30 11:45:46,805 - healtcheck - INFO - 4. Create Routers...
- 2016-06-30 11:45:54,261 - healtcheck - INFO - ...Neutron OK!
- 2016-06-30 11:45:54,264 - healtcheck - INFO - Testing Nova API...
- 2016-06-30 11:47:12,272 - healtcheck - INFO - ...Nova OK!
- 2016-06-30 11:47:12,274 - healtcheck - INFO - Checking if instances get an IP from DHCP...
- :
- :
- 2016-06-30 11:48:17,832 - healtcheck - INFO - ...DHCP OK!
- 2016-06-30 11:48:17,835 - healtcheck - INFO - Health check passed!
- 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
- 2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources...
- 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
- Version 1 is deprecated, use alternative version 2 instead.
- WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead.
- 2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances...
- 2016-06-30 11:48:24,439 - clean_openstack - INFO - -------------------------------------------
- 2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images...
- 2016-06-30 11:48:35,853 - clean_openstack - INFO - -------------------------------------------
- 2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes...
- 2016-06-30 11:48:37,344 - clean_openstack - INFO - -------------------------------------------
- 2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs...
- 2016-06-30 11:48:37,467 - clean_openstack - INFO - -------------------------------------------
- 2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects
- 2016-06-30 11:48:53,633 - clean_openstack - INFO - -------------------------------------------
- 2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups...
- 2016-06-30 11:48:53,689 - clean_openstack - INFO - -------------------------------------------
- 2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users...
- 2016-06-30 11:48:54,444 - clean_openstack - INFO - -------------------------------------------
- 2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants...
- 2016-06-30 11:48:54,711 - clean_openstack - INFO - -------------------------------------------
+
+ 2017-08-16 12:35:51,799 - functest.ci.run_tests - INFO - ############################################
+ 2017-08-16 12:35:51,799 - functest.ci.run_tests - INFO - Running tier 'healthcheck'
+ 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - ############################################
+ 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO -
+ 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - ============================================
+ 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - Running test case 'connection_check'...
+ 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - ============================================
+ 2017-08-16 12:36:00,278 - functest.core.testcase - INFO - The results were successfully pushed to DB
+ 2017-08-16 12:36:00,279 - functest.ci.run_tests - INFO - Test result:
+ +--------------------------+------------------+------------------+----------------+
+ | TEST CASE | PROJECT | DURATION | RESULT |
+ +--------------------------+------------------+------------------+----------------+
+ | connection_check | functest | 00:06 | PASS |
+ +--------------------------+------------------+------------------+----------------+
+ 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO -
+ 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO - ============================================
+ 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO - Running test case 'api_check'...
+ 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO - ============================================
+ 2017-08-16 12:41:04,088 - functest.core.testcase - INFO - The results were successfully pushed to DB
+ 2017-08-16 12:41:04,088 - functest.ci.run_tests - INFO - Test result:
+ +-------------------+------------------+------------------+----------------+
+ | TEST CASE | PROJECT | DURATION | RESULT |
+ +-------------------+------------------+------------------+----------------+
+ | api_check | functest | 05:03 | PASS |
+ +-------------------+------------------+------------------+----------------+
+ 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO -
+ 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO - ============================================
+ 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO - Running test case 'snaps_health_check'...
+ 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO - ============================================
+ 2017-08-16 12:41:39,817 - functest.core.testcase - INFO - The results were successfully pushed to DB
+ 2017-08-16 12:41:39,818 - functest.ci.run_tests - INFO - Test result:
+ +----------------------------+------------------+------------------+----------------+
+ | TEST CASE | PROJECT | DURATION | RESULT |
+ +----------------------------+------------------+------------------+----------------+
+ | snaps_health_check | functest | 00:35 | PASS |
+ +----------------------------+------------------+------------------+----------------+
and
root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh
- Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh'
- 2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file...
- 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
- 2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'...
- 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
- 2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img'...
- 2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net...
- 2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'...
- 2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'...
- 2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48'
- 2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'...
- 2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE.
- 2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'...
- 2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'...
- 2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE.
- 2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'...
- 2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'...
- 2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140'
- 2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2'
- 2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140...
- 2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping...
- 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected!
- 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s.
- 2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK
- 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
- 2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources...
- 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
- Version 1 is deprecated, use alternative version 2 instead.
+ 2017-08-16 12:41:39,821 - functest.ci.run_tests - INFO - ============================================
+ 2017-08-16 12:41:39,821 - functest.ci.run_tests - INFO - Running test case 'vping_ssh'...
+ 2017-08-16 12:41:39,821 - functest.ci.run_tests - INFO - ============================================
+ 2017-08-16 12:42:49,861 - functest.core.testcase - INFO - The results were successfully pushed to DB
+ 2017-08-16 12:42:49,861 - functest.ci.run_tests - INFO - Test result:
+ +-------------------+------------------+------------------+----------------+
+ | TEST CASE | PROJECT | DURATION | RESULT |
+ +-------------------+------------------+------------------+----------------+
+ | vping_ssh | functest | 00:47 | PASS |
+ +-------------------+------------------+------------------+----------------+
:
:
etc.
root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck
Test cases in tier 'healthcheck':
- ['healthcheck']
+ ['connection_check', 'api_check', 'snaps_health_check']
Please note that for some scenarios some test cases might not be launched.
For example, the last example displayed only the 'odl' testcase for the given
-environment. In this particular system the deployment does not support the 'onos' SDN
+environment. In this particular system the deployment does not support the 'ocl' SDN
Controller Test Case; for example.
**Important** If you use the command 'functest tier run <tier_name>', then the
Functest includes a cleaning mechanism in order to remove all the OpenStack
resources except those present before running any test. The script
-*$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up
+*$REPOS_DIR/functest/functest/utils/openstack_snapshot.py* is called once when setting up
the Functest environment (i.e. CLI command 'functest env prepare') to snapshot
all the OpenStack resources (images, networks, volumes, security groups, tenants,
users) so that an eventual cleanup does not remove any of these defaults.
-The script **clean_openstack.py** which is located in
-*$REPOS_DIR/functest/functest/utils/* is normally called after a test execution. It is
-in charge of cleaning the OpenStack resources that are not specified in the
-defaults file generated previously which is stored in
-*/home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container.
+The script **openstack_clean.py** which is located in
+*$REPOS_DIR/functest/functest/utils/* is in charge of cleaning the OpenStack resources
+that are not specified in the defaults file generated previously which is stored in
+*/home/opnfv/functest/conf/openstack_snapshot.yaml* in the Functest docker container.
It is important to mention that if there are new OpenStack resources created
-manually after preparing the Functest environment, they will be removed, unless
-you use the special method of invoking the test case with specific suppression
-of clean up. (See the `Troubleshooting`_ section).
+manually after the snapshot done before running the tests, they will be removed,
+unless you use the special method of invoking the test case with specific
+suppression of clean up. (See the `Troubleshooting`_ section).
The reason to include this cleanup meachanism in Functest is because some
-test suites such as Tempest or Rally create a lot of resources (users,
-tenants, networks, volumes etc.) that are not always properly cleaned, so this
-function has been set to keep the system as clean as it was before a
-full Functest execution.
+test suites create a lot of resources (users, tenants, networks, volumes etc.)
+that are not always properly cleaned, so this function has been set to keep the
+system as clean as it was before a full Functest execution.
Although the Functest CLI provides an easy way to run any test, it is possible to
do a direct call to the desired test script. For example:
- python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/vPing_ssh.py -d
+ python $REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_ssh.py
Automated testing
CI provides some useful information passed to the container as environment
variables:
- * Installer (apex|compass|daisy|fuel|joid), stored in INSTALLER_TYPE
+ * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE
* Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
* The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
- * controller = (odl|onos|ocl|nosdn)
- * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|multisites)
+ * controller = (odl|ocl|nosdn)
+ * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|ovs_dpdk_bar)
* mode = (ha|noha)
The constraints per test case are defined in the Functest configuration file
-*/home/opnfv/repos/functest/functest/ci/testcases.yaml*::
+*/usr/local/lib/python2.7/dist-packages/functest/ci/testcases.yaml*::
tiers:
- -
- name: healthcheck
- order: 0
- ci_loop: '(daily)|(weekly)'
- description : >-
- First tier to be executed to verify the basic
- operations in the VIM.
- testcases:
- -
- name: healthcheck
- criteria: 'status == "PASS"'
- blocking: true
- description: >-
- This test case verifies the basic OpenStack services like
- Keystone, Glance, Cinder, Neutron and Nova.
-
- dependencies:
- installer: ''
- scenario: ''
-
-
name: smoke
order: 1
At the tier level, we define the following parameters:
- * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs
+ * ci_loop: indicate if in automated mode, the test case must be run in dail and/or weekly jobs
* description: a high level view of the test case
For a given test case we defined:
Please note that a system snapshot is taken before any test case execution.
This testcase.yaml file is used for CI, for the CLI and for the automatic reporting.
+
+
+Executing Functest suites (Alpine)
+==================================
+
+As mentioned in the configuration guide `[1]`_, Alpine docker containers have
+been introduced in Euphrates.
+Tier containers have been created.
+Assuming that you pulled the container and your environement is ready, you can
+simply run the tiers by typing (e.g. with functest-healthcheck)::
+
+ sudo docker run --env-file env \
+ -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \
+ -v $(pwd)/images:/home/opnfv/functest/images \
+ opnfv/functest-healthcheck
+
+You should get::
+
+ +----------------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +----------------------------+------------------+---------------------+------------------+----------------+
+ | connection_check | functest | healthcheck | 00:02 | PASS |
+ | api_check | functest | healthcheck | 03:19 | PASS |
+ | snaps_health_check | functest | healthcheck | 00:46 | PASS |
+ +----------------------------+------------------+---------------------+------------------+----------------+
+
+You can run functest-healcheck, functest-smoke, functest-features,
+functest-components and functest-vnf.
+
+Please note that you may also use the CLI for manual tests using Alpine
+containers.
+
+
+Functest internal API
+=====================
+
+An internal API has been introduced in Euphrates. The goal is to trigger
+Functest operations through an API in addition of the CLI.
+This could be considered as a first step towards a pseudo micro services
+approach where the different test projects could expose and consume APIs to the
+other test projects.
+
+In Euphrates the main method of the APIs are:
+
+ * Show environment
+ * Prepare Environment
+ * Show credentials
+ * List all testcases
+ * Show a testcase
+ * List all tiers
+ * Show a tier
+ * List all testcases within given tier
+
+The API can be invoked as follows:
+ http://<functest_url>:5000/api/v1/functest/envs
+
+TODO
+
+.. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/#
Code Review / functest.git / blobdiff