provide a procedure to execute them. In the OPNFV Colorado system release,
a Functest CLI utility is introduced for easier execution of test procedures.
-A overview presentation has been created for the first OPNFV Summit `[4]`_.
+An overview presentation has been created for the first OPNFV Summit `[4]`_.
This document is a continuation of the OPNFV Functest Configuration Guide `[1]`_.
| | to VM2 | |
| +------------------->| |
| | | |
- | | Stablish SSH | |
+ | | Establish SSH | |
| | connection to VM2 | |
| | through floating IP| |
| +------------------->| |
The list of tests can be described as follows:
- * Restconf.basic: Get the controller modules via Restconf
- * Neutron.Networks
+ * Basic Restconf test cases
+ * Connect to Restconf URL
+ * Check the HTTP code status
- * Check OpenStack Networks :: Checking OpenStack Neutron for known networks
- * Check OpenDaylight Networks :: Checking OpenDaylight Neutron API
- * Create Network :: Create new network in OpenStack
- * Check Network :: Check Network created in OpenDaylight
- * Neutron.Networks :: Checking Network created in OpenStack are pushed
+ * Neutron Reachability test cases
+ * Get the complete list of neutron resources (networks, subnets, ports)
- * Neutron.Subnets
+ * Neutron Network test cases
+ * Check OpenStack networks
+ * Check OpenDaylight networks
+ * Create a new network via OpenStack and check the HTTP status code returned by Neutron
+ * Check that the network has also been successfully created in OpenDaylight
- * Check OpenStack Subnets :: Checking OpenStack Neutron for known Subnets
- * Check OpenDaylight subnets :: Checking OpenDaylight Neutron API
- * Create New subnet :: Create new subnet in OpenStack
- * Check New subnet :: Check new subnet created in OpenDaylight
- * Neutron.Subnets :: Checking Subnets created in OpenStack are pushed
+ * Neutron Subnet test cases
+ * Check OpenStack subnets
+ * Check OpenDaylight subnets
+ * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron
+ * Check that the subnet has also been successfully created in OpenDaylight
- * Neutron.Ports
+ * Neutron Port test cases
+ * Check OpenStack Neutron for known ports
+ * Check OpenDaylight ports
+ * Create a new port via OpenStack and check the HTTP status code returned by Neutron
+ * Check that the new port has also been successfully created in OpenDaylight
- * Check OpenStack ports :: Checking OpenStack Neutron for known ports
- * Check OpenDaylight ports :: Checking OpenDaylight Neutron API
- * Create New Port :: Create new port in OpenStack
- * Check New Port :: Check new subnet created in OpenDaylight
- * Neutron.Ports :: Checking Port created in OpenStack are pushed
+ * Delete operations
+ * Delete the port previously created via OpenStack
+ * Check that the port has been also succesfully deleted in OpenDaylight
+ * Delete previously subnet created via OpenStack
+ * Check that the subnet has also been successfully deleted in OpenDaylight
+ * Delete the network created via OpenStack
+ * Check that the network has also been succesfully deleted in OpenDaylight
- * Delete Ports
-
- * Delete previously created subnet in OpenStack
- * Check subnet deleted in OpenDaylight
- * Check subnet deleted in OpenStack
-
- * Delete network
-
- * Delete previously created network in OpenStack
- * Check network deleted in OpenDaylight
- * Check network deleted in OpenStack
+Note: the checks in OpenDaylight are based on the returned HTTP status
+code returned by OpenDaylight.
ONOS
* Delete External Gateway: Delete the External Gateway and check that it is
'NULL' in ONOS
-Open Contrail
-^^^^^^^^^^^^^
-**TODO:**
-
-
Features
--------
-Doctor
-^^^^^^
-
-**TODO:**
-
-
-
-Promise
-^^^^^^^
-
-Promise provides a basic set of test cases as part of the deliverable.
-
-The available 33 test cases can be grouped into 7 test suites:
-
- #. Add a new OpenStack provider into resource pool: Registers
- OpenStack into a new resource pool and adds more capacity associated
- with this pool.
-
- #. Allocation without reservation: Creates a new server in OpenStack
- and adds a new allocation record in Promise shim-layer.
+Most of the features have been developped by feature projects.
+Security_scan has been initiated in Functest repository but should soon
+be declared in its own repository as well.
- #. Allocation using reservation for immediate use: Creates a resource
- reservation record with no start/end time and immediately creates a new
- server in OpenStack and add a new allocation record in Promise
- shim-layer.
+Please refer to the dedicated feature user guides for details:
- #. Reservation for future use: Creates a resource reservation record
- for a future start time, queries, modifies and cancels the newly created
- reservation.
-
- #. Capacity planning: Decreases and increases the available capacity
- from a provider in the future and queries the available collections and
- utilizations.
-
- #. Reservation with conflict: Tries to create reservations for
- immediate and future use with conflict.
-
- #. Cleanup test allocations: Destroys all allocations in OpenStack.
-
-
-bgpvpn
-^^^^^^
-Many telecom network functions are relying on layer-3 infrastructure services,
-within a VNF between components, or towards existing external networks.
-In many cases, these external networks are implemented in MPLS/BGP technology in
-existing service provider wide-area-networks (WAN). This proven technology
-provides a good mechanism for inter-operation of a NFV Infrastructure (NFVI)
-and WAN.
-
-The SDNVPN project defined a 'bgpvpn' test suite.
-This 'bgpvpn' test suite deals with 2 Tempest cases dedicated to the test of
-the OpenStack 'bgpvpn' API:
-
- * test_create_bgpvpn
- * test_create_bgpvpn_as_non_admin_fail
+ * bgpvpn: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html
+ * copper: http://artifacts.opnfv.org/copper/colorado/docs/userguide/index.html
+ * doctor: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html
+ * domino: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html
+ * moon: http://artifacts.opnfv.org/moon/docs/userguide/index.html
+ * multisites: http://artifacts.opnfv.org/multisite/docs/userguide/index.html
+ * onos-sfc: http://artifacts.opnfv.org/onosfw/colorado/userguide/index.html
+ * odl-sfc: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html
+ * promise: http://artifacts.opnfv.org/promise/colorado/docs/userguide/index.html
security_scan
^^^^^^^^^^^^^
* Jenkins Build Initiated
* security_scan.py script is called, and a config file is passed to the script as
an argument.
- * The IP addresses of each NFVi node (compute / control), is gathered.
+ * The IP addresses of each NFVi node (compute / control) are gathered
* A scan profile is matched to the node type.
* The OpenSCAP application is remotely installed onto each target node gathered
on step 3, using upstream packaging (rpm and .deb).
At present, only the Apex installer is supported, with support for other installers due
within D-release.
+
+
VNF
---
:alt: vIMS architecture
+parser
+^^^^^^
+
+See parser user guide for details: `[12]`_
+
+
.. include:: ./runfunctest.rst
Test results
============
-Note that the results are documented per scenario basis. Although most of the test
-cases might show the same output, some of them are not supported by
-certain scenarios. Please select the appropriate scenario and compare the results
-to that referenced in the documentation.
+Manual testing
+--------------
+
+In manual mode test results are displayed in the console and result files
+are put in /home/opnfv/functest/results.
+
+Automated testing
+--------------
+
+In automated mode, test results are displayed in jenkins logs, a summary is provided
+at the end of the job and can be described as follow::
+
+ +==================================================================================================================================================+
+ | FUNCTEST REPORT |
+ +==================================================================================================================================================+
+ | |
+ | Deployment description: |
+ | INSTALLER: fuel |
+ | SCENARIO: os-odl_l2-nofeature-ha |
+ | BUILD TAG: jenkins-functest-fuel-baremetal-daily-master-324 |
+ | CI LOOP: daily |
+ | |
+ +=========================+===============+============+===============+===========================================================================+
+ | TEST CASE | TIER | DURATION | RESULT | URL |
+ +=========================+===============+============+===============+===========================================================================+
+ | healthcheck | healthcheck | 03:07 | PASS | |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | vping_ssh | smoke | 00:56 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac13d79377c54b278bd4c1 |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | vping_userdata | smoke | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac14019377c54b278bd4c2 |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | tempest_smoke_serial | smoke | 16:05 | FAIL | http://testresults.opnfv.org/test/api/v1/results/57ac17ca9377c54b278bd4c3 |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | rally_sanity | smoke | 12:19 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1aad9377c54b278bd4cd |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | odl | sdn_suites | 00:24 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ad09377c54b278bd4ce |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | promise | features | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ae59377c54b278bd4cf |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+
+Results are automatically pushed to the test results database, some additional
+result files are pushed to OPNFV artifact web sites.
+
+Based on the results stored in the result database, a `Functest reporting`_
+portal is also automatically updated. This portal provides information on:
+
+ * The overall status per scenario and per installer
+ * Tempest: Tempest test case including reported errors per scenario and installer
+ * vIMS: vIMS details per scenario and installer
+
+.. figure:: ../images/functest-reporting-status.png
+ :align: center
+ :alt: Functest reporting portal Fuel status page
Test Dashboard
References
==========
-.. _`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/#
+.. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/#
.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
.. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf
.. _`[7]`: https://github.com/OpenSCAP/openscap
.. _`[9]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml
.. _`[11]`: http://robotframework.org/
+.. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
OPNFV main site: opnfvmain_.
.. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
.. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py
.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml
+.. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
Code Review / functest.git / blobdiff