-The results in the console can be described as follows::
-
- FUNCTEST.info: Running vIMS test...
- 2016-02-04 13:46:25,025 - vIMS - INFO - Prepare OpenStack plateform (create tenant and user)
- 2016-02-04 13:46:25,312 - vIMS - INFO - Update OpenStack creds informations
- 2016-02-04 13:46:25,312 - vIMS - INFO - Upload some OS images if it doesn't exist
- 2016-02-04 13:46:25,566 - vIMS - INFO - centos_7 image doesn't exist on glance repository.
- Try downloading this image and upload on glance !
- 2016-02-04 13:47:06,167 - vIMS - INFO - ubuntu_14.04 image doesn't exist on glance repository.
- Try downloading this image and upload on glance !
- 2016-02-04 13:47:26,987 - vIMS - INFO - Update security group quota for this tenant
- 2016-02-04 13:47:27,193 - vIMS - INFO - Update cinder quota for this tenant
- 2016-02-04 13:47:27,746 - vIMS - INFO - Collect flavor id for cloudify manager server
- 2016-02-04 13:47:28,326 - vIMS - INFO - Prepare virtualenv for cloudify-cli
- 2016-02-04 13:48:00,657 - vIMS - INFO - Downloading the cloudify manager server blueprint
- 2016-02-04 13:48:03,391 - vIMS - INFO - Cloudify deployment Start Time:'2016-02-04 13:48:03'
- 2016-02-04 13:48:03,391 - vIMS - INFO - Writing the inputs file
- 2016-02-04 13:48:03,395 - vIMS - INFO - Launching the cloudify-manager deployment
- 2016-02-04 13:56:03,501 - vIMS - INFO - Cloudify-manager server is UP !
- 2016-02-04 13:56:03,502 - vIMS - INFO - Cloudify deployment duration:'480.1'
- 2016-02-04 13:56:03,502 - vIMS - INFO - Collect flavor id for all clearwater vm
- 2016-02-04 13:56:04,093 - vIMS - INFO - vIMS VNF deployment Start Time:'2016-02-04 13:56:04'
- 2016-02-04 13:56:04,093 - vIMS - INFO - Downloading the openstack-blueprint.yaml blueprint
- 2016-02-04 13:56:06,265 - vIMS - INFO - Writing the inputs file
- 2016-02-04 13:56:06,268 - vIMS - INFO - Launching the clearwater deployment
- 2016-02-04 14:11:27,101 - vIMS - INFO - The deployment of clearwater-opnfv is ended
- 2016-02-04 14:11:27,103 - vIMS - INFO - vIMS VNF deployment duration:'923.0'
- 2016-02-04 14:14:31,976 - vIMS - INFO - vIMS functional test Start Time:'2016-02-04 14:14:31'
- 2016-02-04 14:15:45,880 - vIMS - INFO - vIMS functional test duration:'73.9'
- 2016-02-04 14:15:46,113 - vIMS - INFO - Launching the clearwater-opnfv undeployment
- 2016-02-04 14:18:12,604 - vIMS - INFO - Launching the cloudify-manager undeployment
- 2016-02-04 14:18:51,808 - vIMS - INFO - Cloudify-manager server has been successfully removed!
- 2016-02-04 14:18:51,870 - vIMS - INFO - Removing vIMS tenant ..
- 2016-02-04 14:18:52,131 - vIMS - INFO - Removing vIMS user ..
-
-Please note that vIMS traces can bee summarized in several steps:
-
- * INFO: environment prepared successfully => environment OK
- * INFO - Cloudify-manager server is UP ! => orchestrator deployed
- * INFO - The deployment of clearwater-opnfv is ended => VNF deployed
- * Multiple Identities (UDP) - (6505550771, 6505550675) Passed => tests run
- * DEBUG - Pushing results to DB.... => tests saved
-
-
-Promise
-^^^^^^^
-
-The results can be observed in the console::
-
- Running test case: promise
- ----------------------------------------------
- FUNCTEST.info: Running PROMISE test case...
- 2016-02-04 07:10:37,735 - Promise- INFO - Creating tenant 'promise'...
- 2016-02-04 07:10:37,893 - Promise- INFO - Adding role '59828986a9a94dfaa852548599fde628' to tenant 'promise'...
- 2016-02-04 07:10:38,005 - Promise- INFO - Creating user 'promiser'...
- 2016-02-04 07:10:38,128 - Promise- INFO - Updating OpenStack credentials...
- 2016-02-04 07:10:38,157 - Promise- INFO - Creating image 'promise-img' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'...
- 2016-02-04 07:10:42,016 - Promise- INFO - Creating flavor 'promise-flavor'...
- 2016-02-04 07:10:42,836 - Promise- INFO - Exporting environment variables...
- 2016-02-04 07:10:42,836 - Promise- INFO - Running command: DEBUG=1 npm run -s test -- --reporter json
- 2016-02-04 07:10:51,505 - Promise- INFO - The test succeeded.
- ....
- **********************************
- Promise test summary
- **********************************
- Test start: Thu Feb 04 07:10:42 UTC 2016
- Test end: Thu Feb 04 07:10:51 UTC 2016
- Execution time: 8.7
- Total tests executed: 33
- Total tests failed: 0
- **********************************
-
-
-
-Functest in test Dashboard
-==========================
-
-The OPNFV testing group created a test collection database to collect the test
-results from CI.
-Any test project running on any lab integrated in CI can push the results to
-this database.
-This database can be used afterwards to see the evolution of the tests and
-compare the results versus the installers, the scenario or the labs.
-
-You can find more information about the dashboard from Testing Dashboard wiki
-page `[6]`_.
-
-
-Overall Architecture
---------------------
-
-The Test result management in Brahmaputra can be summarized as follows::
-
- +-------------+ +-------------+ +-------------+
- | | | | | |
- | Test | | Test | | Test |
- | Project #1 | | Project #2 | | Project #N |
- | | | | | |
- +-------------+ +-------------+ +-------------+
- | | |
- V V V
- +-----------------------------------------+
- | |
- | Test Rest API front end |
- | http://testresults.opnfv.org/testapi |
- | |
- +-----------------------------------------+
- A |
- | V
- | +-------------------------+
- | | |
- | | Test Results DB |
- | | Mongo DB |
- | | |
- | +-------------------------+
- |
- |
- +----------------------+
- | |
- | test Dashboard |
- | |
- +----------------------+
-
-The Test dashboard URL is: TODO LF
-A proto Test dashboard has been realized: http://testresults.opnfv.org/proto/
-
-Test API description
---------------------
-
-The Test API is used to declare pods, projects, test cases and test results. An
-additional method dashboard has been added to post-process the raw results. The
-data model is very basic, 4 objects are created:
-
- * Pods
- * Test projects
- * Test cases
- * Test results
-
-Pods::
-
- {
- "id": <ID>,
- "details": <URL description of the POD>,
- "creation_date": YYYY-MM-DD HH:MM:SS ,
- "name": <The POD Name>,
- "mode": <metal or virtual>
- },
-
-Test project::
-
- {
- "id": <ID>,
- "name": <Name of the Project>,
- "creation_date": "YYYY-MM-DD HH:MM:SS",
- "description": <Short description>
- },
-
-Test case::
-
- {
- "id": <ID>,
- "name":<Name of the test case>,
- "creation_date": "YYYY-MM-DD HH:MM:SS",
- "description": <short description>,
- "url":<URL for longer description>
- },
-
-Test results::
-
- {
- "_id": <ID,
- "project_name": <Reference to project>,
- "pod_name": <Reference to POD where the test was executed>,
- "version": <Scenario on which the test was executed>,
- "installer": <Installer Apex or Compass or Fuel or Joid>,
- "description": <Short description>,
- "creation_date": "YYYY-MM-DD HH:MM:SS",
- "case_name": <Reference to the test case>
- "details":{
- <- the results to be put here ->
- }
-
-
-For Brahmaputra, we got:
-
- * 16 pods
- * 18 projects
- * 101 test cases
-
-The projects and the test cases have been frozen in December.
-But all were not ready for Brahmaputra.
-
-
-
-The API can described as follows:
-
-Version:
-
- +--------+--------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+==========================+=========================================+
- | GET | /version | Get API version |
- +--------+--------------------------+-----------------------------------------+
-
-
-Pods:
-
- +--------+--------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+==========================+=========================================+
- | GET | /pods | Get the list of declared Labs (PODs) |
- +--------+--------------------------+-----------------------------------------+
- | POST | /pods | Declare a new POD |
- | | | Content-Type: application/json |
- | | | { |
- | | | "name": "pod_foo", |
- | | | "creation_date": "YYYY-MM-DD HH:MM:SS"|
- | | | } |
- +--------+--------------------------+-----------------------------------------+
-
-Projects:
-
- +--------+--------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+==========================+=========================================+
- | GET | /test_projects | Get the list of test projects |
- +--------+--------------------------+-----------------------------------------+
- | GET |/test_projects/{project} | Get details on {project} |
- | | | |
- +--------+--------------------------+-----------------------------------------+
- | POST | /test_projects | Add a new test project |
- | | | Content-Type: application/json |
- | | | { |
- | | | "name": "project_foo", |
- | | | "description": "whatever you want" |
- | | | } |
- +--------+--------------------------+-----------------------------------------+
- | PUT | /test_projects/{project} | Update a test project |
- | | | |
- | | | Content-Type: application/json |
- | | | { |
- | | | <the field(s) you want to modify> |
- | | | } |
- +--------+--------------------------+-----------------------------------------+
- | DELETE | /test_projects/{project} | Delete a test project |
- +--------+--------------------------+-----------------------------------------+
-
-
-Test cases:
-
- +--------+--------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+==========================+=========================================+
- | GET | /test_projects/{project}/| Get the list of test cases of {project} |
- | | cases | |
- +--------+--------------------------+-----------------------------------------+
- | POST | /test_projects/{project}/| Add a new test case to {project} |
- | | cases | Content-Type: application/json |
- | | | { |
- | | | "name": "case_foo", |
- | | | "description": "whatever you want" |
- | | | "creation_date": "YYYY-MM-DD HH:MM:SS"|
- | | | "url": "whatever you want" |
- | | | } |
- +--------+--------------------------+-----------------------------------------+
- | PUT | /test_projects/{project}?| Modify a test case of {project} |
- | | case_name={case} | |
- | | | Content-Type: application/json |
- | | | { |
- | | | <the field(s) you want to modify> |
- | | | } |
- +--------+--------------------------+-----------------------------------------+
- | DELETE | /test_projects/{project}/| Delete a test case |
- | | case_name={case} | |
- +--------+--------------------------+-----------------------------------------+
-
-Test Results:
-
- +--------+--------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+==========================+=========================================+
- | GET |/results/project={project}| Get the test results of {project} |
- +--------+--------------------------+-----------------------------------------+
- | GET |/results/case={case} | Get the test results of {case} |
- +--------+--------------------------+-----------------------------------------+
- | GET |/results?pod={pod} | get the results on pod {pod} |
- +--------+--------------------------+-----------------------------------------+
- | GET |/results?installer={inst} | Get the test results of installer {inst}|
- +--------+--------------------------+-----------------------------------------+
- | GET |/results?version={version}| Get the test results of scenario |
- | | | {version}. Initially the version param |
- | | | was reflecting git version, in Functest |
- | | | it was decided to move to scenario |
- +--------+--------------------------+-----------------------------------------+
- | GET |/results?project={project}| Get all the results of the test case |
- | |&case={case} | {case} of the project {project} with |
- | |&version={scenario} | version {scenario} installed by |
- | |&installer={installer} | {installer} on POD {pod} stored since |
- | |&pod={pod} | {days} days |
- | | | {project_name} and {case_name} are |
- | |&period={days} | mandatory, the other parameters are |
- | | | optional. |
- +--------+--------------------------+-----------------------------------------+
- | POST | /results | Add a new test results |
- | | | Content-Type: application/json |
- | | | { |
- | | | "project_name": "project_foo", |
- | | | "case_name": "case_foo", |
- | | | "pod_name": "pod_foo", |
- | | | "installer": "installer_foo", |
- | | | "version": "scenario_foo", |
- | | | "details": <your results> |
- | | | } |
- +--------+--------------------------+-----------------------------------------+
-
-
-Dashboard:
-
- +--------+--------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+==========================+=========================================+
- | GET |/dashboard? | Get all the dashboard ready results of |
- | |&project={project} | {case} of the project {project} |
- | |&case={case} | version {scenario} installed by |
- | |&version={scenario} | {installer} on POD {pod} stored since |
- | |&installer={installer} | {days} days |
- | |&pod={pod} | |
- | |&period={days} | {project_name} and {case_name} are |
- | | | mandatory, the other parameters are |
- | | | optional. |
- +--------+--------------------------+-----------------------------------------+
-
-
-The results with dashboard method are post-processed from raw results.
-Please note that dashboard results are not stored. Only raw results are stored.