-Openstack capabilities (snapshot, clean, tacker,..).
-These functions can be found under <repo>/functest/utils and can be described as
-follows:
-
-functest/utils/
-|-- config.py
-|-- constants.py
-|-- env.py
-|-- functest_logger.py
-|-- functest_utils.py
-|-- openstack_clean.py
-|-- openstack_snapshot.py
-|-- openstack_tacker.py
-`-- openstack_utils.py
-
-Note that for Openstack, keystone v3 is now deployed by default by compass,
-fuel and joid in Danube. All installers still support keysone v2 (deprecated in
-next version).
-
-Test collection framework
-=========================
-
-The OPNFV testing group created a test collection database to collect
-the test results from CI:
-
-
- http://testresults.opnfv.org/test/swagger/spec.html
-
- Authentication: opnfv/api@opnfv
-
-Any test project running on any lab integrated in CI can push the
-results to this database.
-This database can be used to see the evolution of the tests and compare
-the results versus the installers, the scenarios or the labs.
-
-
-Overall Architecture
---------------------
-The Test result management can be summarized as follows::
-
- +-------------+ +-------------+ +-------------+
- | | | | | |
- | Test | | Test | | Test |
- | Project #1 | | Project #2 | | Project #N |
- | | | | | |
- +-------------+ +-------------+ +-------------+
- | | |
- V V V
- +-----------------------------------------+
- | |
- | Test Rest API front end |
- | http://testresults.opnfv.org/test |
- | |
- +-----------------------------------------+
- A |
- | V
- | +-------------------------+
- | | |
- | | Test Results DB |
- | | Mongo DB |
- | | |
- | +-------------------------+
- |
- |
- +----------------------+
- | |
- | test Dashboard |
- | |
- +----------------------+
-
-Test API description
---------------------
-The Test API is used to declare pods, projects, test cases and test
-results. Pods are the pods used to run the tests.
-The results pushed in the database are related to pods, projects and
-cases. If you try to push results of test done on non referenced pod,
-the API will return an error message.
-
-An additional method dashboard has been added to post-process
-the raw results in release Brahmaputra (deprecated in Colorado).
-
-The data model is very basic, 4 objects are created:
-
- * Pods
- * Projects
- * Testcases
- * Results
-
-Pods::
-
- {
- "id": <ID>,
- "details": <URL description of the POD>,
- "creation_date": "YYYY-MM-DD HH:MM:SS",
- "name": <The POD Name>,
- "mode": <metal or virtual>,
- "role": <ci-pod or community-pod or single-node>
- },
-
-Projects::
-
- {
- "id": <ID>,
- "name": <Name of the Project>,
- "creation_date": "YYYY-MM-DD HH:MM:SS",
- "description": <Short description>
- },
-
-Testcases::
-
- {
- "id": <ID>,
- "name":<Name of the test case>,
- "project_name":<Name of belonged project>,
- "creation_date": "YYYY-MM-DD HH:MM:SS",
- "description": <short description>,
- "url":<URL for longer description>
- },
-
-Results::
-
- {
- "_id": <ID>,
- "case_name": <Reference to the test case>,
- "project_name": <Reference to project>,
- "pod_name": <Reference to POD where the test was executed>,
- "installer": <Installer Apex or Compass or Fuel or Joid>,
- "version": <master or Colorado or Brahmaputra>,
- "start_date": "YYYY-MM-DD HH:MM:SS",
- "stop_date": "YYYY-MM-DD HH:MM:SS",
- "build_tag": <such as "jenkins-functest-fuel-baremetal-daily-master-108">,
- "scenario": <Scenario on which the test was executed>,
- "criteria": <PASS or FAILED>,
- "trust_indicator": {
- "current": 0,
- "histories": []
- }
- }
-
-The API can described as follows. For detailed information, please go to
-
- http://testresults.opnfv.org/test/swagger/spec.html
-
- Authentication: opnfv/api@opnfv
-
-Version:
-
- +--------+--------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+==========================+=========================================+
- | GET | /versions | Get all supported API versions |
- +--------+--------------------------+-----------------------------------------+
-
-
-Pods:
-
- +--------+----------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+============================+=========================================+
- | GET | /api/v1/pods | Get the list of declared Labs (PODs) |
- +--------+----------------------------+-----------------------------------------+
- | POST | /api/v1/pods | Declare a new POD |
- | | | Content-Type: application/json |
- | | | { |
- | | | "name": "pod_foo", |
- | | | "mode": "metal", |
- | | | "role": "ci-pod", |
- | | | "details": "it is a ci pod" |
- | | | } |
- +--------+----------------------------+-----------------------------------------+
- | GET | /api/v1/pods/{pod_name} | Get a declared POD |
- +--------+----------------------------+-----------------------------------------+
-
-Projects:
-
- +--------+----------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+============================+=========================================+
- | GET | /api/v1/projects | Get the list of declared projects |
- +--------+----------------------------+-----------------------------------------+
- | POST | /api/v1/projects | Declare a new test project |
- | | | Content-Type: application/json |
- | | | { |
- | | | "name": "project_foo", |
- | | | "description": "whatever you want" |
- | | | } |
- +--------+----------------------------+-----------------------------------------+
- | DELETE | /api/v1/projects/{project} | Delete a test project |
- +--------+----------------------------+-----------------------------------------+
- | GET | /api/v1/projects/{project} | Get details on a {project} |
- | | | |
- +--------+----------------------------+-----------------------------------------+
- | PUT | /api/v1/projects/{project} | Update a test project |
- | | | |
- | | | Content-Type: application/json |
- | | | { |
- | | | <the field(s) you want to modify> |
- | | | } |
- +--------+----------------------------+-----------------------------------------+
-
-
-Testcases:
-
- +--------+----------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+============================+=========================================+
- | GET | /api/v1/projects/{project}/| Get the list of testcases of {project} |
- | | cases | |
- +--------+----------------------------+-----------------------------------------+
- | POST | /api/v1/projects/{project}/| Add a new test case to {project} |
- | | cases | Content-Type: application/json |
- | | | { |
- | | | "name": "case_foo", |
- | | | "description": "whatever you want" |
- | | | "url": "whatever you want" |
- | | | } |
- +--------+----------------------------+-----------------------------------------+
- | DELETE | /api/v1/projects/{project}/| Delete a test case |
- | | cases/{case} | |
- +--------+----------------------------+-----------------------------------------+
- | GET | /api/v1/projects/{project}/| Get a declared test case |
- | | cases/{case} | |
- +--------+----------------------------+-----------------------------------------+
- | PUT | /api/v1/projects/{project}?| Modify a test case of {project} |
- | | cases/{case} | |
- | | | Content-Type: application/json |
- | | | { |
- | | | <the field(s) you want to modify> |
- | | | } |
- +--------+----------------------------+-----------------------------------------+
-
-Results:
-
- +--------+----------------------------+------------------------------------------+
- | Method | Path | Description |
- +========+============================+==========================================+
- | GET | /api/v1/results | Get all the test results |
- +--------+----------------------------+------------------------------------------+
- | POST | /api/v1/results | Add a new test results |
- | | | Content-Type: application/json |
- | | | { |
- | | | "project_name": "project_foo", |
- | | | "scenario": "odl-l2", |
- | | | "stop_date": "2016-05-28T14:42:58.384Z", |
- | | | "trust_indicator": 0.5, |
- | | | "case_name": "vPing", |
- | | | "build_tag": "", |
- | | | "version": "Colorado", |
- | | | "pod_name": "pod_foo", |
- | | | "criteria": "PASS", |
- | | | "installer": "fuel", |
- | | | "start_date": "2016-05-28T14:41:58.384Z",|
- | | | "details": <your results> |
- | | | } |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of {case} |
- | | case={case} | |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of build_tag |
- | | build_tag={tag} | {tag}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get last {N} records of test results |
- | | last={N} | |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of scenario |
- | | scenario={scenario} | {scenario}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of trust_indicator |
- | | trust_indicator={ind} | {ind}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of last days |
- | | period={period} | {period}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of {project} |
- | | project={project} | |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of version |
- | | version={version} | {version}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of criteria |
- | | criteria={criteria} | {criteria}. |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | get the results on pod {pod} |
- | | pod={pod} | |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the test results of installer {inst} |
- | | installer={inst} | |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results? | Get the results according to combined |
- | | <query conditions> | query conditions supported above |
- +--------+----------------------------+------------------------------------------+
- | GET | /api/v1/results/{result_id}| Get the test result by result_id |
- +--------+----------------------------+------------------------------------------+
-
-Scenarios:
-
- +--------+----------------------------+-----------------------------------------+
- | Method | Path | Description |
- +========+============================+=========================================+
- | GET | /api/v1/scenarios | Get the list of declared scenarios |
- +--------+----------------------------+-----------------------------------------+
- | POST | /api/v1/scenario | Declare a new scenario |
- +--------+----------------------------+-----------------------------------------+
- | GET | /api/v1/scenario? | Get a declared scenario |
- | | <query conditions> | |
- +--------+----------------------------+-----------------------------------------+
-
-
-The code of the API is hosted in the releng repository `[6]`_.
-The static documentation of the API can be found at `[17]`_.
-The test API has been dockerized and may be installed locally in your
-lab. See `[15]`_ for details.
-
-The deployment of the test API has been automated.
-A jenkins job manages:
- * the unit tests of the test api
- * the creation of a new docker file
- * the deployment of the new test api
- * the archive of the old test api
- * the backup of the Mongo DB
-
-Test API Authorization
-~~~~~~~~~~~~~~~~~~~~~~
-
-PUT/DELETE/POST operations of the testapi now require token based authorization. The token needs
-to be added in the request using a header 'X-Auth-Token' for access to the database.
-
-e.g::
- headers['X-Auth-Token']
-
-The value of the header i.e the token can be accessed in the jenkins environment variable
-*TestApiToken*. The token value is added as a masked password.
-
-.. code-block:: python
-
- headers['X-Auth-Token'] = os.environ.get('TestApiToken')
-
-The above example is in Python. Token based authentication has been added so that only ci pods
-jenkins job can have access to the database.
-
-Please note that currently token authorization is implemented but is not yet enabled.
-
- Automatic reporting
- ===================
-
- An automatic reporting page has been created in order to provide a
- consistant view of the scenarios.
- In this page, each scenario is evaluated according to test criteria.
- The code for the automatic reporting is available at `[8]`_.
-
- The results are collected from the centralized database every day and,
- per scenario. A score is calculated based on the results from the last
- 10 days. This score is the addition of single test scores. Each test
- case has a success criteria reflected in the criteria field from the
- results.
-
- Considering an instance of a scenario os-odl_l2-nofeature-ha, the
- scoring is the addition of the scores of all the runnable tests from the
- categories (tiers healthcheck, smoke and features)
- corresponding to this scenario.
-
-
- +---------------------+---------+---------+---------+---------+
- | Test | Apex | Compass | Fuel | Joid |
- +=====================+=========+=========+=========+=========+
- | vPing_ssh | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | vPing_userdata | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | tempest_smoke_serial| X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | rally_sanity | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | odl | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | promise | | | X | X |
- +---------------------+---------+---------+---------+---------+
- | doctor | X | | X | |
- +---------------------+---------+---------+---------+---------+
- | security_scan | X | | | |
- +---------------------+---------+---------+---------+---------+
- | parser | | | X | |
- +---------------------+---------+---------+---------+---------+
- | copper | X | | | X |
- +---------------------+---------+---------+---------+---------+
-
- All the testcases listed in the table are runnable on
- os-odl_l2-nofeature scenarios.
- If no result is available or if all the results are failed, the test
- case get 0 point.
- If it was succesfull at least once but not anymore during the 4 runs,
- the case get 1 point (it worked once).
- If at least 3 of the last 4 runs were successful, the case get 2 points.
- If the last 4 runs of the test are successful, the test get 3 points.
-
- In the example above, the target score for fuel/os-odl_l2-nofeature-ha
- is 3x6 = 18 points.
-
- The scenario is validated per installer when we got 3 points for all
- individual test cases (e.g 18/18).
- Please note that complex or long duration tests are not considered for
- the scoring. The success criteria are not always easy to define and may
- require specific hardware configuration. These results however provide
- a good level of trust on the scenario.
-
- A web page is automatically generated every day to display the status.
- This page can be found at `[9]`_. For the status, click on Status menu,
- you may also get feedback for vims and tempest_smoke_serial test cases.
-
- Any validated scenario is stored in a local file on the web server. In
- fact as we are using a sliding windows to get results, it may happen
- that a successful scenarios is no more run (because considered as
- stable) and then the number of iterations (4 needed) would not be
- sufficient to get the green status.
-
- Please note that other test cases (e.g. sfc_odl, bgpvpn) need also
- ODL configuration addons and as a consequence specific scenario.
- There are not considered as runnable on the generic odl_l2 scenario.
-
-Dashboard
-=========
-
-Dashboard is used to provide a consistant view of the results collected
-in CI.
-The results showed on the dashboard are post processed from the Database,
-which only contains raw results.