--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Huawei Technologies Co.,Ltd, and others
+
+====================================
+Dovetail as a Generic Test Framework
+====================================
+
+.. toctree::
+ :maxdepth: 2
+
+
+Overview
+========
+
+Dovetail is responsible for the technical realization of the OPNFV Verified
+Program (OVP) and other compliance verification projects within the scope of
+the Linux Foundation Networking (LFN) umbrella projects.
+Dovetail provides a generic framework for executing a specific set of test cases
+which define the scope of a given compliance verification program, such as OVP.
+
+This document aims at introducing what Dovetail generic framework looks like and
+how to develop within this framework.
+
+
+Introduction of Dovetail Framework
+==================================
+
+The following diagram illustrates Dovetail generic framework.
+
+.. image:: ../../../images/dovetail_generic_framework.png
+ :align: center
+ :scale: 50%
+
+In this diagram, there are 5 main parts, `TestcaseFactory`, `TestRunnerFactory`,
+`CrawlerFactory`, `CheckerFactory` and test case groups.
+
+- **TestcaseFactory**: For each project, there needs to create its own
+ testcase class such as `FunctestTestcase` and `OnapVtpTestcase`. All these
+ classes are based on class `Testcase`. There are already many functions in this
+ base class which are mainly used to parse test case configuration files. If no
+ other special requirements exist, it only needs to initialize these classes with
+ different types. Otherwise, it needs to overload or add some functions.
+
+- **TestRunnerFactory**: Similar to `TestcaseFactory`, each project has its own
+ test runner class. Dovetail supports 2 kinds of test runners, `DockerRunner`
+ and `ShellRunner`. For projects based on Docker, it needs to create
+ their own test runner classes such as `FunctestRunner` which inherit from class
+ `DockerRunner`. For other projects that are based on Shell, it can use `ShellRunner`
+ directly. Test case runners provide many functions to support test cases runs
+ such as preparing test tool of each project, run all the commands defined by
+ each test case and clean the environment.
+
+- **Test case groups**: Each group is composed of one project configuration file
+ and a set of test cases belonging to this project. These groups are used as the
+ input of test runners to provide information of projects and test cases. For
+ `ShellRunner`, it only needs test case configuratons as the input.
+
+- **CrawlerFactory**: This is used to parse the results of test cases and record
+ them with unified format. The original result data report by each project is
+ different. So it needs to create different crawler classes for different projects
+ to parse their results.
+
+- **CheckerFactory**: This is used to check the result data generated by crawler.
+ Each project should have its own checker class due to the different requirements
+ of different projects.
+
+
+Development with Dovetail Framework
+===================================
+
+Everyone who is interested in developing Dovetail framework to integrate new upstream
+test cases will face one of the two following scenarios:
+
+- **Adding test cases that belong to integrated projects**: There are already some
+ projects integrated in Dovetail. These projects are coming from OPNFV (Open Platform
+ for NFV) and ONAP (Open Network Automation Platform) communities. It will be
+ much easier to add new test cases that belong to these projects.
+
+- **Adding test cases that not belong to integrated projects**: The test cases
+ may belong to other projects that haven't been integrated into Dovetail yet.
+ These projects could be in OPNFV, ONAP or other communities. This scenario is a
+ little more complicated.
+
+
+Test cases belonging to integrated projects
+-------------------------------------------
+
+Dovetail framework already includes a large amount of test cases. All these test
+cases are implemented by upstream projects in OPNFV and ONAP. The upstream
+projects already integrated in Dovetail are Functest, Yardstick and Bottlenecks
+from OPNFV and VNF SDK and VVP from ONAP.
+
+In order to add a test case belonging to one of these projects, there
+only need to add one test case configuration file which is in yaml format.
+Following is the introduction about how to use the file to add one new test case.
+Please refer to `Dovetail test case github
+<https://github.com/opnfv/dovetail/tree/master/etc/testcase>`_
+for all configuration files of all test cases.
+
+.. code-block:: bash
+
+ ---
+ Test case name in Dovetail:
+ name: Test case name in Dovetail
+ objective: Test case description
+ validate:
+ type: 'shell' or name of the project already integrated in Dovetail
+ testcase: The test case name called in this project
+ image_name: Name of the Docker image used to run this test
+ pre_condition:
+ - 'Commands needed to be executed before running this test'
+ - 'e.g. cp src_file dest_file'
+ cmds:
+ - 'Commands used to run this test case'
+ post_condition:
+ - 'Commands needed to be executed after running this test case'
+ report:
+ source_archive_files:
+ - test.log
+ dest_archive_files:
+ - path/to/archive/test.log
+ check_results_file: results.json
+ sub_testcase_list:
+ - sub_test_1
+ - sub_test_2
+ - sub_test_3
+
+This is the complete format of test case configuration file. Here are some
+detailed description for each of the configuration options.
+
+- **Test case name in Dovetail**: All test cases should be named as 'xxx.yyy.zzz'.
+ This is the name in Dovetail and has no relationship with its name in its own
+ project. The first part is used to identify the project where this test case
+ come from (e.g. functest, onap-vtp). The second part is used to classify this
+ test case according to test area (e.g. healthcheck, ha). Dovetail supports to
+ run whole test cases in one test suite with the same test area. Also the area
+ is used to group all test cases and generate the summary report at the end of
+ the test. The last part is special for this test case itself (e.g. image,
+ haproxy, csar). It's better to keep the file name the same as the test case
+ name to make it easier to find the config file according to this test case
+ name in Dovetail.
+
+- **validate**: This is the main section to define how to run this test case.
+
+ - **type**: This is the type of this test case. It can be `shell` which means
+ running this test case with Linux bash commands within Dovetail container. Also it
+ can be one of the projects already integrated in Dovetail (functest, yardstick,
+ bottlenecks, onap-vtp and onap-vvp). Then this type is used to map to its project
+ configuration yaml file. For example, in order to add a test case
+ in OPNFV project Functest to Dovetail framework, the type here should be
+ `functest`, and will map to `functest_config.yml` for more configurations
+ in project level. Please refer to `Dovetail project config github
+ <https://github.com/opnfv/dovetail/tree/master/etc/conf>`_ for more details.
+
+ - **testcase**: This is the name defined in its own project. One test case can
+ be uniquely identified by `type` and `testcase`. Take the test case
+ `functest.vping.ssh` as an example. Its `type` is 'functest' and `testcase`
+ is 'vping_ssh'. With these 2 properties, it can be uniquely identified. Users only
+ need to know that there is a test case named `functest.vping.ssh` in OVP
+ compliance test scope. Dovetail Framework will run `vping_ssh` within Functest
+ Docker container.
+
+ - **image_name**: [optional] If the type is `shell`, there is no need to give
+ this. For other types, there are default docker images defined in their project
+ configuration files. If this test case uses a different docker image, it needs
+ to overwrite it by adding `image_name` here. The `image_name` here should only
+ be the docker image name without tag. The tag is defined in project's configuration
+ file for all test cases belonging to this project.
+
+ - **pre_condition**: [optional] A list of all preparations needed by this
+ test case. If the list is the same as the default one in its project configuration
+ file, then there is no need to repeat it here. Otherwise, it's necessary to
+ overwrite it. If its type is `shell`, then all commands in `pre_condition`,
+ `cmds` and `post_condition` should be executable within Dovetail Ubuntu 14.04
+ Docker container. If its type is one of the Docker runner projects, then all
+ commands should be executable within their own containers. For Functest, it's
+ alpine 3.8. For Yardstick and Bottlenecks it's Ubuntu 16.04. For VNF SDK it's
+ Ubuntu 14.04. Also all these commands should not require network connection
+ because some commercial platforms may be offline environments in private labs.
+
+ - **cmds**: [optional] A list of all commands used to run this test case.
+
+ - **post_condition**: [optional] A list of all commands needed after executing
+ this test case such as some clean up operations.
+
+- **report**: This is the section for this test case to archive some log files and
+ provide the result file for reporting PASS or FAIL.
+
+ - **source_archive_files**: [optional] If there is no need to archive any files,
+ this section can be removed. Otherwise, this is a list of all source files
+ needed to be archived. All files generated by all integrated projects will be
+ put under `$DOVETAIL_HOME/results`. In order to classify and avoid overwriting
+ them, it needs to rename some important files or move them to new directories.
+ Navigating directory `$DOVETAIL_HOME/results` to find out all files
+ needed to be archived. The paths here should be relative ones according to
+ `$DOVETAIL_HOME/results`.
+
+ - **dest_archive_files**: [optional] This should be a list corresponding to the
+ list of `source_archive_files`. Also all paths here should be relative ones
+ according to `$DOVETAIL_HOME/results`.
+
+ - **check_results_file**: This should be the name and relative path of the result
+ file generated by this test case. Dovetail will parse this file to get the
+ result (PASS or FAIL).
+
+ - **sub_testcase_list**: [optional] This section is almost only for Tempest tests
+ in Functest. Take `functest.tempest.osinterop` as an example. The `sub_testcase_list`
+ here is an check list for this kind of tempest tests. Only when all sub test
+ cases list here are passed, this test case can be taken as PASS. The other kind
+ of tempest tests is `tempest_custom` such as `functest.tempest.image`. Besides
+ taking the `sub_testcase_list` as the check list, it's also used to generate an
+ input file of Functest to define the list of sub test cases to be tested.
+
+
+Test cases not belonging to integrated projects
+-----------------------------------------------
+
+If test cases waiting to be added into Dovetail do not belong to any project
+that is already integrated into Dovetail framework, then besides adding the test
+case configuration files introduced before, there are some other files needed to
+be added or modified.
+
+
+Step 1: Add a project configuration file
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For a new test case that belongs to a new project, it needs to create a project
+configuration file to define this new project in Dovetail first. Now Dovetail
+only supports integration of projects by using their Docker images. If this test
+case should be run with shell runner, then can only add test case configuration
+files with `type` 'shell' as describing before and skip the following steps. Following is
+the introduction of how to use project configuration file to add one new project
+into Dovetail. Please refer to `Dovetail projects configuration github
+<https://github.com/opnfv/dovetail/tree/master/etc/conf>`_ for all configuration
+files of all integrated projects.
+
+.. code-block:: bash
+
+ ---
+
+ {% set validate_testcase = validate_testcase or '' %}
+ {% set testcase = testcase or '' %}
+ {% set dovetail_home = dovetail_home or '' %}
+ {% set debug = debug or 'false' %}
+ {% set build_tag = build_tag or '' %}
+ {% set userconfig_dir = '/tmp/userconfig' %}
+ {% set patches_dir = '/tmp/patches' %}
+ {% set result_dir = '/tmp/results' %}
+
+ project name:
+ image_name: name of the docker image
+ docker_tag: tag of the docker image
+ opts: options needed such as '-itd'
+ envs: envs used to create containers such as '-e DEBUG={{debug}}'
+ volumes:
+ - '-v {{dovetail_home}}/pre_config:/home/opnfv/pre_config'
+ - '-v {{dovetail_home}}/userconfig:{{userconfig_dir}}'
+ - '-v {{dovetail_home}}/patches:{{patches_dir}}'
+ - '-v {{dovetail_home}}/results:{{result_dir}}'
+ patches_dir: {{patches_dir}}
+ pre_condition:
+ - 'Commands needed to be executed before running this test'
+ cmds:
+ - 'Commands used to run this test case'
+ post_condition:
+ - 'Commands needed to be executed after running this test case'
+ openrc: absolute path of openstack credential files
+ extra_container:
+ - container1_name
+ - container2_name
+
+This is the complete format of project configuration file. Here are some
+detailed description for each of the configuration options.
+
+- **Jinja Template**: At the begining of this yaml file, it uses Jinja template
+ to define some parameters that will be used somewhere in this file (e.g. result_dir).
+ Also there are some parameters providing by Dovetail framework as input of this
+ file, and other parameters can be defined by using these ones (e.g. testcase and
+ dovetail_home). The whole input parameters are list below.
+
+ - **validate_testcase**: This is the name of the test case instance which calls this
+ project configuration file. The name is provided by the configuration file
+ of this test case (validate -> testcase).
+
+ - **testcase**: This is the name of the test case which calls this project
+ configuration file. Different from `validate_testcase`, this is the name
+ defined in Dovetail not its own project.
+
+ - **os_insecure**: This is only for test cases aiming at OpenStack. This is
+ `True` or `False` according to `env_config.sh` file.
+
+ - **cacert**: This is also only for OpenStack test cases. It is the absolute
+ path of the OpenStack certificate provided in `env_config.sh` file.
+
+ - **deploy_scenario**: This is the input when running Dovetail with option
+ `--deploy-scenario`.
+
+ - **ram_num**: This is the input when running Dovetail with option
+ `--ram-num`.
+
+ - **dovetail_home**: This is the `DOVETAIL_HOME` getting from the ENV.
+
+ - **debug**: This is `True` or `False` according to the command running test
+ cases with or without option `--debug`.
+
+ - **build_tag**: This is a string includes the UUID generated by Dovetail.
+
+ - **host_url**: This is only for ONAP VNF SDK to get the HOST_URL provided
+ in `env_config.sh` file.
+
+ - **csar_file**: This is also only for ONAP VNF SDK to get the CSAR_FILE
+ provided in `env_config.sh` file.
+
+- **project name**: This is the project name defined in Dovetail. For example
+ OPNFV Functest project is named as 'functest' here in Dovetail. This project
+ name will be used by test case configuration files as well as somewhere in
+ Dovetail source code.
+
+- **image_name**: This is the name of the default Docker image for all test cases
+ within this project. Each test case can overwrite it with its own configuration.
+
+- **docker_tag**: This is the tag of all Docker images for all test cases within
+ this project. For each release, it should use one Docker image with a stable
+ and official release version.
+
+- **opts**: Here are all options used to run Docker containers except envs and
+ volume mappings (e.g. '-it --privileged=true').
+
+- **envs**: Here are all envs used to run Docker containers (e.g. '-e ONE=one
+ -e TWO=two').
+
+- **volumes**: A volume mapping list used to run Docker containers. Every project
+ should at least map the `$DOVETAIL_HOME/pre_config` and `$DOVETAIL_HOME/results`
+ in the test host to containers to get config files and collect all result files.
+
+- **patches_dir**: This is an absolute path of the patches applied to the containers.
+
+- **pre_condition**: A list of all default preparations needed by this project.
+ It can be overwritten by configurations of test cases.
+
+- **cmds**: A list of all default commands used to run all test cases within
+ this project. Also it can be overwritten by configurations of test cases.
+
+- **post_condition**: A list of all default cleaning commands needed by this
+ project.
+
+- **openrc**: [optional] If the system under test is OpenStack, then it needs to
+ provide the absolute path here to copy the credential file in the Test Host to
+ containers.
+
+
+Step 2: Add related classes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+After adding the project and test case configuration files, there also need to
+add some related classes into the source code.
+
+- **Test Case class**: Each project should have its own test case class in
+ `testcase.py` for `TestcaseFactory`.
+
+- **Test Runner class**: Each project should have its own test runner class in
+ `test_runner.py` for `TestRunnerFactory`.
+
+- **Crawler class**: Each project should have its own test results crawler class
+ in `report.py` for `CrawlerFactory`.
+
+- **Checker class**: Each project should have its own test results checker class
+ in `report.py` for `CheckerFactory`.
+
+
+Step 3: Create related logs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If the classes added in step2 have function `create_log`, then need to call
+these functions in `run.py` to initial the log instances at the very begining.
+
+
+Step 4: Update unit tests
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A patch is not going to be verified without 100% coverage when applying acceptance check.
Code Review / dovetail.git / commitdiff
