The goal of this guide consists in providing some guidelines for new developers
involved in test areas.
-For the description of the ecosystem, see `[1]`_.
+For the description of the ecosystem, see `[DEV1]`_.
=================
Developer journey
* Develop new test cases
* Develop frameworks
* Develop tooling (reporting, dashboards, graphs, middleware,...)
+ * Troubleshoot results
+ * Post-process results
These different tasks may be done within a specific project or as a shared
resource accross the different projects.
specific tooling, please report to the test project user guide. The tooling used
by several test projects will be detailed in this document.
+The best event to meet the testing community is probably the plugfest. Such an
+event is organized after each release. Most of the test projects are present.
+
+The summit is also a good opportunity to meet most of the actors `[DEV4]`_.
+
Be involved in the testing group
================================
long duration, documentation, ...) and align tooling or best practices.
A weekly meeting is organized, the agenda may be amended by any participant.
-2 slots have been defined (US and APAC). Agendas and minutes are public.See
-`[8]`_ for details.
+2 slots have been defined (US/Europe and APAC). Agendas and minutes are public.
+See `[DEV3]`_ for details.
The testing group IRC channel is #opnfv-testperf
Best practices
Repository structure
----------------------
+--------------------
Most of the projects have a similar structure, which can be defined as follows::
the different projects and reduce the risk of overlapping. In fact if project A
provides an API to deploy a traffic generator, it is better to reuse it rather
than implementing a new way to deploy it. This approach has not been implemented
-yet but the prerequisite consiting in exposing and API has already been done by
+yet but the prerequisites consiting in exposing and API has already been done by
several test projects.
Dockerization
-------------
-Dockerization has been introduced in Brahmaputra and adopted by the test
+Dockerization has been introduced in Brahmaputra and adopted by most of the test
projects. Docker containers are pulled on the jumphost of OPNFV POD.
-
-
-
-
-Unit tests
-----------
-
-
-Traffic generators
-------------------
+<TODO Jose/Mark/Alec>
+
+Code quality
+------------
+
+It is recommended to control the quality of the code of the testing projects,
+and more precisely to implement some verifications before any merge:
+ * pep8
+ * pylint
+ * unit tests (python 2.7)
+ * unit tests (python 3.5)
+
+
+The code of the test project must be covered by unit tests. The coverage
+shall be reasonable and not decrease when adding new features to the framework.
+The use of tox is recommended.
+It is possible to implement strict rules (no decrease of pylint score, unit
+test coverages) on critical python classes.
+
+
+Third party tooling
+-------------------
+
+Several test projects integrate third party tooling for code quality check
+and/or traffic generation. Some of the tools can be listed as follows:
+
++---------------+----------------------+------------------------------------+
+| Project | Tool | Comments |
++===============+======================+====================================+
+| Bottlenecks | TODO | |
++---------------+----------------------+------------------------------------+
+| Functest | Tempest | OpenStack test tooling |
+| | Rally | OpenStack test tooling |
+| | Refstack | OpenStack test tooling |
+| | RobotFramework | Used for ODL tests |
++---------------+----------------------+------------------------------------+
+| QTIP | Unixbench | |
+| | RAMSpeed | |
+| | nDPI | |
+| | openSSL | |
+| | inxi | |
++---------------+----------------------+------------------------------------+
+| Storperf | TODO | |
++---------------+----------------------+------------------------------------+
+| VSPERF | TODO | |
++---------------+----------------------+------------------------------------+
+| Yardstick | Moongen | Traffic generator |
+| | Trex | Traffic generator |
+| | Pktgen | Traffic generator |
+| | IxLoad, IxNet | Traffic generator |
+| | SPEC | Compute |
+| | Unixbench | Compute |
+| | RAMSpeed | Compute |
+| | LMBench | Compute |
+| | Iperf3 | Network |
+| | Netperf | Network |
+| | Pktgen-DPDK | Network |
+| | Testpmd | Network |
+| | L2fwd | Network |
+| | Fio | Storage |
+| | Bonnie++ | Storage |
++---------------+----------------------+------------------------------------+
======================================
Testing group configuration parameters
======================================
-
Testing categories
==================
and tier fields, it shall be possible to create heuristic maps.
-=================
-TestAPI framework
-=================
-
-The OPNFV testing group created a test collection database to collect
-the test results from CI:
-
-
- http://testresults.opnfv.org/test/swagger/spec.html
-
-Any test project running on any lab integrated in CI can push the
-results to this database.
-This database can be used to see the evolution of the tests and compare
-the results versus the installers, the scenarios or the labs.
-It is used to produce a dashboard with the current test status of the project.
-
-
-Overall Architecture
-====================
-The Test result management can be summarized as follows::
-
- +-------------+ +-------------+ +-------------+
- | | | | | |
- | Test | | Test | | Test |
- | Project #1 | | Project #2 | | Project #N |
- | | | | | |
- +-------------+ +-------------+ +-------------+
- | | |
- V V V
- +-----------------------------------------+
- | |
- | Test Rest API front end |
- | |
- +-----------------------------------------+
- | |
- | V
- | +-------------------------+
- | | |
- | | Test Results DB |
- | | Mongo DB |
- | | |
- | +-------------------------+
- |
- |
- +----------------------+
- | |
- | test Dashboard |
- | |
- +----------------------+
-
-TestAPI description
-===================
-The TestAPI is used to declare pods, projects, test cases and test
-results. Pods are the sets of bare metal or virtual servers and networking
-equipments used to run the tests.
-
-The results pushed in the database are related to pods, projects and test cases.
-If you try to push results of test done on non referenced pod, the API will
-return an error message.
-
-An additional method dashboard has been added to post-process
-the raw results in release Brahmaputra (deprecated in Colorado).
-
-The data model is very basic, 5 objects are created:
-
- * Pods
- * Projects
- * Testcases
- * Results
- * Scenarios
-
-The code of the API is hosted in the releng repository `[6]`_.
-The static documentation of the API can be found at `[7]`_.
-The TestAPI has been dockerized and may be installed locally in your
-lab. See `[15]`_ for details.
-
-The deployment of the TestAPI has been automated.
-A jenkins job manages:
-
- * the unit tests of the TestAPI
- * the creation of a new docker file
- * the deployment of the new TestAPI
- * the archive of the old TestAPI
- * the backup of the Mongo DB
-
-TestAPI Authorization
----------------------
-
-PUT/DELETE/POST operations of the TestAPI now require token based authorization. The token needs
-to be added in the request using a header 'X-Auth-Token' for access to the database.
-
-e.g::
- headers['X-Auth-Token']
-
-The value of the header i.e the token can be accessed in the jenkins environment variable
-*TestApiToken*. The token value is added as a masked password.
-
-.. code-block:: python
-
- headers['X-Auth-Token'] = os.environ.get('TestApiToken')
-
-The above example is in Python. Token based authentication has been added so that only ci pods
-jenkins job can have access to the database.
-
-Please note that currently token authorization is implemented but is not yet enabled.
-
-===============================
-Feedback from the testing group
-================================
-
-Test case catalog
-===================
-
-A test case catalog has been realized. Roll over the project then click to get
-the list of test cases, click on the case to get more details.
-
-.. raw:: html
- :url: http://testresults.opnfv.org/reporting2/reporting/index.html#!/select/visual
-
-Reporting
-=========
-
-An automatic reporting page has been created in order to provide a
-consistent view of the scenarios.
-
-In this page, each scenario is evaluated according to test criteria.
-The code for the automatic reporting is available at `[8]`_.
-
-The results are collected from the centralized database every day and,
-per scenario. A score is calculated based on the results from the last
-10 days.
-
-Dashboard
-=========
-
-Dashboard is used to provide a consistent view of the results collected in CI.
-The results showed on the dashboard are post processed from the Database,
-which only contains raw results.
-
-It can be used in addition of the reporting page (high level view) to allow
-the creation of specific graphs according to what the test owner wants to show.
-
-In Brahmaputra, a basic home made dashboard was created in Functest.
-In Colorado, Yardstick adopted Grafana (time based graphs) and ELK (complex
-graphs).
-Since Danube, the testing community decided to adopt ELK framework and to rely
-on bitergia. It was not implemented for Danube but it is planned for Euphrates.
-
-Bitergia already provides a dashboard for code and infrastructure.
-A new Test tab will be added. The dataset will be built by consuming
-the TestAPI.
-
-See `[3]`_ for details.
-
-
=======
How TOs
=======
Where can I find information on the different test projects?
===========================================================
+On http://docs.opnfv.org! A section is dedicated to the testing projects. You
+will find the overview of the ecosystem and the links to the project documents.
+
+Another source is the testing wiki on https://wiki.opnfv.org/display/testing
+
+You may also contact the testing group on the IRC channel #opnfv-testperf or by
+mail at test-wg AT lists.opnfv.org (testing group) or opnfv-tech-discuss AT
+lists.opnfv.org (generic technical discussions).
How can I contribute to a test project?
=======================================
+As any project, the best solution is to contact the project. The project
+members with their email address can be found under
+https://git.opnfv.org/<project>/tree/INFO
+
+You may also send a mail to the testing mailing list or use the IRC channel
+#opnfv-testperf
Where can I find hardware resources?
====================================
+You should discuss this topic with the project you are working with. If you need
+access to an OPNFV community POD, it is possible to contact the infrastructure
+group. Depending on your needs (scenario/installer/tooling), it should be
+possible to find free time slots on one OPNFV community POD from the Pharos
+federation. Create a JIRA ticket to describe your needs on
+https://jira.opnfv.org/projects/INFRA.
+You must already be an OPNFV contributor. See
+https://wiki.opnfv.org/display/DEV/Developer+Getting+Started.
+
+Please note that lots of projects have their own "how to contribute" or
+"get started" page on the OPNFV wiki.
How do I integrate my tests in CI?
==================================
-
+It shall be discussed directly with the project you are working with. It is
+done through jenkins jobs calling testing project files but the way to onboard
+cases differ from one project to another.
How to declare my tests in the test Database?
=============================================
+If you have access to the test API swagger (access granted to contributors), you
+may use the swagger interface of the test API to declare your project.
+The URL is http://testresults.opnfv.org/test/swagger/spec.html.
+
+.. figure:: ../../../images/swaggerUI.png
+ :align: center
+ :alt: Testing Group Test API swagger
+Click on *Spec*, the list of available methods must be displayed.
+
+.. figure:: ../../../images/API-operations.png
+ :align: center
+ :alt: Testing Group Test API swagger
+
+For the declaration of a new project use the POST /api/v1/projects method.
+For the declaration of new test cases in an existing project, use the POST
+ /api/v1/projects/{project_name}/cases method
+
+ .. figure:: ../../../images/CreateCase.png
+ :align: center
+ :alt: Testing group declare new test case
How to push your results into the Test Database?
================================================
If you want to push your results from CI, you just have to call the API
at the end of your script.
-You can also reuse a python function defined in functest_utils.py `[5]`_
+You can also reuse a python function defined in functest_utils.py `[DEV2]`_
Where can I find the documentation on the test API?
===================================================
+The Test API is now documented in this document (see sections above).
+You may also find autogenerated documentation in
http://artifacts.opnfv.org/releng/docs/testapi.html
-
-
+A web protal is also under construction for certification at
+http://testresults.opnfv.org/test/#/
I have tests, to which category should I declare them?
======================================================
-
-
+See table above.
The main ambiguity could be between features and VNF.
In fact sometimes you have to spawn VMs to demonstrate the capabilities of the
References
==========
-_`[1]`: http://docs.opnfv.org/en/stable-danube/testing/ecosystem/overview.html
-
-_`[2]`: http://www.opnfv.org
-
-_`[3]`: https://wiki.opnfv.org/display/testing/Result+alignment+for+ELK+post-processing
-
-_`[4]`: https://wiki.opnfv.org/display/INF/CI+Scenario+Naming
-
-_`[5]`: https://git.opnfv.org/functest/tree/functest/utils/functest_utils.py#176
+`[DEV1]`_: OPNFV Testing Ecosystem
-_`[6]`: https://git.opnfv.org/functest/tree/releng
+`[DEV2]`_: Python code sample to push results into the Database
-_`[7]`: http://artifacts.opnfv.org/releng/docs/testapi.html
+`[DEV3]`_: Testing group wiki page
-_`[8]`: https://wiki.opnfv.org/display/meetings/Test+Working+Group+Weekly+Meeting
+`[DEV4]`_: Conversation with the testing community, OPNFV Beijing Summit
+.. _`[DEV1]`: http://docs.opnfv.org/en/latest/testing/ecosystem/index.html
+.. _`[DEV2]`: https://git.opnfv.org/functest/tree/functest/utils/functest_utils.py#176
+.. _`[DEV3]`: https://wiki.opnfv.org/display/meetings/Test+Working+Group+Weekly+Meeting
+.. _`[DEV4]`: https://www.youtube.com/watch?v=f9VAUdEqHoA
IRC support chan: #opnfv-testperf