Release notes and test case decriptions, etc.
Change-Id: I0c67c02118dfd8ea3d8a4fc23225dc6d027ba3db
Signed-off-by: Yang Yu <Gabriel.yuyang@huawei.com>
| Nov 15th, 2017 | 1.6.1 | Bottlenecks Euphrates release 5.1 |
| | | |
+----------------+--------------------+-----------------------------------+
+| Apr 10th, 2018 | 1.7 | Bottlenecks Fraser release 6.0 |
+| | | |
++----------------+--------------------+-----------------------------------+
Summary
=======
Release Data
============
+Fraser Release Data
+-----------------------
+
++--------------------------------------+--------------------------------+
+| **Project** | Bottlenecks |
+| | |
++--------------------------------------+--------------------------------+
+| **Repo/tag** | * Bottlenecks/6.0.0 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Bottlenecks Docker image tag** | * 6.0.0 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Release designation** | * 6.0.0 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Release date** | * Apr 10th 2018 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Purpose of the delivery** | Fraser stable release |
+| | |
++--------------------------------------+--------------------------------+
+
Euphrates Release Data
-----------------------
+--------------------------------------+--------------------------------+
-Danube Deliverables
-===================
+Release Deliverables
+====================
Software Deliverables
---------------------
Documentatiion Deliverables
---------------------------
+**Bottlenecks documentation <fraser>**
+
+* Release Notes: http://docs.opnfv.org/en/stable-fraser/submodules/bottlenecks/docs/release/release-notes/release_notes.html
+* User Guide: http://docs.opnfv.org/en/stable-fraser/submodules/bottlenecks/docs/testing/user/userguide/index.html
+* Developer Guide: http://docs.opnfv.org/en/stable-fraser/submodules/bottlenecks/docs/testing/developer/devguide/index.html
+
**Bottlenecks documentation <euphrates>**
* Release Notes: http://docs.opnfv.org/en/stable-euphrates/submodules/bottlenecks/docs/release/release-notes/release_notes.html
* http://artifacts.opnfv.org/bottlenecks/colorado/1.0/releasenotes/index.html
* User Guide: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/docs/userguide/index.html
+Reason for Fraser
+=================
+
+* Introduction of data-plane soak tests aiming at benchmarking long duration stability of OPNFV platform
+* Introduction of security tests by cooperating with Moon project to validate the software security based authentication schemes
+* Introduction of multi-stack storage tests and Storperf storage test
+* Testing framework provides local supports for preparing images for openstack resulting in offline running supports for Bottlenecks
+
+These tests cases and refactoring further enhance the capability of Bottlenecks project and let it be adaptable to different usages.
+
Reason for Euphrates
====================
Installer Requirements
======================
-Bottlenecks Euphrates release is installer-agnostic which means that
+Bottlenecks releases after Euphrates are installer-agnostic which means that
the test cases could be executed over different deployments.
-However, the pod description file and openstack rc file
+However, the pod description file,
+openstack rc file and the admin credentials
are required to automatically get the SUT informations.
=====================
Operation System
----------------
-Ubuntu 14.04 is the default system. However, we do not do any linux core operation or call any system related api directly. The software could be runned in higher versions of Ubuntu with high probability. Thorough tests are not performed.
+Ubuntu 14.04 is the default system. However, we do not do any linux core operation or call any system related api directly. The software could be runned in higher versions of Ubuntu with high probability.
Cloud OS
--------
+-------------+----------------------+----------------------+
| Euphrates | Any | POSCA |
+-------------+----------------------+----------------------+
+| Fraser | Any | POSCA |
++-------------+----------------------+----------------------+
Test suite & Test case Description
==================================
-+--------+-------------------------------------------+
-|POSCA | posca_factor_ping |
-| +-------------------------------------------+
-| | posca_factor_system_bandwidth |
-| +-------------------------------------------+
-| | posca_facotor_soak_througputs |
-| +-------------------------------------------+
-| | posca_feature_vnf_scale_up |
-| +-------------------------------------------+
-| | posca_feature_vnf_scale_out |
-| +-------------------------------------------+
-| | posca_factor_storperf |
-| +-------------------------------------------+
-| | posca_factor_multistack_storage_parallel |
-| +-------------------------------------------+
-| | posca_factor_multistack_storage |
-+--------+-------------------------------------------+
++--------+----+-------------------------------------------+
+|POSCA | 1 | posca_factor_ping |
+| +----+-------------------------------------------+
+| | 2 | posca_factor_system_bandwidth |
+| +----+-------------------------------------------+
+| | 3 | posca_facotor_soak_througputs |
+| +----+-------------------------------------------+
+| | 4 | posca_feature_vnf_scale_up |
+| +----+-------------------------------------------+
+| | 5 | posca_feature_vnf_scale_out |
+| +----+-------------------------------------------+
+| | 6 | posca_factor_storperf |
+| +----+-------------------------------------------+
+| | 7 | posca_factor_multistack_storage_parallel |
+| +----+-------------------------------------------+
+| | 8 | posca_factor_multistack_storage |
+| +----+-------------------------------------------+
+| | 9 | posca_feature_moon_resources |
+| +----+-------------------------------------------+
+| | 10 | posca_feature_moon_tenants |
++--------+----+-------------------------------------------+
As for the abandoned test suite in the previous Bottlenecks releases, please
refer to http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/testing/user/userguide/deprecated.html.
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Tech and others.
+
+*************************************
+POSCA Factor Test of Soak Throughputs
+*************************************
+
+Test Case
+========
+
++-----------------------------------------------------------------------------+
+|Bottlenecks POSCA Soak Test Throughputs |
+| |
++--------------+--------------------------------------------------------------+
+|test case name| posca_factor_soak_throughputs |
+| | |
++--------------+--------------------------------------------------------------+
+|description | Long duration stability tests of data-plane traffic |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | config file: |
+| | /testsuite/posca/testcase_cfg/... |
+| | posca_factor_soak_throughputs.yaml |
+| | |
++--------------+--------------------------------------------------------------+
+|test result | THROUGHPUT,THROUGHPUT_UNITS,MEAN_LATENCY,LOCAL_CPU_UTIL, |
+| | REMOTE_CPU_UTIL,LOCAL_BYTES_SENT,REMOTE_BYTES_RECVD |
++--------------+--------------------------------------------------------------+
+
+Configuration
+============
+::
+
+ load_manager:
+ scenarios:
+ tool: netperf
+ test_duration_hours: 1
+ vim_pair_ttl: 300
+ vim_pair_lazy_cre_delay: 2
+ package_size:
+ threshhold:
+ package_loss: 0%
+ latency: 300
+
+ runners:
+ stack_create: yardstick
+ flavor:
+ yardstick_test_dir: "samples"
+ yardstick_testcase: "netperf_soak"
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Tech and others.
+
+************************************************************
+POSCA feature Test of Moon Security for resources per tenant
+************************************************************
+
+Test Case
+=========
+
++-----------------------------------------------------------------------------+
+|Bottlenecks POSCA Soak Test Throughputs |
+| |
++--------------+--------------------------------------------------------------+
+|test case name| posca_feature_moon_resources |
+| | |
++--------------+--------------------------------------------------------------+
+|description | Moon authentication capability test for maximum number of |
+| | authentication operations per tenant |
++--------------+--------------------------------------------------------------+
+|configuration | config file: |
+| | /testsuite/posca/testcase_cfg/... |
+| | posca_feature_moon_resources.yaml |
+| | |
++--------------+--------------------------------------------------------------+
+|test result | number of tenants, max number of users |
+| | |
++--------------+--------------------------------------------------------------+
+
+Configuration
+============
+::
+
+ load_manager:
+ scenarios:
+ tool: https request
+ # info that the cpus and memes have the same number of data.
+ pdp_name: pdp
+ policy_name: "MLS Policy example"
+ model_name: MLS
+ tenants: 1,5,10,20
+ subject_number: 10
+ object_number: 10
+ timeout: 0.2
+
+ runners:
+ stack_create: yardstick
+ Debug: False
+ yardstick_test_dir: "samples"
+ yardstick_testcase: "moon_resource"
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Tech and others.
+
+***********************************************
+POSCA feature Test of Moon Security for Tenants
+***********************************************
+
+Test Case
+=========
+
++-----------------------------------------------------------------------------+
+|Bottlenecks POSCA Soak Test Throughputs |
+| |
++--------------+--------------------------------------------------------------+
+|test case name| posca_feature_moon_tenants |
+| | |
++--------------+--------------------------------------------------------------+
+|description | Moon authentication capability test for maximum tenants |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | config file: |
+| | /testsuite/posca/testcase_cfg/... |
+| | posca_feature_moon_tenants.yaml |
+| | |
++--------------+--------------------------------------------------------------+
+|test result | Max number of tenants |
+| | |
++--------------+--------------------------------------------------------------+
+
+Configuration
+============
+::
+
+ load_manager:
+ scenarios:
+ tool: https request
+ # info that the cpus and memes have the same number of data.
+ pdp_name: pdp
+ policy_name: "MLS Policy example"
+ model_name: MLS
+ subject_number: 20
+ object_number: 20
+ timeout: 0.003
+ initial_tenants: 0
+ steps_tenants: 1
+ tolerate_time: 20
+ SLA: 5
+
+ runners:
+ stack_create: yardstick
+ Debug: False
+ yardstick_test_dir: "samples"
+ yardstick_testcase: "moon_tenant"
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Tech and others.
+
+***********************************
+POSCA feature Test of VNF Scale Out
+***********************************
+
+Test Case
+=========
+
++-----------------------------------------------------------------------------+
+|Bottlenecks POSCA Soak Test Throughputs |
+| |
++--------------+--------------------------------------------------------------+
+|test case name| posca_feature_nfv_scale_out |
+| | |
++--------------+--------------------------------------------------------------+
+|description | SampleVNF Scale Out Test |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | config file: |
+| | /testsuite/posca/testcase_cfg/... |
+| | posca_feature_nfv_scale_out.yaml |
+| | |
++--------------+--------------------------------------------------------------+
+|test result | throughputs, latency, loss rate |
+| | |
++--------------+--------------------------------------------------------------+
+
+Configuration
+============
+::
+
+ load_manager:
+ scenarios:
+ number_vnfs: 1, 2, 4
+ iterations: 10
+ interval: 35
+
+ runners:
+ stack_create: yardstick
+ flavor:
+ yardstick_test_dir: "samples/vnf_samples/nsut/acl"
+ yardstick_testcase: "tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_correlated_traffic_scale_out"
export OS_CACERT=/tmp/os_cacert
If you have deployed your openstack environment by compass,
-you could use the following command to get the required files. For Fuel, Apex and JOID installer, we only provide limited support now
+you could use the following command to get the required files. As to Fuel, Apex and JOID installers, we only provide limited support now
for retrieving the configuration/description files. If you find that the following command can not do the magic, you should put the
required files in /tmp manually.
bash ./utils/env_prepare/config_prepare.sh -i <installer> [--debug]
-Note that if we execute the command above, then admin_rc.sh and pod.yml gets created automatically in /tmp folder along with the line `export OS_CACERT=/tmp/os_cacert` added in admin_rc.sh file.
+Note that if we execute the command above, then admin_rc.sh and pod.yml will be created automatically in /tmp folder along with the line `export OS_CACERT=/tmp/os_cacert` added in admin_rc.sh file.
Executing Specified Testcase
./posca/posca_factor_system_bandwidth.rst
./posca/posca_factor_ping.rst
+ ./posca/posca_factor_storperf.rst
+ ./posca/posca_factor_multistack_storage.rst
+ ./posca/posca_factor_multistack_storage_parallel.rst
+ ./posca/posca_factor_soak_throughputs.rst
+ ./posca/posca_feature_moon_resources.rst
+ ./posca/posca_feature_moon_tenants.rst
+ ./posca/posca_feature_vnf_scale_out.rst
Code Review / bottlenecks.git / commitdiff