Add test case related to measure compute score and its energy consumption impact 24/69024/1
authorRoland Picard <roland.picard@orange.com>
Tue, 26 Nov 2019 16:21:08 +0000 (17:21 +0100)
committerRoland Picard <roland.picard@orange.com>
Tue, 26 Nov 2019 16:21:08 +0000 (17:21 +0100)
Test case opnfv_yardstick_tc015.yaml is added that runs a CPU test and monitor
energy consumtion impact synchronized with cpu load.
opnfv_yardstick_tc015.rst is added to provide documentation of the test case.
energy.yaml sample is added to show how to add energy testing in any test case.

JIRA: YARDSTICK-1624
Co-authored by: Sun Guannan <sungn2@lenovo.com>

Signed-off-by: Roland Picard <roland.picard@orange.com>
Change-Id: I2403c38bbf320f96d2ed1d9e262560388efc44c9

docs/testing/user/userguide/15-list-of-tcs.rst
docs/testing/user/userguide/opnfv_yardstick_tc015.rst [new file with mode: 0755]
samples/energy.yaml [new file with mode: 0755]
tests/opnfv/test_cases/opnfv_yardstick_tc015.yaml [new file with mode: 0755]

index 8990800..b727aa3 100644 (file)
-.. This work is licensed under a Creative Commons Attribution 4.0 International
-.. License.
-.. http://creativecommons.org/licenses/by/4.0
-.. (c) OPNFV, Ericsson AB and others.
-
-====================
-Yardstick Test Cases
-====================
-
-Abstract
-========
-
-This chapter lists available Yardstick test cases.
-Yardstick test cases are divided in two main categories:
-
-* *Generic NFVI Test Cases* - Test Cases developed to realize the methodology
-  described in :doc:`02-methodology`
-
-* *OPNFV Feature Test Cases* - Test Cases developed to verify one or more
-  aspect of a feature delivered by an OPNFV Project.
-
-Generic NFVI Test Case Descriptions
-===================================
-
-.. toctree::
-   :maxdepth: 1
-
-   opnfv_yardstick_tc001.rst
-   opnfv_yardstick_tc002.rst
-   opnfv_yardstick_tc004.rst
-   opnfv_yardstick_tc005.rst
-   opnfv_yardstick_tc006.rst
-   opnfv_yardstick_tc008.rst
-   opnfv_yardstick_tc009.rst
-   opnfv_yardstick_tc010.rst
-   opnfv_yardstick_tc011.rst
-   opnfv_yardstick_tc012.rst
-   opnfv_yardstick_tc014.rst
-   opnfv_yardstick_tc024.rst
-   opnfv_yardstick_tc037.rst
-   opnfv_yardstick_tc038.rst
-   opnfv_yardstick_tc042.rst
-   opnfv_yardstick_tc043.rst
-   opnfv_yardstick_tc044.rst
-   opnfv_yardstick_tc055.rst
-   opnfv_yardstick_tc061.rst
-   opnfv_yardstick_tc063.rst
-   opnfv_yardstick_tc069.rst
-   opnfv_yardstick_tc070.rst
-   opnfv_yardstick_tc071.rst
-   opnfv_yardstick_tc072.rst
-   opnfv_yardstick_tc073.rst
-   opnfv_yardstick_tc074.rst
-   opnfv_yardstick_tc075.rst
-   opnfv_yardstick_tc076.rst
-   opnfv_yardstick_tc078.rst
-   opnfv_yardstick_tc079.rst
-   opnfv_yardstick_tc080.rst
-   opnfv_yardstick_tc081.rst
-   opnfv_yardstick_tc083.rst
-   opnfv_yardstick_tc084.rst
-
-OPNFV Feature Test Cases
-========================
-
-H A
----
-
-.. toctree::
-   :maxdepth: 1
-
-   opnfv_yardstick_tc019.rst
-   opnfv_yardstick_tc025.rst
-   opnfv_yardstick_tc045.rst
-   opnfv_yardstick_tc046.rst
-   opnfv_yardstick_tc047.rst
-   opnfv_yardstick_tc048.rst
-   opnfv_yardstick_tc049.rst
-   opnfv_yardstick_tc050.rst
-   opnfv_yardstick_tc051.rst
-   opnfv_yardstick_tc052.rst
-   opnfv_yardstick_tc053.rst
-   opnfv_yardstick_tc054.rst
-   opnfv_yardstick_tc056.rst
-   opnfv_yardstick_tc057.rst
-   opnfv_yardstick_tc058.rst
-   opnfv_yardstick_tc087.rst
-   opnfv_yardstick_tc088.rst
-   opnfv_yardstick_tc089.rst
-   opnfv_yardstick_tc090.rst
-   opnfv_yardstick_tc091.rst
-   opnfv_yardstick_tc092.rst
-   opnfv_yardstick_tc093.rst
-
-IPv6
-----
-
-.. toctree::
-   :maxdepth: 1
-
-   opnfv_yardstick_tc027.rst
-
-KVM
----
-
-.. toctree::
-   :maxdepth: 1
-
-   opnfv_yardstick_tc028.rst
-
-Parser
-------
-
-.. toctree::
-   :maxdepth: 1
-
-   opnfv_yardstick_tc040.rst
-
-StorPerf
---------
-
-.. toctree::
-   :maxdepth: 1
-
-   opnfv_yardstick_tc074.rst
-
-Templates
-=========
-
-.. toctree::
-   :maxdepth: 1
-
-   testcase_description_v2_template
-   Yardstick_task_templates
-
+.. This work is licensed under a Creative Commons Attribution 4.0 International\r
+.. License.\r
+.. http://creativecommons.org/licenses/by/4.0\r
+.. (c) OPNFV, Ericsson AB and others.\r
+\r
+====================\r
+Yardstick Test Cases\r
+====================\r
+\r
+Abstract\r
+========\r
+\r
+This chapter lists available Yardstick test cases.\r
+Yardstick test cases are divided in two main categories:\r
+\r
+* *Generic NFVI Test Cases* - Test Cases developed to realize the methodology\r
+  described in :doc:`02-methodology`\r
+\r
+* *OPNFV Feature Test Cases* - Test Cases developed to verify one or more\r
+  aspect of a feature delivered by an OPNFV Project.\r
+\r
+Generic NFVI Test Case Descriptions\r
+===================================\r
+\r
+.. toctree::\r
+   :maxdepth: 1\r
+\r
+   opnfv_yardstick_tc001.rst\r
+   opnfv_yardstick_tc002.rst\r
+   opnfv_yardstick_tc004.rst\r
+   opnfv_yardstick_tc005.rst\r
+   opnfv_yardstick_tc006.rst\r
+   opnfv_yardstick_tc008.rst\r
+   opnfv_yardstick_tc009.rst\r
+   opnfv_yardstick_tc010.rst\r
+   opnfv_yardstick_tc011.rst\r
+   opnfv_yardstick_tc012.rst\r
+   opnfv_yardstick_tc014.rst\r
+   opnfv_yardstick_tc015.rst\r
+   opnfv_yardstick_tc024.rst\r
+   opnfv_yardstick_tc037.rst\r
+   opnfv_yardstick_tc038.rst\r
+   opnfv_yardstick_tc042.rst\r
+   opnfv_yardstick_tc043.rst\r
+   opnfv_yardstick_tc044.rst\r
+   opnfv_yardstick_tc055.rst\r
+   opnfv_yardstick_tc061.rst\r
+   opnfv_yardstick_tc063.rst\r
+   opnfv_yardstick_tc069.rst\r
+   opnfv_yardstick_tc070.rst\r
+   opnfv_yardstick_tc071.rst\r
+   opnfv_yardstick_tc072.rst\r
+   opnfv_yardstick_tc073.rst\r
+   opnfv_yardstick_tc074.rst\r
+   opnfv_yardstick_tc075.rst\r
+   opnfv_yardstick_tc076.rst\r
+   opnfv_yardstick_tc078.rst\r
+   opnfv_yardstick_tc079.rst\r
+   opnfv_yardstick_tc080.rst\r
+   opnfv_yardstick_tc081.rst\r
+   opnfv_yardstick_tc083.rst\r
+   opnfv_yardstick_tc084.rst\r
+\r
+OPNFV Feature Test Cases\r
+========================\r
+\r
+H A\r
+---\r
+\r
+.. toctree::\r
+   :maxdepth: 1\r
+\r
+   opnfv_yardstick_tc019.rst\r
+   opnfv_yardstick_tc025.rst\r
+   opnfv_yardstick_tc045.rst\r
+   opnfv_yardstick_tc046.rst\r
+   opnfv_yardstick_tc047.rst\r
+   opnfv_yardstick_tc048.rst\r
+   opnfv_yardstick_tc049.rst\r
+   opnfv_yardstick_tc050.rst\r
+   opnfv_yardstick_tc051.rst\r
+   opnfv_yardstick_tc052.rst\r
+   opnfv_yardstick_tc053.rst\r
+   opnfv_yardstick_tc054.rst\r
+   opnfv_yardstick_tc056.rst\r
+   opnfv_yardstick_tc057.rst\r
+   opnfv_yardstick_tc058.rst\r
+   opnfv_yardstick_tc087.rst\r
+   opnfv_yardstick_tc088.rst\r
+   opnfv_yardstick_tc089.rst\r
+   opnfv_yardstick_tc090.rst\r
+   opnfv_yardstick_tc091.rst\r
+   opnfv_yardstick_tc092.rst\r
+   opnfv_yardstick_tc093.rst\r
+\r
+IPv6\r
+----\r
+\r
+.. toctree::\r
+   :maxdepth: 1\r
+\r
+   opnfv_yardstick_tc027.rst\r
+\r
+KVM\r
+---\r
+\r
+.. toctree::\r
+   :maxdepth: 1\r
+\r
+   opnfv_yardstick_tc028.rst\r
+\r
+Parser\r
+------\r
+\r
+.. toctree::\r
+   :maxdepth: 1\r
+\r
+   opnfv_yardstick_tc040.rst\r
+\r
+StorPerf\r
+--------\r
+\r
+.. toctree::\r
+   :maxdepth: 1\r
+\r
+   opnfv_yardstick_tc074.rst\r
+\r
+Templates\r
+=========\r
+\r
+.. toctree::\r
+   :maxdepth: 1\r
+\r
+   testcase_description_v2_template\r
+   Yardstick_task_templates\r
+\r
diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc015.rst b/docs/testing/user/userguide/opnfv_yardstick_tc015.rst
new file mode 100755 (executable)
index 0000000..277614a
--- /dev/null
@@ -0,0 +1,141 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International\r
+.. License.\r
+.. http://creativecommons.org/licenses/by/4.0\r
+.. (c) OPNFV, Orange and others.\r
+\r
+*************************************\r
+Yardstick Test Case Description TC015\r
+*************************************\r
+\r
+.. _unixbench: https://github.com/kdlucas/byte-unixbench/blob/master/UnixBench\r
+\r
++-----------------------------------------------------------------------------+\r
+| Processing speed with impact on energy consumption and CPU load             |\r
+|                                                                             |\r
++--------------+--------------------------------------------------------------+\r
+|test case id  | OPNFV_YARDSTICK_TC015_PROCESSING SPEED                       |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|metric        | score of single cpu running,                                 |\r
+|              | score of parallel running,                                   |\r
+|              | energy consumption                                           |\r
+|              | cpu load                                                     |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|test purpose  | The purpose of TC015 is to evaluate the IaaS compute         |\r
+|              | performance with regards to CPU processing speed with        |\r
+|              | its impact on the energy consumption                         |\r
+|              | It measures score of single cpu running and parallel         |\r
+|              | running. Energy consumption and cpu load are monitored while |\r
+|              | the cpu test is running.                                     |\r
+|              |                                                              |\r
+|              | The purpose is also to be able to spot the trends.           |\r
+|              | Test results, graphs and similar shall be stored for         |\r
+|              | comparison reasons and product evolution understanding       |\r
+|              | between different OPNFV versions and/or configurations,      |\r
+|              | different server types.                                      |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|test tool     | UnixBench                                                    |\r
+|              |                                                              |\r
+|              | Unixbench is the most used CPU benchmarking software tool.   |\r
+|              | It can measure the performance of bash scripts, CPUs in      |\r
+|              | multithreading and single threading. It can also measure the |\r
+|              | performance for parallel tasks. Also, specific disk IO for   |\r
+|              | small and large files are performed. You can use it to       |\r
+|              | measure either linux dedicated servers and linux vps         |\r
+|              | servers, running CentOS, Debian, Ubuntu, Fedora and other    |\r
+|              | distros.                                                     |\r
+|              |                                                              |\r
+|              | (UnixBench is not always part of a Linux distribution, hence |\r
+|              | it needs to be installed. As an example see the              |\r
+|              | /yardstick/tools/ directory for how to generate a Linux      |\r
+|              | image with UnixBench included.)                              |\r
+|              |                                                              |\r
+|              | Redfish API                                                  |\r
+|              | This HTTPS interface is provided by BMC of every telco grade |\r
+|              | server. Is is a standard interface.                          |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|test          | The UnixBench runs system benchmarks on a compute, getting   |\r
+|description   | information on the CPUs in the system. If the system has     |\r
+|              | more than one CPU, the tests will be run twice -- once with  |\r
+|              | a single copy of each test running at once, and once with N  |\r
+|              | N copies, where N is the number of CPUs.                     |\r
+|              |                                                              |\r
+|              | UnixBench will process a set of results from a single test   |\r
+|              | by averaging the individual pass results into a single final |\r
+|              | value.                                                       |\r
+|              |                                                              |\r
+|              | While the cpu test is running Energy scenario run in         |\r
+|              | background to monitor the number of watt consumed by the     |\r
+|              | compute server on the fly. The same is done using Cpuload    |\r
+|              | scenario to monitor the overall percentage of CPU used on    |\r
+|              | the fly. This enables to balance the CPU score with its      |\r
+|              | impact on energy consumption. Synchronized measurements      |\r
+|              | enables to look at any relation between CPU load and energy  |\r
+|              | consumption.                                                 |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|configuration | file: opnfv_yardstick_tc015.yaml                             |\r
+|              |                                                              |\r
+|              | run_mode:                                                    |\r
+|              |    Run Energy and Cpuload in background                      |\r
+|              |    Run unixbench in quiet mode or verbose mode               |\r
+|              |    test_type: dhry2reg, whetstone and so on                  |\r
+|              |                                                              |\r
+|              | Duration and Interval are set globally for Energy and        |\r
+|              | Cpuload, aligned with duration of UnixBench test.            |\r
+|              | SLA can be set for each scenario type. Default is NA.        |\r
+|              | For SLA with single_score and parallel_score, both can be    |\r
+|              | set by user, default is NA.                                  |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|applicability | Test shall be applied to node context only                   |\r
+|              | It can be configured with different:                         |\r
+|              |                                                              |\r
+|              |  * test types: dhry2reg, whetstone                           |\r
+|              |                                                              |\r
+|              | Default values exist.                                        |\r
+|              |                                                              |\r
+|              | SLA (optional) : min_score: The minimun UnixBench score that |\r
+|              | is accepted.                                                 |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|usability     | This test case is one of Yardstick's generic test. Thus it   |\r
+|              | is runnable on most of the scenarios.                        |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|references    | unixbench_                                                   |\r
+|              |                                                              |\r
+|              | ETSI-NFV-TST001                                              |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|pre-test      | The target shall have unixbench installed on it.             |\r
+|conditions    |                                                              |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|test sequence | description and expected result                              |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|step 1        | Yardstick is connected with the target node using ssh.       |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|step 2        | Energy and Cpuload are launched silently in background one   |\r
+|              | after the other.                                             |\r
+|              | Then UnixBench is invoked. All the tests are executed using  |\r
+|              | the "Run" script in the top-level of UnixBench directory.    |\r
+|              | The "Run" script will run a standard "index" test, and save  |\r
+|              | the report in the "results" directory. Then the report is    |\r
+|              | processed by "unixbench_benchmark" and checked against the   |\r
+|              | SLA.                                                         |\r
+|              | While unibench runs energy and cpu load are catched          |\r
+|              | periodically according to interval value.                    |\r
+|              |                                                              |\r
+|              | Result: Logs are stored.                                     |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
+|test verdict  | Fails only if SLA is not passed, or if there is a test case  |\r
+|              | execution problem.                                           |\r
+|              |                                                              |\r
++--------------+--------------------------------------------------------------+\r
diff --git a/samples/energy.yaml b/samples/energy.yaml
new file mode 100755 (executable)
index 0000000..bc8f822
--- /dev/null
@@ -0,0 +1,53 @@
+##############################################################################\r
+# Copyright (c) 2019 Orange and others.\r
+#\r
+# All rights reserved. This program and the accompanying materials\r
+# are made available under the terms of the Apache License, Version 2.0\r
+# which accompanies this distribution, and is available at\r
+# http://www.apache.org/licenses/LICENSE-2.0\r
+##############################################################################\r
+---\r
+# Sample benchmark task config file\r
+# Monitor energy consumption during a cpu test\r
+# Should be used inside test cases where UnixBench is replaced by other\r
+# scenario type. This enables to easily consider energy consumption as a\r
+# mandatory metric for any performance test\r
+\r
+schema: "yardstick:task:0.1"\r
+\r
+{% set target = target or "node5"%}\r
+{% set file = file or '/etc/yardstick/pod.yaml' %}\r
+{% set duree = 60 %}\r
+{% set intervalle = 5 %}\r
+scenarios:\r
+\r
+-\r
+  type: Energy\r
+  run_in_background: true\r
+  target: {{target}}.LF\r
+\r
+  runner:\r
+    type: Duration\r
+    duration: {{duree}}\r
+    interval: {{intervalle}}\r
+\r
+  sla:\r
+    action: monitor\r
+\r
+-\r
+  type: UnixBench\r
+  options:\r
+    run_mode: 'verbose'\r
+    test_type: 'dhry2reg'\r
+  host: {{target}}.LF\r
+\r
+  runner:\r
+    type: Iteration\r
+    iterations: 1\r
+    interval: 1\r
+\r
+\r
+context:\r
+  type: Node\r
+  name: LF\r
+  file: {{file}}\r
diff --git a/tests/opnfv/test_cases/opnfv_yardstick_tc015.yaml b/tests/opnfv/test_cases/opnfv_yardstick_tc015.yaml
new file mode 100755 (executable)
index 0000000..f35c833
--- /dev/null
@@ -0,0 +1,60 @@
+##############################################################################\r
+# Copyright (c) 2019 Orange and others.\r
+#\r
+# All rights reserved. This program and the accompanying materials\r
+# are made available under the terms of the Apache License, Version 2.0\r
+# which accompanies this distribution, and is available at\r
+# http://www.apache.org/licenses/LICENSE-2.0\r
+##############################################################################\r
+---\r
+\r
+schema: "yardstick:task:0.1"\r
+description: >\r
+    Yardstick TC015 config file;\r
+    Measure Processing speed using unixbench with its impact on energy consumption and CPU load.\r
+\r
+{% set target = target or "node5"%}\r
+{% set file = file or '/etc/yardstick/pod.yaml' %}\r
+{% set duree = 60 %}\r
+{% set intervalle = 5 %}\r
+scenarios:\r
+\r
+-\r
+  type: Energy\r
+  run_in_background: true\r
+  target: {{target}}.LF\r
+\r
+  runner:\r
+    type: Duration\r
+    duration: {{duree}}\r
+    interval: {{intervalle}}\r
+\r
+-\r
+  type: CPUload\r
+  run_in_background: true\r
+  options:\r
+    interval: 1\r
+    count: 1\r
+  host: {{target}}.LF\r
+  runner:\r
+    type: Duration\r
+    duration: {{duree}}\r
+    interval: {{intervalle}}\r
+\r
+-\r
+  type: UnixBench\r
+  options:\r
+    run_mode: 'verbose'\r
+    test_type: 'dhry2reg'\r
+  host: {{target}}.LF\r
+\r
+  runner:\r
+    type: Iteration\r
+    iterations: 1\r
+    interval: 1\r
+\r
+\r
+context:\r
+  type: Node\r
+  name: LF\r
+  file: {{file}}\r