docs/testing/user/userguide/opnfv_yardstick_tc012.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International
   2 .. License.
   3 .. http://creativecommons.org/licenses/by/4.0
   4 .. (c) OPNFV, Ericsson AB and others.
   5
   6 *************************************
   7 Yardstick Test Case Description TC012
   8 *************************************
   9
  10 .. _bw_mem: http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html
  11
  12 +-----------------------------------------------------------------------------+
  13 |Memory Bandwidth                                                             |
  14 |                                                                             |
  15 +--------------+--------------------------------------------------------------+
  16 |test case id  | OPNFV_YARDSTICK_TC012_MEMORY BANDWIDTH                       |
  17 |              |                                                              |
  18 +--------------+--------------------------------------------------------------+
  19 |metric        | Memory read/write bandwidth (MBps)                           |
  20 |              |                                                              |
  21 +--------------+--------------------------------------------------------------+
  22 |test purpose  | The purpose of TC012 is to evaluate the IaaS compute         |
  23 |              | performance with regards to memory throughput.               |
  24 |              | It measures the rate at which data can be read from and      |
  25 |              | written to the memory (this includes all levels of memory).  |
  26 |              |                                                              |
  27 |              | The purpose is also to be able to spot the trends.           |
  28 |              | Test results, graphs and similar shall be stored for         |
  29 |              | comparison reasons and product evolution understanding       |
  30 |              | between different OPNFV versions and/or configurations.      |
  31 |              |                                                              |
  32 +--------------+--------------------------------------------------------------+
  33 |test tool     | LMbench                                                      |
  34 |              |                                                              |
  35 |              | LMbench is a suite of operating system microbenchmarks.      |
  36 |              | This test uses bw_mem tool from that suite including:        |
  37 |              |  * Cached file read                                          |
  38 |              |  * Memory copy (bcopy)                                       |
  39 |              |  * Memory read                                               |
  40 |              |  * Memory write                                              |
  41 |              |  * Pipe                                                      |
  42 |              |  * TCP                                                       |
  43 |              |                                                              |
  44 |              | (LMbench is not always part of a Linux distribution, hence   |
  45 |              | it needs to be installed. As an example see the              |
  46 |              | /yardstick/tools/ directory for how to generate a Linux      |
  47 |              | image with LMbench included.)                                |
  48 |              |                                                              |
  49 +--------------+--------------------------------------------------------------+
  50 |test          | LMbench bw_mem benchmark allocates twice the specified       |
  51 |description   | amount of memory, zeros it, and then times the copying of    |
  52 |              | the first half to the second half. The benchmark is invoked  |
  53 |              | in a host VM on a compute blade. Results are reported in     |
  54 |              | megabytes moved per second.                                  |
  55 |              |                                                              |
  56 +--------------+--------------------------------------------------------------+
  57 |configuration | File: opnfv_yardstick_tc012.yaml                             |
  58 |              |                                                              |
  59 |              | * SLA (optional): 15000 (MBps) min_bw: The minimum amount of |
  60 |              |   memory bandwidth that is accepted.                         |
  61 |              | * Size: 10 240 kB - test allocates twice that size           |
  62 |              |   (20 480kB) zeros it and then measures the time it takes to |
  63 |              |   copy from one side to another.                             |
  64 |              | * Benchmark: rdwr - measures the time to read data into      |
  65 |              |   memory and then write data to the same location.           |
  66 |              | * Warmup: 0 - the number of iterations to perform before     |
  67 |              |   taking actual measurements.                                |
  68 |              | * Iterations: 10 - test is run 10 times iteratively.         |
  69 |              | * Interval: 1 - there is 1 second delay between each         |
  70 |              |   iteration.                                                 |
  71 |              |                                                              |
  72 |              | SLA is optional. The SLA in this test case serves as an      |
  73 |              | example. Considerably higher bandwidth is expected.          |
  74 |              | However, to cover most configurations, both baremetal and    |
  75 |              | fully virtualized  ones, this value should be possible to    |
  76 |              | achieve and acceptable for black box testing.                |
  77 |              | Many heavy IO applications start to suffer badly if the      |
  78 |              | read/write bandwidths are lower than this.                   |
  79 |              |                                                              |
  80 +--------------+--------------------------------------------------------------+
  81 |applicability | Test can be configured with different:                       |
  82 |              |                                                              |
  83 |              |  * memory sizes;                                             |
  84 |              |  * memory operations (such as rd, wr, rdwr, cp, frd, fwr,    |
  85 |              |    fcp, bzero, bcopy);                                       |
  86 |              |  * number of warmup iterations;                              |
  87 |              |  * iterations and intervals.                                 |
  88 |              |                                                              |
  89 |              | Default values exist.                                        |
  90 |              |                                                              |
  91 |              | SLA (optional) : min_bandwidth: The minimun memory bandwidth |
  92 |              | that is accepted.                                            |
  93 |              |                                                              |
  94 +--------------+--------------------------------------------------------------+
  95 |usability     | This test case is one of Yardstick's generic test. Thus it   |
  96 |              | is runnable on most of the scenarios.                        |
  97 |              |                                                              |
  98 +--------------+--------------------------------------------------------------+
  99 |references    | LMbench bw_mem_                                              |
 100 |              |                                                              |
 101 |              | ETSI-NFV-TST001                                              |
 102 |              |                                                              |
 103 +--------------+--------------------------------------------------------------+
 104 |pre-test      | The test case image needs to be installed into Glance        |
 105 |conditions    | with Lmbench included in the image.                          |
 106 |              |                                                              |
 107 |              | No POD specific requirements have been identified.           |
 108 |              |                                                              |
 109 +--------------+--------------------------------------------------------------+
 110 |test sequence | description and expected result                              |
 111 |              |                                                              |
 112 +--------------+--------------------------------------------------------------+
 113 |step 1        | A host VM with LMbench installed is booted.                  |
 114 |              |                                                              |
 115 +--------------+--------------------------------------------------------------+
 116 |step 2        | Yardstick is connected with the host VM by using ssh.        |
 117 |              | "lmbench_bandwidth_benchmark" bash script is copied from     |
 118 |              | Jump Host to the host VM via ssh tunnel.                     |
 119 |              |                                                              |
 120 +--------------+--------------------------------------------------------------+
 121 |step 3        | 'lmbench_bandwidth_benchmark' script is invoked. LMbench's   |
 122 |              | bw_mem benchmark starts to measures memory read/write        |
 123 |              | bandwidth. Memory read/write bandwidth results are recorded  |
 124 |              | and checked against the SLA. Logs are produced and stored.   |
 125 |              |                                                              |
 126 |              | Result: Logs are stored.                                     |
 127 |              |                                                              |
 128 +--------------+--------------------------------------------------------------+
 129 |step 4        | The host VM is deleted.                                      |
 130 |              |                                                              |
 131 +--------------+--------------------------------------------------------------+
 132 |test verdict  | Test fails if the measured memory bandwidth is below the SLA |
 133 |              | value or if there is a test case execution problem.          |
 134 |              |                                                              |
 135 +--------------+--------------------------------------------------------------+