docs/testing/user/userguide/opnfv_yardstick_tc010.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International
   2 .. License.
   3 .. http://creativecommons.org/licenses/by/4.0
   4 .. (c) OPNFV, Ericsson AB and others.
   5
   6 *************************************
   7 Yardstick Test Case Description TC010
   8 *************************************
   9
  10 .. _lat_mem_rd: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html
  11
  12 +-----------------------------------------------------------------------------+
  13 |Memory Latency                                                               |
  14 |                                                                             |
  15 +--------------+--------------------------------------------------------------+
  16 |test case id  | OPNFV_YARDSTICK_TC010_MEMORY LATENCY                         |
  17 |              |                                                              |
  18 +--------------+--------------------------------------------------------------+
  19 |metric        | Memory read latency (nanoseconds)                            |
  20 |              |                                                              |
  21 +--------------+--------------------------------------------------------------+
  22 |test purpose  | The purpose of TC010 is to evaluate the IaaS compute         |
  23 |              | performance with regards to memory read latency.             |
  24 |              | It measures the memory read latency for varying memory sizes |
  25 |              | and strides. Whole memory hierarchy is measured.             |
  26 |              |                                                              |
  27 |              | The purpose is also to be able to spot the trends.           |
  28 |              | Test results, graphs and similar shall be stored for         |
  29 |              | comparison reasons and product evolution understanding       |
  30 |              | between different OPNFV versions and/or configurations.      |
  31 |              |                                                              |
  32 +--------------+--------------------------------------------------------------+
  33 |test tool     | Lmbench                                                      |
  34 |              |                                                              |
  35 |              | Lmbench is a suite of operating system microbenchmarks. This |
  36 |              | test uses lat_mem_rd tool from that suite including:         |
  37 |              |  * Context switching                                         |
  38 |              |  * Networking: connection establishment, pipe, TCP, UDP, and |
  39 |              |    RPC hot potato                                            |
  40 |              |  * File system creates and deletes                           |
  41 |              |  * Process creation                                          |
  42 |              |  * Signal handling                                           |
  43 |              |  * System call overhead                                      |
  44 |              |  * Memory read latency                                       |
  45 |              |                                                              |
  46 |              | (LMbench is not always part of a Linux distribution, hence   |
  47 |              | it needs to be installed. As an example see the              |
  48 |              | /yardstick/tools/ directory for how to generate a Linux      |
  49 |              | image with LMbench included.)                                |
  50 |              |                                                              |
  51 +--------------+--------------------------------------------------------------+
  52 |test          | LMbench lat_mem_rd benchmark measures memory read latency    |
  53 |description   | for varying memory sizes and strides.                        |
  54 |              |                                                              |
  55 |              | The benchmark runs as two nested loops. The outer loop is    |
  56 |              | the stride size. The inner loop is the array size. For each  |
  57 |              | array size, the benchmark creates a ring of pointers that    |
  58 |              | point backward one stride.Traversing the array is done by:   |
  59 |              |                                                              |
  60 |              |         p = (char **)*p;                                     |
  61 |              |                                                              |
  62 |              | in a for loop (the over head of the for loop is not          |
  63 |              | significant; the loop is an unrolled loop 100 loads long).   |
  64 |              | The size of the array varies from 512 bytes to (typically)   |
  65 |              | eight megabytes. For the small sizes, the cache will have an |
  66 |              | effect, and the loads will be much faster. This becomes much |
  67 |              | more apparent when the data is plotted.                      |
  68 |              |                                                              |
  69 |              | Only data accesses are measured; the instruction cache is    |
  70 |              | not measured.                                                |
  71 |              |                                                              |
  72 |              | The results are reported in nanoseconds per load and have    |
  73 |              | been verified accurate to within a few nanoseconds on an SGI |
  74 |              | Indy.                                                        |
  75 |              |                                                              |
  76 +--------------+--------------------------------------------------------------+
  77 |configuration | File: opnfv_yardstick_tc010.yaml                             |
  78 |              |                                                              |
  79 |              | * SLA (max_latency): 30 nanoseconds                          |
  80 |              | * Stride - 128 bytes                                         |
  81 |              | * Stop size - 64 megabytes                                   |
  82 |              | * Iterations: 10 - test is run 10 times iteratively.         |
  83 |              | * Interval: 1 - there is 1 second delay between each         |
  84 |              |   iteration.                                                 |
  85 |              |                                                              |
  86 |              | SLA is optional. The SLA in this test case serves as an      |
  87 |              | example. Considerably lower read latency is expected.        |
  88 |              | However, to cover most configurations, both baremetal and    |
  89 |              | fully virtualized  ones, this value should be possible to    |
  90 |              | achieve and acceptable for black box testing.                |
  91 |              | Many heavy IO applications start to suffer badly if the      |
  92 |              | read latency is higher than this.                            |
  93 |              |                                                              |
  94 +--------------+--------------------------------------------------------------+
  95 |applicability | Test can be configured with different:                       |
  96 |              |                                                              |
  97 |              | * strides;                                                   |
  98 |              | * stop_size;                                                 |
  99 |              | * iterations and intervals.                                  |
 100 |              |                                                              |
 101 |              | Default values exist.                                        |
 102 |              |                                                              |
 103 |              | SLA (optional) : max_latency: The maximum memory latency     |
 104 |              | that is accepted.                                            |
 105 |              |                                                              |
 106 +--------------+--------------------------------------------------------------+
 107 |usability     | This test case is one of Yardstick's generic test. Thus it   |
 108 |              | is runnable on most of the scenarios.                        |
 109 |              |                                                              |
 110 +--------------+--------------------------------------------------------------+
 111 |references    | LMbench lat_mem_rd_                                          |
 112 |              |                                                              |
 113 |              | ETSI-NFV-TST001                                              |
 114 |              |                                                              |
 115 +--------------+--------------------------------------------------------------+
 116 |pre-test      | The test case image needs to be installed into Glance        |
 117 |conditions    | with Lmbench included in the image.                          |
 118 |              |                                                              |
 119 |              | No POD specific requirements have been identified.           |
 120 |              |                                                              |
 121 +--------------+--------------------------------------------------------------+
 122 |test sequence | description and expected result                              |
 123 |              |                                                              |
 124 +--------------+--------------------------------------------------------------+
 125 |step 1        | The host is installed as client. LMbench's lat_mem_rd tool   |
 126 |              | is invoked and logs are produced and stored.                 |
 127 |              |                                                              |
 128 |              | Result: logs are stored.                                     |
 129 |              |                                                              |
 130 +--------------+--------------------------------------------------------------+
 131 |step 1        | A host VM with LMbench installed is booted.                  |
 132 |              |                                                              |
 133 +--------------+--------------------------------------------------------------+
 134 |step 2        | Yardstick is connected with the host VM by using ssh.        |
 135 |              | 'lmbench_latency_benchmark' bash script is copyied from Jump |
 136 |              | Host to the host VM via the ssh tunnel.                      |
 137 |              |                                                              |
 138 +--------------+--------------------------------------------------------------+
 139 |step 3        | 'lmbench_latency_benchmark' script is invoked. LMbench's     |
 140 |              | lat_mem_rd benchmark starts to measures memory read latency  |
 141 |              | for varying memory sizes and strides. Memory read latency    |
 142 |              | are recorded and checked against the SLA. Logs are produced  |
 143 |              | and stored.                                                  |
 144 |              |                                                              |
 145 |              | Result: Logs are stored.                                     |
 146 |              |                                                              |
 147 +--------------+--------------------------------------------------------------+
 148 |step 4        | The host VM is deleted.                                      |
 149 |              |                                                              |
 150 +--------------+--------------------------------------------------------------+
 151 |test verdict  | Test fails if the measured memory latency is above the SLA   |
 152 |              | value or if there is a test case execution problem.          |
 153 |              |                                                              |
 154 +--------------+--------------------------------------------------------------+