docs/testing/user/userguide/opnfv_yardstick_tc084.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International
   2 .. License.
   3 .. http://creativecommons.org/licenses/by/4.0
   4 .. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
   5
   6 *************************************
   7 Yardstick Test Case Description TC084
   8 *************************************
   9
  10 .. _spec_cpu_2006: https://www.spec.org/cpu2006/
  11
  12 +-----------------------------------------------------------------------------+
  13 |Compute Performance                                                          |
  14 |                                                                             |
  15 +--------------+--------------------------------------------------------------+
  16 |test case id  | OPNFV_YARDSTICK_TC084_SPEC CPU 2006 FOR VM                   |
  17 |              |                                                              |
  18 +--------------+--------------------------------------------------------------+
  19 |metric        | compute-intensive performance                                |
  20 |              |                                                              |
  21 +--------------+--------------------------------------------------------------+
  22 |test purpose  | The purpose of TC084 is to evaluate the IaaS compute         |
  23 |              | performance by using SPEC CPU 2006 benchmark. The SPEC CPU   |
  24 |              | 2006 benchmark has several different ways to measure         |
  25 |              | computer performance. One way is to measure how fast the     |
  26 |              | computer completes a single task; this is called a speed     |
  27 |              | measurement. Another way is to measure how many tasks        |
  28 |              | computer can accomplish in a certain amount of time; this is |
  29 |              | called a throughput, capacity or rate measurement.           |
  30 |              |                                                              |
  31 +--------------+--------------------------------------------------------------+
  32 |test tool     | SPEC CPU 2006                                                |
  33 |              |                                                              |
  34 |              | The SPEC CPU 2006 benchmark is SPEC's industry-standardized, |
  35 |              | CPU-intensive benchmark suite, stressing a system's          |
  36 |              | processor, memory subsystem and compiler. This benchmark     |
  37 |              | suite includes the SPECint benchmarks and the SPECfp         |
  38 |              | benchmarks. The SPECint 2006 benchmark contains 12 different |
  39 |              | benchmark tests and the SPECfp 2006 benchmark contains 19    |
  40 |              | different benchmark tests.                                   |
  41 |              |                                                              |
  42 |              | SPEC CPU 2006 is not always part of a Linux distribution.    |
  43 |              | SPEC requires that users purchase a license and agree with   |
  44 |              | their terms and conditions. For this test case, users must   |
  45 |              | manually download cpu2006-1.2.iso from the SPEC website and  |
  46 |              | save it under the yardstick/resources folder (e.g. /home/    |
  47 |              | opnfv/repos/yardstick/yardstick/resources/cpu2006-1.2.iso)   |
  48 |              | SPEC CPU® 2006 benchmark is available for purchase via the   |
  49 |              | SPEC order form (https://www.spec.org/order.html).           |
  50 |              |                                                              |
  51 +--------------+--------------------------------------------------------------+
  52 |test          | This test case uses SPEC CPU 2006 benchmark to measure       |
  53 |description   | compute-intensive performance of VMs.                        |
  54 |              |                                                              |
  55 +--------------+--------------------------------------------------------------+
  56 |configuration | file: opnfv_yardstick_tc084.yaml                             |
  57 |              |                                                              |
  58 |              | benchmark_subset is set to int.                              |
  59 |              |                                                              |
  60 |              | SLA is not available in this test case.                      |
  61 |              |                                                              |
  62 +--------------+--------------------------------------------------------------+
  63 |applicability | Test can be configured with different:                       |
  64 |              |                                                              |
  65 |              |  * benchmark_subset - a subset of SPEC CPU 2006 benchmarks   |
  66 |              |    to run;                                                   |
  67 |              |  * SPECint_benchmark - a SPECint benchmark to run;           |
  68 |              |  * SPECint_benchmark - a SPECfp benchmark to run;            |
  69 |              |  * output_format - desired report format;                    |
  70 |              |  * runspec_config - SPEC CPU 2006 config file provided to    |
  71 |              |    the runspec binary;                                       |
  72 |              |  * runspec_iterations - the number of benchmark iterations   |
  73 |              |    to execute. For a reportable run, must be 3;              |
  74 |              |  * runspec_tune - tuning to use (base, peak, or all). For a  |
  75 |              |    reportable run, must be either base or all. Reportable    |
  76 |              |    runs do base first, then (optionally) peak;               |
  77 |              |  * runspec_size - size of input data to run (test, train, or |
  78 |              |    ref). Reportable runs ensure that your binaries can       |
  79 |              |    produce correct results with the test and train workloads |
  80 |              |                                                              |
  81 +--------------+--------------------------------------------------------------+
  82 |usability     | This test case is used for executing SPEC CPU 2006 benchmark |
  83 |              | on virtual machines. The SPECint 2006 benchmark takes        |
  84 |              | approximately 5 hours. (The time may vary due to different   |
  85 |              | VM cpu configurations)                                       |
  86 |              |                                                              |
  87 +--------------+--------------------------------------------------------------+
  88 |references    | spec_cpu_2006_                                               |
  89 |              |                                                              |
  90 |              | ETSI-NFV-TST001                                              |
  91 |              |                                                              |
  92 +--------------+--------------------------------------------------------------+
  93 |pre-test      | To run and install SPEC CPU 2006, the following are          |
  94 |conditions    | required:                                                    |
  95 |              |  * For SPECint 2006: Both C99 and C++98 compilers are        |
  96 |              |    installed in VM images;                                   |
  97 |              |  * For SPECfp 2006: All three of C99, C++98 and Fortran-95   |
  98 |              |    compilers installed in VM images;                         |
  99 |              |  * At least 4GB of disk space availabile on VM.              |
 100 |              |                                                              |
 101 |              |  gcc 4.8.* and g++ 4.8.* version have been tested in Ubuntu  |
 102 |              |  14.04, Ubuntu 16.04 and Redhat Enterprise Linux 7.4 image.  |
 103 |              |  Higher gcc and g++ version may cause compiling error.       |
 104 |              |                                                              |
 105 |              |  For more SPEC CPU 2006 dependencies please visit            |
 106 |              |  (https://www.spec.org/cpu2006/Docs/techsupport.html)        |
 107 |              |                                                              |
 108 +--------------+--------------------------------------------------------------+
 109 |test sequence | description and expected result                              |
 110 |              |                                                              |
 111 +--------------+--------------------------------------------------------------+
 112 |step 1        | cpu2006-1.2.iso has been saved under the yardstick/resources |
 113 |              | folder (e.g. /home/opnfv/repos/yardstick/yardstick/resources |
 114 |              | /cpu2006-1.2.iso). Additionally, to use your custom runspec  |
 115 |              | config file you can save it under the yardstick/resources/   |
 116 |              | files folder and specify the config file name in the         |
 117 |              | runspec_config parameter.                                    |
 118 |              |                                                              |
 119 +--------------+--------------------------------------------------------------+
 120 |step 2        | Upload SPEC CPU 2006 ISO to the target VM using scp and      |
 121 |              | install SPEC CPU 2006.                                       |
 122 |              |                                                              |
 123 +--------------+--------------------------------------------------------------+
 124 |step 3        | Connect to the target server using SSH.                      |
 125 |              | If custom runspec config file is used, copy this file from   |
 126 |              | yardstick to the target VM via the SSH tunnel.               |
 127 |              |                                                              |
 128 +--------------+--------------------------------------------------------------+
 129 |step 4        | SPEC CPU 2006 benchmark is invoked and SPEC CPU 2006 metrics |
 130 |              | are generated.                                               |
 131 |              |                                                              |
 132 +--------------+--------------------------------------------------------------+
 133 |step 5        | Text, HTML, CSV, PDF, and Configuration file outputs for the |
 134 |              | SPEC CPU 2006 metrics are fetched from the VM and stored     |
 135 |              | under /tmp/result folder.                                    |
 136 |              |                                                              |
 137 +--------------+--------------------------------------------------------------+
 138 |test verdict  | None. SPEC CPU 2006 results are collected and stored.        |
 139 |              |                                                              |
 140 +--------------+--------------------------------------------------------------+