1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
4 .. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
6 *************************************
7 Yardstick Test Case Description TC084
8 *************************************
10 .. _spec_cpu_2006: https://www.spec.org/cpu2006/
12 +-----------------------------------------------------------------------------+
13 |Compute Performance |
15 +--------------+--------------------------------------------------------------+
16 |test case id | OPNFV_YARDSTICK_TC084_SPEC CPU 2006 FOR VM |
18 +--------------+--------------------------------------------------------------+
19 |metric | compute-intensive performance |
21 +--------------+--------------------------------------------------------------+
22 |test purpose | The purpose of TC084 is to evaluate the IaaS compute |
23 | | performance by using SPEC CPU 2006 benchmark. The SPEC CPU |
24 | | 2006 benchmark has several different ways to measure |
25 | | computer performance. One way is to measure how fast the |
26 | | computer completes a single task; this is called a speed |
27 | | measurement. Another way is to measure how many tasks |
28 | | computer can accomplish in a certain amount of time; this is |
29 | | called a throughput, capacity or rate measurement. |
31 +--------------+--------------------------------------------------------------+
32 |test tool | SPEC CPU 2006 |
34 | | The SPEC CPU 2006 benchmark is SPEC's industry-standardized, |
35 | | CPU-intensive benchmark suite, stressing a system's |
36 | | processor, memory subsystem and compiler. This benchmark |
37 | | suite includes the SPECint benchmarks and the SPECfp |
38 | | benchmarks. The SPECint 2006 benchmark contains 12 different |
39 | | benchmark tests and the SPECfp 2006 benchmark contains 19 |
40 | | different benchmark tests. |
42 | | SPEC CPU 2006 is not always part of a Linux distribution. |
43 | | SPEC requires that users purchase a license and agree with |
44 | | their terms and conditions. For this test case, users must |
45 | | manually download cpu2006-1.2.iso from the SPEC website and |
46 | | save it under the yardstick/resources folder (e.g. /home/ |
47 | | opnfv/repos/yardstick/yardstick/resources/cpu2006-1.2.iso) |
48 | | SPEC CPU® 2006 benchmark is available for purchase via the |
49 | | SPEC order form (https://www.spec.org/order.html). |
51 +--------------+--------------------------------------------------------------+
52 |test | This test case uses SPEC CPU 2006 benchmark to measure |
53 |description | compute-intensive performance of VMs. |
55 +--------------+--------------------------------------------------------------+
56 |configuration | file: opnfv_yardstick_tc084.yaml |
58 | | benchmark_subset is set to int. |
60 | | SLA is not available in this test case. |
62 +--------------+--------------------------------------------------------------+
63 |applicability | Test can be configured with different: |
65 | | * benchmark_subset - a subset of SPEC CPU 2006 benchmarks |
67 | | * SPECint_benchmark - a SPECint benchmark to run; |
68 | | * SPECint_benchmark - a SPECfp benchmark to run; |
69 | | * output_format - desired report format; |
70 | | * runspec_config - SPEC CPU 2006 config file provided to |
71 | | the runspec binary; |
72 | | * runspec_iterations - the number of benchmark iterations |
73 | | to execute. For a reportable run, must be 3; |
74 | | * runspec_tune - tuning to use (base, peak, or all). For a |
75 | | reportable run, must be either base or all. Reportable |
76 | | runs do base first, then (optionally) peak; |
77 | | * runspec_size - size of input data to run (test, train, or |
78 | | ref). Reportable runs ensure that your binaries can |
79 | | produce correct results with the test and train workloads |
81 +--------------+--------------------------------------------------------------+
82 |usability | This test case is used for executing SPEC CPU 2006 benchmark |
83 | | on virtual machines. The SPECint 2006 benchmark takes |
84 | | approximately 5 hours. (The time may vary due to different |
85 | | VM cpu configurations) |
87 +--------------+--------------------------------------------------------------+
88 |references | spec_cpu_2006_ |
92 +--------------+--------------------------------------------------------------+
93 |pre-test | To run and install SPEC CPU 2006, the following are |
94 |conditions | required: |
96 | | * For SPECint 2006: Both C99 and C++98 compilers are |
97 | | installed in VM images; |
98 | | * For SPECfp 2006: All three of C99, C++98 and Fortran-95 |
99 | | compilers installed in VM images; |
100 | | * At least 4GB of disk space availabile on VM. |
102 | | gcc 4.8.* and g++ 4.8.* version have been tested in Ubuntu |
103 | | 14.04, Ubuntu 16.04 and Redhat Enterprise Linux 7.4 image. |
104 | | Higher gcc and g++ version may cause compiling error. |
106 | | For more SPEC CPU 2006 dependencies please visit |
107 | | (https://www.spec.org/cpu2006/Docs/techsupport.html) |
109 +--------------+--------------------------------------------------------------+
110 |test sequence | description and expected result |
112 +--------------+--------------------------------------------------------------+
113 |step 1 | cpu2006-1.2.iso has been saved under the yardstick/resources |
114 | | folder (e.g. /home/opnfv/repos/yardstick/yardstick/resources |
115 | | /cpu2006-1.2.iso). Additionally, to use your custom runspec |
116 | | config file you can save it under the yardstick/resources/ |
117 | | files folder and specify the config file name in the |
118 | | runspec_config parameter. |
120 +--------------+--------------------------------------------------------------+
121 |step 2 | Upload SPEC CPU 2006 ISO to the target VM using scp and |
122 | | install SPEC CPU 2006. |
124 +--------------+--------------------------------------------------------------+
125 |step 3 | Connect to the target server using SSH. |
126 | | If custom runspec config file is used, copy this file from |
127 | | yardstick to the target VM via the SSH tunnel. |
129 +--------------+--------------------------------------------------------------+
130 |step 4 | SPEC CPU 2006 benchmark is invoked and SPEC CPU 2006 metrics |
133 +--------------+--------------------------------------------------------------+
134 |step 5 | Text, HTML, CSV, PDF, and Configuration file outputs for the |
135 | | SPEC CPU 2006 metrics are fetched from the VM and stored |
136 | | under /tmp/result folder. |
138 +--------------+--------------------------------------------------------------+
139 |test verdict | None. SPEC CPU 2006 results are collected and stored. |
141 +--------------+--------------------------------------------------------------+