1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
6 ======================================
7 Test Results for os-onos-nofeature-ha
8 ======================================
17 .. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
18 .. _POD6: https://wiki.opnfv.org/pharos?&#community_test_labs
20 Overview of test results
21 ------------------------
23 See Grafana_ for viewing test result metrics for each respective test case. It
24 is possible to chose which specific scenarios to look at, and then to zoom in
25 on the details of each run test scenario as well.
27 All of the test case results below are based on 5 scenario test runs, each run
28 on the Intel POD6_ between September 13 and 16 in 2016.
32 The round-trip-time (RTT) between 2 VMs on different blades is measured using
33 ping. Most test run measurements result on average between 1.50 and 1.68 ms.
34 Only one test run has reached greatest RTT spike of 2.62 ms, which has
35 the smallest RTT of 1.00 ms. The other four runs have no similar spike at all,
36 the minimum and average RTTs of which are approx. 1.06 ms and 1.32 ms. SLA set
37 to be 10 ms. The SLA value is used as a reference, it has not been defined by
42 The IO read bandwidth actually refers to the storage throughput, which is
43 measured by fio and the greatest IO read bandwidth of the four runs is 175.4
44 MB/s. The IO read bandwidth of the four runs looks similar on different four
45 days, with an average between 58.1 and 62.0 MB/s, except one on Sep. 14, for
46 its maximum storage throughput is only 133.0 MB/s. One of the runs has a
47 minimum BW of 497 KM/s and other has a maximum BW of 177.4 MB/s. The SLA of read
48 bandwidth sets to be 400 MB/s, which is used as a reference, and it has not
49 been defined by OPNFV.
51 The results of storage IOPS for the five runs look similar with each other. The
52 IO read times per second of the five test runs have an average value between
53 1.20 K/s and 1.61 K/s, and meanwhile, the minimum result is only 41 times per
58 The tool we use to measure memory read latency is lmbench, which is a series of
59 micro benchmarks intended to measure basic operating system and hardware system
60 metrics. The memory read latency of the five runs is between 1.146 ns and 1.172
61 ns on average. The variations within each test run are quite different, some
62 vary from a large range and others have a small change. For example, the
63 largest change is on September 13, the memory read latency of which is ranging
64 from 1.152 ns to 1.221 ns. However, the results on September 14 change very
65 little. The SLA sets to be 30 ns. The SLA value is used as a reference, it has
66 not been defined by OPNFV.
70 Iperf3 is a tool for evaluating the packet delay variation between 2 VMs on
71 different blades. The reported packet delay variations of the five test runs
72 differ from each other. In general, the packet delay of the first two runs look
73 similar, for they both stay stable within each run. And the mean packet delay of
74 of them are 0.07714 ms and 0.07982 ms respectively. Of the five runs, the third
75 has the worst result, because the packet delay reaches 0.08384 ms. The trend of
76 therest two runs look the same, for the average packet delay are 0.07808 ms and
77 0.07727 ms respectively. The SLA value sets to be 10 ms. The SLA value is used
78 as a reference, it has not been defined by OPNFV.
82 Lmbench is also used to measure the memory read and write bandwidth, in which
83 we use bw_mem to obtain the results. Among the five test runs, the memory
84 bandwidth of last three test runs almost keep stable within each run, which is
85 11.64, 11.71 and 11.61 GB/s on average. However, the memory read and write
86 bandwidth on Sep. 13 has a large range, for it ranges from 6.68 GB/s to 11.73
87 GB/s. Here SLA set to be 15 GB/s. The SLA value is used as a reference, it has
88 not been defined by OPNFV.
92 The Unixbench is used to evaluate the IaaS processing speed with regards to
93 score of single cpu running and parallel running. It can be seen from the
94 dashboard that the processing test results vary from scores 3208 to 3314, and
95 there is only one result one date. No SLA set.
99 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
100 on different blades are measured when increasing the amount of UDP flows sent
101 between the VMs using pktgen as packet generator tool.
103 Round trip times and packet throughput between VMs can typically be affected by
104 the amount of flows set up and result in higher RTT and less PPS throughput.
106 The mean packet throughput of the five test runs is between 259.6 kpps and
107 318.4 kpps, of which the result of the second run is the highest. The RTT
108 results of all the test runs keep flat at approx. 20 ms. It is obvious that the
109 PPS results are not as consistent as the RTT results.
111 The No. flows of the five test runs are 240 k on average and the PPS results
112 look a little waved since the largest packet throughput is 398.9 kpps and the
113 minimum throughput is 250.6 kpps respectively.
115 There are no errors of packets received in the five runs, but there are still
116 lost packets in all the test runs. The RTT values obtained by ping of the five
117 runs have the similar average vaue, that is between 17 ms and 22 ms, of which
118 the worest RTT is 53 ms on Sep. 14th.
120 CPU load is measured by mpstat, and CPU load of the four test runs seem a
121 little similar, since the minimum value and the peak of CPU load is between 0
122 percent and 10 percent respectively. And the best result is obtained on Sep.
123 13rd, with an CPU load of 10 percent.
127 With the block size changing from 1 kb to 512 kb, the memory write bandwidth
128 tends to become larger first and then smaller within every run test, which
129 rangs from 21.6 GB/s to 26.8 GB/s and then to 18.4 GB/s on average. Since the
130 test id is one, it is that only the INT memory write bandwidth is tested. On
131 the whole, when the block size is 8 kb and 16 kb, the memory write bandwidth
132 look similar with a minimal BW of 23.0 GB/s and peak value of 28.6 GB/s. And
133 then with the block size becoming larger, the memory write bandwidth tends to
134 decrease. SLA sets to be 7 GB/s. The SLA value is used as a a reference, it has
135 not been defined by OPNFV.
139 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
140 on different blades are measured when increasing the amount of UDP flows sent
141 between the VMs using pktgen as packet generator tool.
143 Round trip times and packet throughput between VMs can typically be affected by
144 the amount of flows set up and result in higher RTT and less PPS throughput.
146 The network latency is measured by ping, and the results of the five test runs
147 look similar with each other, and within these test runs, the maximum RTT can
148 reach 53 ms and the average RTT is usually approx. 18 ms. The network latency
149 tested on Sep. 14 shows that it has a peak latency of 53 ms. But on the whole,
150 the average RTTs of the five runs keep flat and the network latency is
153 Memory utilization is measured by free, which can display amount of free and
154 used memory in the system. The largest amount of used memory is 272 MiB on Sep
155 14. In general, the mean used memory of the five test runs have the similar
156 trend and the minimum memory used size is approx. 150 MiB, and the average
157 used memory size is about 250 MiB. On the other hand, for the mean free memory,
158 the five test runs have the similar trend, whose mean free memory change from
159 218 MiB to 342 MiB, with an average value of approx. 38 MiB.
161 Packet throughput and packet loss can be measured by pktgen, which is a tool
162 in the network for generating traffic loads for network experiments. The mean
163 packet throughput of the five test runs seem quite different, ranging from
164 285.29 kpps to 297.76 kpps. The average number of flows in these tests is
165 240000, and each run has a minimum number of flows of 2 and a maximum number
166 of flows of 1.001 Mil. At the same time, the corresponding packet throughput
167 differ between 250.6k and 398.9k with an average packet throughput between
168 277.2 K and 318.4 K. In summary, the PPS results seem consistent. Within each
169 test run of the five runs, when number of flows becomes larger, the packet
170 throughput seems not larger at the same time.
174 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
175 on different blades are measured when increasing the amount of UDP flows sent
176 between the VMs using pktgen as packet generator tool.
178 Round trip times and packet throughput between VMs can typically be affected by
179 the amount of flows set up and result in higher RTT and less PPS throughput.
181 The network latency is measured by ping, and the results of the five test runs
182 look similar with each other. Within each test run, the maximum RTT is only 49
183 ms and the average RTT is usually approx. 20 ms. On the whole, the average
184 RTTs of the five runs keep stable and the network latency is relatively short.
186 Cache utilization is measured by cachestat, which can display size of cache and
187 buffer in the system. Cache utilization statistics are collected during UDP
188 flows sent between the VMs using pktgen as packet generator tool.The largest
189 cache size is 215 MiB in the four runs, and the smallest cache size is 95 MiB.
190 On the whole, the average cache size of the five runs change a little and is
191 about 200 MiB, except the one on Sep. 14th, the mean cache size is very small,
192 which keeps 102 MiB. Meanwhile, the tread of the buffer size keep flat, since
193 they have a minimum value of 7 MiB and a maximum value of 8 MiB, with an
194 average value of about 7.8 MiB.
196 Packet throughput can be measured by pktgen, which is a tool in the network for
197 generating traffic loads for network experiments. The mean packet throughput of
198 the four test runs seem quite different, ranging from 285.29 kpps to 297.76
199 kpps. The average number of flows in these tests is 239.7k, and each run has a
200 minimum number of flows of 2 and a maximum number of flows of 1.001 Mil. At the
201 same time, the corresponding packet throughput differ between 227.3k and 398.9k
202 with an average packet throughput between 277.2k and 318.4k. Within each test
203 run of the five runs, when number of flows becomes larger, the packet
204 throughput seems not larger in the meantime.
208 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
209 on different blades are measured when increasing the amount of UDP flows sent
210 between the VMs using pktgen as packet generator tool.
212 Round trip times and packet throughput between VMs can typically be affected by
213 the amount of flows set up and result in higher RTT and less PPS throughput.
215 The RTT results are similar throughout the different test dates and runs
216 between 0 ms and 49 ms with an average leatency of less than 22 ms. The PPS
217 results are not as consistent as the RTT results, for the mean packet
218 throughput of the five runs differ from 250.6 kpps to 398.9 kpps.
220 Network utilization is measured by sar, that is system activity reporter, which
221 can display the average statistics for the time since the system was started.
222 Network utilization statistics are collected during UDP flows sent between the
223 VMs using pktgen as packet generator tool. The largest total number of packets
224 transmitted per second look similar for four test runs, whose values change a
225 lot from 10 pps to 399 kpps, except the one on Sep. 14th, whose total number
226 of transmitted per second keep stable, that is 10 pps. Similarly, the total
227 number of packets received per second look the same for four runs, except the
228 one on Sep. 14th, whose value is only 10 pps.
230 In some test runs when running with less than approx. 90000 flows the PPS
231 throughput is normally flatter compared to when running with more flows, after
232 which the PPS throughput decreases. For the other test runs there is however no
233 significant change to the PPS throughput when the number of flows are
234 increased. In some test runs the PPS is also greater with 250000 flows
235 compared to other test runs where the PPS result is less with only 2 flows.
237 There are lost packets reported in most of the test runs. There is no observed
238 correlation between the amount of flows and the amount of lost packets.
239 The lost amount of packets normally differs a lot per test run.
241 Detailed test results
242 ---------------------
243 The scenario was run on Intel POD5_ with:
247 OpenVirtualSwitch 2.5.90
248 OpenDayLight Beryllium
250 Rationale for decisions
251 -----------------------
254 Conclusions and recommendations
255 -------------------------------
256 Tests were successfully executed and metrics collected.
257 No SLA was verified. To be decided on in next release of OPNFV.
Code Review / yardstick.git / blob