1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
6 ======================================
7 Test Results for os-nosdn-nofeature-ha
8 ======================================
17 .. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
18 .. _POD1: https://wiki.opnfv.org/pharos?&#community_test_labs
21 Overview of test results
22 ------------------------
24 See Grafana_ for viewing test result metrics for each respective test case. It
25 is possible to chose which specific scenarios to look at, and then to zoom in
26 on the details of each run test scenario as well.
28 All of the test case results below are based on 4 scenario test
29 runs, each run on the LF POD1_ between August 25 and 28 in
34 The round-trip-time (RTT) between 2 VMs on different blades is measured using
35 ping. Most test run measurements result on average between 0.74 and 1.08 ms.
36 A few runs start with a 0.99 - 1.07 ms RTT spike (This could be because of
37 normal ARP handling). One test run has a greater RTT spike of 1.35 ms.
38 To be able to draw conclusions more runs should be made. SLA set to 10 ms.
39 The SLA value is used as a reference, it has not been defined by OPNFV.
43 The IO read bandwidth looks similar between different dates, with an
44 average between approx. 128 and 136 MB/s. Within each test run the results
45 vary, with a minimum 5 MB/s and maximum 446 MB/s on the totality. Most runs
46 have a minimum BW of 5 MB/s (one run at 6 MB/s). The maximum BW varies more in
47 absolute numbers between the dates, between 416 and 446 MB/s.
48 SLA set to 400 MB/s. The SLA value is used as a reference, it has not been
53 The measurements for memory latency are similar between test dates and result
54 in approx. 1.09 ns. The variations within each test run are similar, between
56 SLA set to 30 ns. The SLA value is used as a reference, it has not been defined
61 Packet delay variation between 2 VMs on different blades is measured using
62 Iperf3. The reported packet delay variation varies between 0.0025 and 0.0148 ms,
63 with an average delay variation between 0.0056 ms and 0.0157 ms.
67 Between test dates, the average measurements for memory bandwidth result in
68 approx. 19.70 GB/s. Within each test run the results vary more, with a minimal
69 BW of 18.16 GB/s and maximum of 20.13 GB/s on the totality.
70 SLA set to 15 GB/s. The SLA value is used as a reference, it has not been
75 The Unixbench processor test run results vary between scores 3224.4 and 3842.8,
76 one result each date. The average score on the total is 3659.5.
81 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
82 on different blades are measured when increasing the amount of UDP flows sent
83 between the VMs using pktgen as packet generator tool.
85 Round trip times and packet throughput between VMs can typically be affected by
86 the amount of flows set up and result in higher RTT and less PPS throughput.
88 The RTT results are similar throughout the different test dates and runs at
89 approx. 15 ms. Some test runs show an increase with many flows, in the range
90 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
91 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
93 In some test runs when running with less than approx. 10000 flows the PPS
94 throughput is normally flatter compared to when running with more flows, after
95 which the PPS throughput decreases. Around 20 percent decrease in the worst
96 case. For the other test runs there is however no significant change to the PPS
97 throughput when the number of flows are increased. In some test runs the PPS
98 is also greater with 1000000 flows compared to other test runs where the PPS
99 result is less with only 2 flows.
101 The average PPS throughput in the different runs varies between 414000 and
102 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
103 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
104 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
107 There are lost packets reported in most of the test runs. There is no observed
108 correlation between the amount of flows and the amount of lost packets.
109 The lost amount of packets normally range between 100 and 1000 per test run,
110 but there are spikes in the range of 10000 lost packets as well, and even
111 more in a rare cases.
113 CPU utilization statistics are collected during UDP flows sent between the VMs
114 using pktgen as packet generator tool. The average measurements for CPU
115 utilization ratio vary between 1% to 2%. The peak of CPU utilization ratio
120 Between test dates, the average measurements for memory bandwidth vary between
121 22.6 and 29.1 GB/s. Within each test run the results vary more, with a minimal
122 BW of 20.0 GB/s and maximum of 29.5 GB/s on the totality.
123 SLA set to 6 GB/s. The SLA value is used as a reference, it has not been
128 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
129 on different blades are measured when increasing the amount of UDP flows sent
130 between the VMs using pktgen as packet generator tool.
132 Round trip times and packet throughput between VMs can typically be affected by
133 the amount of flows set up and result in higher RTT and less PPS throughput.
135 The RTT results are similar throughout the different test dates and runs at
136 approx. 15 ms. Some test runs show an increase with many flows, in the range
137 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
138 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
140 In some test runs when running with less than approx. 10000 flows the PPS
141 throughput is normally flatter compared to when running with more flows, after
142 which the PPS throughput decreases. Around 20 percent decrease in the worst
143 case. For the other test runs there is however no significant change to the PPS
144 throughput when the number of flows are increased. In some test runs the PPS
145 is also greater with 1000000 flows compared to other test runs where the PPS
146 result is less with only 2 flows.
148 The average PPS throughput in the different runs varies between 414000 and
149 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
150 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
151 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
154 There are lost packets reported in most of the test runs. There is no observed
155 correlation between the amount of flows and the amount of lost packets.
156 The lost amount of packets normally range between 100 and 1000 per test run,
157 but there are spikes in the range of 10000 lost packets as well, and even
158 more in a rare cases.
160 Memory utilization statistics are collected during UDP flows sent between the
161 VMs using pktgen as packet generator tool. The average measurements for memory
162 utilization vary between 225MB to 246MB. The peak of memory utilization appears
167 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
168 on different blades are measured when increasing the amount of UDP flows sent
169 between the VMs using pktgen as packet generator tool.
171 Round trip times and packet throughput between VMs can typically be affected by
172 the amount of flows set up and result in higher RTT and less PPS throughput.
174 The RTT results are similar throughout the different test dates and runs at
175 approx. 15 ms. Some test runs show an increase with many flows, in the range
176 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
177 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
179 In some test runs when running with less than approx. 10000 flows the PPS
180 throughput is normally flatter compared to when running with more flows, after
181 which the PPS throughput decreases. Around 20 percent decrease in the worst
182 case. For the other test runs there is however no significant change to the PPS
183 throughput when the number of flows are increased. In some test runs the PPS
184 is also greater with 1000000 flows compared to other test runs where the PPS
185 result is less with only 2 flows.
187 The average PPS throughput in the different runs varies between 414000 and
188 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
189 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
190 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
193 There are lost packets reported in most of the test runs. There is no observed
194 correlation between the amount of flows and the amount of lost packets.
195 The lost amount of packets normally range between 100 and 1000 per test run,
196 but there are spikes in the range of 10000 lost packets as well, and even
197 more in a rare cases.
199 Cache utilization statistics are collected during UDP flows sent between the
200 VMs using pktgen as packet generator tool. The average measurements for cache
201 utilization vary between 205MB to 212MB.
205 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
206 on different blades are measured when increasing the amount of UDP flows sent
207 between the VMs using pktgen as packet generator tool.
209 Round trip times and packet throughput between VMs can typically be affected by
210 the amount of flows set up and result in higher RTT and less PPS throughput.
212 The RTT results are similar throughout the different test dates and runs at
213 approx. 15 ms. Some test runs show an increase with many flows, in the range
214 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
215 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
217 In some test runs when running with less than approx. 10000 flows the PPS
218 throughput is normally flatter compared to when running with more flows, after
219 which the PPS throughput decreases. Around 20 percent decrease in the worst
220 case. For the other test runs there is however no significant change to the PPS
221 throughput when the number of flows are increased. In some test runs the PPS
222 is also greater with 1000000 flows compared to other test runs where the PPS
223 result is less with only 2 flows.
225 The average PPS throughput in the different runs varies between 414000 and
226 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
227 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
228 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
231 There are lost packets reported in most of the test runs. There is no observed
232 correlation between the amount of flows and the amount of lost packets.
233 The lost amount of packets normally range between 100 and 1000 per test run,
234 but there are spikes in the range of 10000 lost packets as well, and even
235 more in a rare cases.
237 Network utilization statistics are collected during UDP flows sent between the
238 VMs using pktgen as packet generator tool. Total number of packets received per
239 second was average on 200 kpps and total number of packets transmitted per
240 second was average on 600 kpps.
242 Detailed test results
243 ---------------------
244 The scenario was run on LF POD1_ with:
247 OpenVirtualSwitch 2.5.90
248 OpenDayLight Beryllium
250 Rationale for decisions
251 -----------------------
254 Tests were successfully executed and metrics collected.
255 No SLA was verified. To be decided on in next release of OPNFV.
261 .. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
262 .. _POD5: https://wiki.opnfv.org/pharos?&#community_test_labs
265 Overview of test results
266 ------------------------
268 See Grafana_ for viewing test result metrics for each respective test case. It
269 is possible to chose which specific scenarios to look at, and then to zoom in
270 on the details of each run test scenario as well.
272 All of the test case results below are based on 4 scenario test runs, each run
273 on the Intel POD5_ between September 11 and 14 in 2016.
277 The round-trip-time (RTT) between 2 VMs on different blades is measured using
278 ping. Most test run measurements result on average between 1.59 and 1.70 ms.
279 Two test runs have reached the same greater RTT spike of 3.06 ms, which are
280 1.66 and 1.70 ms average, but only one has the lower RTT of 1.35 ms. The other
281 two runs have no similar spike at all. To be able to draw conclusions more runs
282 should be made. SLA set to be 10 ms. The SLA value is used as a reference, it
283 has not been defined by OPNFV.
287 The IO read bandwidth actually refers to the storage throughput and the
288 greatest IO read bandwidth of the four runs is 173.3 MB/s. The IO read
289 bandwidth of the four runs looks similar on different four days, with an
290 average between 32.7 and 60.4 MB/s. One of the runs has a minimum BW of 429
291 KM/s and other has a maximum BW of 173.3 MB/s. The SLA of read bandwidth sets
292 to be 400 MB/s, which is used as a reference, and it has not been defined by
297 The tool we use to measure memory read latency is lmbench, which is a series of
298 micro benchmarks intended to measure basic operating system and hardware system
299 metrics. The memory read latency of the four runs is 1.1 ns on average. The
300 variations within each test run are different, some vary from a large range and
301 others have a small change. For example, the largest change is on September 14,
302 the memory read latency of which is ranging from 1.12 ns to 1.22 ns. However,
303 the results on September 12 change very little, which range from 1.14 ns to
304 1.17 ns. The SLA sets to be 30 ns. The SLA value is used as a reference, it has
305 not been defined by OPNFV.
309 Iperf3 is a tool for evaluating the pocket delay variation between 2 VMs on
310 different blades. The reported pocket delay variations of the four test runs
311 differ from each other. The results on September 13 within the date look
312 similar and the values are between 0.0087 and 0.0190 ms, which is 0.0126 ms on
313 average. However, on the fourth day, the pocket delay variation has a large
314 wide change within the date, which ranges from 0.0032 ms to 0.0121 ms and has
315 the minimum average value. The pocket delay variations of other two test runs
316 look relatively similar, which are 0.0076 ms and 0.0152 ms on average. The SLA
317 value sets to be 10 ms. The SLA value is used as a reference, it has not been
322 Lmbench is also used to measure the memory read and write bandwidth, in which
323 we use bw_mem to obtain the results. Among the four test runs, the memory
324 bandwidth within the second day almost keep stable, which is 11.58 GB/s on
325 average. And the memory bandwidth of the fourth day look similar as that of the
326 second day, both of which remain stable. The other two test runs relatively
327 change from a large wide range, in which the minimum memory bandwidth is 11.22
328 GB/s and the maximum bandwidth is 16.65 GB/s with an average bandwidth of about
329 12.20 GB/s. Here SLA set to be 15 GB/s. The SLA value is used as a reference,
330 it has not been defined by OPNFV.
334 The Unixbench is used to measure processing speed, that is instructions per
335 second. It can be seen from the dashboard that the processing test results
336 vary from scores 3272 to 3444, and there is only one result one date. The
337 overall average score is 3371. No SLA set.
341 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
342 on different blades are measured when increasing the amount of UDP flows sent
343 between the VMs using pktgen as packet generator tool.
345 Round trip times and packet throughput between VMs can typically be affected by
346 the amount of flows set up and result in higher RTT and less PPS throughput.
348 The mean packet throughput of the four test runs is 119.85, 128.02, 121.40 and
349 126.08 kpps, of which the result of the second is the highest. The RTT results
350 of all the test runs keep flat at approx. 37 ms. It is obvious that the PPS
351 results are not as consistent as the RTT results.
353 The No. flows of the four test runs are 240 k on average and the PPS results
354 look a little waved since the largest packet throughput is 184 kpps and the
355 minimum throughput is 49 K respectively.
357 There are no errors of packets received in the four runs, but there are still
358 lost packets in all the test runs. The RTT values obtained by ping of the four
359 runs have the similar average vaue, that is 38 ms, of which the worest RTT is
362 CPU load of the four test runs have a large change, since the minimum value and
363 the peak of CPU load is 0 percent and 51 percent respectively. And the best
364 result is obtained on Sep. 14th.
368 With the block size changing from 1 kb to 512 kb, the memory write bandwidth
369 tends to become larger first and then smaller within every run test, which
370 rangs from 22.3 GB/s to 26.8 GB/s and then to 18.5 GB/s on average. Since the
371 test id is one, it is that only the INT memory write bandwidth is tested. On
372 the whole, when the block size is 8 kb and 16 kb, the memory write bandwidth
373 look similar with a minimal BW of 22.5 GB/s and peak value of 28.7 GB/s. SLA
374 sets to be 7 GB/s. The SLA value is used as a a reference, it has not been
379 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
380 on different blades are measured when increasing the amount of UDP flows sent
381 between the VMs using pktgen as packet generator tool.
383 Round trip times and packet throughput between VMs can typically be affected by
384 the amount of flows set up and result in higher RTT and less PPS throughput.
386 The network latency is measured by ping, and the results of the four test runs
387 look similar with each other. Within each test run, the maximum RTT can reach
388 more than 80 ms and the average RTT is usually approx. 38 ms. On the whole, the
389 average RTTs of the four runs keep flat.
391 Memory utilization is measured by free, which can display amount of free and
392 used memory in the system. The largest amount of used memory is 268 MiB on Sep
393 14, which also has the largest minimum memory. Besides, the rest three test
394 runs have the similar used memory. On the other hand, the free memory of the
395 four runs have the same smallest minimum value, that is about 223 MiB, and the
396 maximum free memory of three runs have the similar result, that is 337 MiB,
397 except that on Sep. 14th, whose maximum free memory is 254 MiB. On the whole,
398 all the test runs have similar average free memory.
400 Network throughput and packet loss can be measured by pktgen, which is a tool
401 in the network for generating traffic loads for network experiments. The mean
402 network throughput of the four test runs seem quite different, ranging from
403 119.85 kpps to 128.02 kpps. The average number of flows in these tests is
404 24000, and each run has a minimum number of flows of 2 and a maximum number
405 of flows of 1.001 Mil. At the same time, the corresponding packet throughput
406 differ between 49.4k and 193.3k with an average packet throughput of approx.
407 125k. On the whole, the PPS results seem consistent. Within each test run of
408 the four runs, when number of flows becomes larger, the packet throughput seems
409 not larger in the meantime.
413 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
414 on different blades are measured when increasing the amount of UDP flows sent
415 between the VMs using pktgen as packet generator tool.
417 Round trip times and packet throughput between VMs can typically be affected by
418 the amount of flows set up and result in higher RTT and less PPS throughput.
420 The network latency is measured by ping, and the results of the four test runs
421 look similar with each other. Within each test run, the maximum RTT can reach
422 more than 94 ms and the average RTT is usually approx. 35 ms. On the whole, the
423 average RTTs of the four runs keep flat.
425 Cache utilization is measured by cachestat, which can display size of cache and
426 buffer in the system. Cache utilization statistics are collected during UDP
427 flows sent between the VMs using pktgen as packet generator tool.The largest
428 cache size is 212 MiB in the four runs, and the smallest cache size is 75 MiB.
429 On the whole, the average cache size of the four runs is approx. 208 MiB.
430 Meanwhile, the tread of the buffer size looks similar with each other.
432 Packet throughput can be measured by pktgen, which is a tool in the network for
433 generating traffic loads for network experiments. The mean packet throughput of
434 the four test runs seem quite different, ranging from 119.85 kpps to 128.02
435 kpps. The average number of flows in these tests is 239.7k, and each run has a
436 minimum number of flows of 2 and a maximum number of flows of 1.001 Mil. At the
437 same time, the corresponding packet throughput differ between 49.4k and 193.3k
438 with an average packet throughput of approx. 125k. On the whole, the PPS results
439 seem consistent. Within each test run of the four runs, when number of flows
440 becomes larger, the packet throughput seems not larger in the meantime.
444 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
445 on different blades are measured when increasing the amount of UDP flows sent
446 between the VMs using pktgen as packet generator tool.
448 Round trip times and packet throughput between VMs can typically be affected by
449 the amount of flows set up and result in higher RTT and less PPS throughput.
451 The RTT results are similar throughout the different test dates and runs at
452 approx. 32 ms. The PPS results are not as consistent as the RTT results.
454 Network utilization is measured by sar, that is system activity reporter, which
455 can display the average statistics for the time since the system was started.
456 Network utilization statistics are collected during UDP flows sent between the
457 VMs using pktgen as packet generator tool. The largest total number of packets
458 transmitted per second differs from each other, in which the smallest number of
459 packets transmitted per second is 6 pps on Sep. 12ed and the largest of that is
460 210.8 kpps. Meanwhile, the largest total number of packets received per second
461 differs from each other, in which the smallest number of packets received per
462 second is 2 pps on Sep. 13rd and the largest of that is 250.2 kpps.
464 In some test runs when running with less than approx. 90000 flows the PPS
465 throughput is normally flatter compared to when running with more flows, after
466 which the PPS throughput decreases. For the other test runs there is however no
467 significant change to the PPS throughput when the number of flows are
468 increased. In some test runs the PPS is also greater with 1000000 flows
469 compared to other test runs where the PPS result is less with only 2 flows.
471 There are lost packets reported in most of the test runs. There is no observed
472 correlation between the amount of flows and the amount of lost packets.
473 The lost amount of packets normally differs a lot per test run.
475 Detailed test results
476 ---------------------
477 The scenario was run on Intel POD5_ with:
480 OpenVirtualSwitch 2.5.90
481 OpenDayLight Beryllium
483 Rationale for decisions
484 -----------------------
487 Conclusions and recommendations
488 -------------------------------
489 Tests were successfully executed and metrics collected.
490 No SLA was verified. To be decided on in next release of OPNFV.
Code Review / yardstick.git / blob