1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
4 .. (c) OPNFV, Intel Corporation and others.
6 PROX - Packet pROcessing eXecution engine.
7 ==========================================
9 Change since previous release, support has been added for the following:
11 8 workloads for automated dataplane benchmarking using DATS
13 L4 stateful traffic generation and flow extraction tool
14 lua configuration files for easy table population
15 New modes: impair, lb5tuple, mirror, nat, decapnsh, encapnsh and genl4
16 helper script for automated VM core pinning for Qemu
17 New screens for viewing information regarding DPDK rings (screen 5) and L4 generation (screen 6)
18 Improved command editing using libedit
19 Improved ncurses display
20 Rename of dppd-bng zip file to dppd-prox
21 Latency histogram collection
23 PROX COMMANDS AND SCREENS
24 -------------------------
26 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
27 | **RUNTIME COMMAND** | **DESCRIPTION** | **EXAMPLE** |
28 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
29 |quit | Stop all cores and quit | |
30 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
31 |help <substr> | Show list of commands that have <substr> as a substring. | |
32 | | If no substring is provided, all commands are shown. | |
33 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
34 |verbose <level> | Set the verbosity level of some printed messages. | |
35 | | Possible values are: 0 (default value, error messages only), | verbose 1 |
36 | | 1 (+ warnings), 2 (+ info) and 3 (+ debugging) | |
37 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
38 |thread info <core_id> <task_id> | Show task specific information | |
39 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
40 |update interval <value> | Update statistics refresh rate, in msec (must be >=10). | |
41 | | Default is 1 second | update interval 500 |
42 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
43 |rx tx info | Print connections between tasks on all cores | |
44 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
45 |start <core list>|all <task_id> | Start cores specified in <core list> or all cores. | start all |
46 | | If <task_id> is not specified, all tasks for the specified cores | start 1 |
47 | | will be started. | start 1s0-4s0 |
48 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
49 |stop <core list>|all <task_id> | Stop cores specified in <core list> or all cores. | |
50 | | If <task_id> is not specified, all tasks for the specified | stop 1 |
51 | | cores will be stopped. | |
52 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
53 |dump <coreid> <taskid> <nbpkts> | Create a hex dump of <nb_packets> from <task_id> on <core_id> | dump 2 1 5 |
54 | | showing how packets have changed between RX and TX. | |
55 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
56 |dump_rx <coreid> <taskid> <nbpkts> | Create a hex dump of <nb_packets> from <task_id> on <coreid> at RX | dump_rx 2 1 5 |
57 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
58 |dump_tx <coreid> <taskid> <nbpkts> | Create a hex dump of <nb_packets> from <task_id> on <coreid> at TX | dump_tx 2 1 5 |
59 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
60 |rx distr start | Start gathering statistical distribution of received packets | |
61 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
62 |rx distr stop | Stop gathering statistical distribution of received packets | |
63 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
64 |rx distr reset | Reset gathered statistical distribution of received packets | |
65 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
66 |rx distr show | Display gathered statistical distribution of received packets | |
67 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
68 |rate <port id> <queue id> <rate> | Set transmit rate in Mb/s. This does not include preamble, SFD and IFG | rate 0 0 1000 |
69 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
70 |count <core id> <task id> <count> | Generate <count> packets, then pause generating | count 1 0 5 |
71 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
72 |pkt_size <coreid> <taskid> <pktsize> | Set the packet size to <pkt_size> | pkt_size 1 3 255 |
73 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
74 |speed <core_id> <task_id> <speed percentage> | Change the speed to <speed percentage> of a |
75 | | 10 Gbps line at which packets are being generated | speed 1 0 50 |
76 | | on core <core_id> in task <task_id> |
77 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
78 |speed_byte <core_id> <task_id> <speed> | Change speed to <speed>. The speed is specified in units of bytes per sec | |
79 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
80 |set value <core_id> <task_id> <offset> | Set <value_len> bytes to <value> at offset <offset> in packets | |
81 | <value> <value_len> | generated on <core_id> <task_id> | set value 4 1 14 10 1 |
82 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
83 | reset values all | Undo all `set value` commands on all cores/tasks | |
84 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
85 |reset values <core id> <task id> | Undo all `set value` commands on specified core/task | |
86 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
87 |arp add <core id> <task id> <port id> | | |
88 | <gre id> <svlan> <cvlan> <ip addr> | | |
89 | <mac addr> <user> | Add a single ARP entry into a CPE table on <core id>/<task id> | |
90 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
91 |rule add <core id> <task id> svlan_id&mask | | |
92 | cvlan_id&mask ip_proto&mask | | |
93 | source_ip/prefix destination_ip/prefix | | |
94 | range dport_range action | Add a rule to the ACL table on <core id>/<task id> | |
95 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
96 |route add <core id> <task id> | | |
97 | <ip/prefix> <next hop id> | Add a route to the routing table on core <core id> <task id> | route add 10.0.16.0/24 9 |
98 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
99 |reset stats | Reset all statistics | |
100 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
101 |tot stats | Print total RX and TX packets | |
102 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
103 |tot ierrors per sec | Print total number of ierrors per second | |
104 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
105 |pps stats | Print RX and TX packet rate in unit of packet per second | |
106 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
107 |lat stats <core id> <task id> | Print min,max,avg latency as measured during last sampling interval | lat stats 1 0 |
108 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
109 |lat packets <core id> <task id> | Print the latency for each of the last set of packets | |
110 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
111 |core stats <core id> <task id> | Print rx/tx/drop for task <task id> running on core <core id> | |
112 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
113 |port_stats <port id> | Print rate for no_mbufs, ierrors, rx_bytes, tx_bytes, rx_pkts, | |
114 | | tx_pkts and totals for RX, TX, no_mbufs ierrors for port <port id> | |
115 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
116 |ring info all | Get information about ring, such as ring size and | |
117 | | number of elements in the ring | |
118 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
119 |ring info <core id> <task id> | Get information about ring on core <core id> | |
120 | | in task <task id>, such as ring size and number of elements in the ring | ring info 1 0 |
121 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
122 |port info <port id> [brief] | Get port related information, such as MAC address, socket, | |
123 | | number of descriptors..., . Adding `brief` after command | |
124 | | prints short version of output. | port info 1 |
125 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
126 |port up <port id> | Set the port up (all ports are up at startup) | port up 1 |
127 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
128 |port down <port id> | Set the port down | port down 1 |
129 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
130 |port xstats <port id> | Get extra statistics for the port | port xstats 1 |
131 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
132 |version | Show version | |
133 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
134 |port_stats <port id> | Print rate for no_mbufs, ierrors, rx_bytes, tx_bytes, rx_pkts, | |
135 | | tx_pkts and totals for RX, TX, no_mbufs ierrors for port <port id> | |
136 +----------------------------------------------+---------------------------------------------------------------------------+----------------------------+
138 While PROX is running, F1 to F6 change the view on the system. Pressing F1 switches to the main screen showing per core statistics. When PROX is started,
139 this is the screen shown by default. Pressing F2 switches to show port-based information. Pressing F3 shows information (i.e. occupancy, memory usage, ...)
140 about memory pools. If there are tasks with mode=lat, F4 displays latency measurements made during the last second by each of those tasks.
141 F5 displays DPDK ring information. F6 is for L4 generation. If no command has been entered, numbers 1 to 6 can also be used to change the view on the system.
142 This is provided to allow changing screens in environments that do not pass function keys to PROX.
144 Page Up and Page Down can be used to view per core statistics that would otherwise not fit on the screen. Escape quits PROX.
145 The history of previously entered commands can be navigated using the Up and Down arrows. Statistics can be reset with F12.
149 Run PROX with the "--help" argument to display the usage text and the list of supported options as shown below.
150 PROX supports many compilation flags to enable or disable features. For these flags, refer to the Makefile.
151 Refer to the README file for more information on how to run PROX for specific use cases.
155 Usage: ./build/prox [-f CONFIG_FILE] [-l LOG_FILE] [-p] [-o DISPLAY] [-v] [-a|-e] [-m|-s|-i] [-n] [-w DEF] [-q] [-k] [-d] [-z] [-r VAL] [-u] [-t]
156 -f CONFIG_FILE : configuration file to load, ./prox.cfg by default
157 -l LOG_FILE : log file name, ./prox.log by default
158 -p : include PID in log file name if default log file is used
159 -o DISPLAY: Set display to use, can be 'curses' (default), 'cli' or 'none'
160 -v verbosity : initial logging verbosity
161 -a : autostart all cores (by default)
163 -n : Create NULL devices instead of using PCI devices, useful together with -i
164 -m : list supported task modes and exit
165 -s : check configuration file syntax and exit
166 -i : check initialization sequence and exit
167 -u : Listen on UDS /tmp/prox.sock
168 -t : Listen on TCP port 8474
169 -q : Pass argument to Lua interpreter, useful to define variables
170 -w : define variable using syntax varname=value
171 takes precedence over variables defined in CONFIG_FILE
172 -k : Log statistics to file "stats_dump" in current directory
173 -d : Run as daemon, the parent process will block until PROX is not initialized
174 -z : Ignore CPU topology, implies -i
175 -r : Change initial screen refresh rate. If set to a lower than 0.001 seconds,
176 screen refreshing will be disabled
178 CONFIGURATION FILE FORMAT
179 -------------------------
180 The configuration file is divided into multiple sections, each of which is used to define some parameters and options.
181 Sections are created using the [section name] syntax. The list of sections, where # represents an integer, is as follows:
191 In each section, entries are created using the key=value syntax.
192 Comments are created using the ; symbol: all characters from the ;
193 symbol to the end of line are ignored. A # symbol at the beginning of the section name comments
194 the whole section out: all entries in the section are treated as comments and are ignored. For example:
199 parameter name=parameter value ; this entry is ignored because the section is commented out
202 The following parameters are supported:
205 -m ; Specifies the amount of memory used. If not provided, all hugepages will be used.
206 -n ; Specifies the number of memory channels. Use -n4 for latest Intel Xeon based platforms
207 -r ; Specifies the number of memory ranks.
208 eal ; Specifies DPDK EAL extra options. Those options will be passed blindly to DPDK.
211 DPDK ports are usually referenced by their port_id, i.e. an integer starting from 0.
212 Using port_id in the configuration file is tedious, since the same port_id can appear at
213 different places (rx port, tx port, routing tables),
214 and those ports might change (e.g. if cables are swapped).
215 In order to make the configuration file easier to read and modify,
216 DPDK ports are given a name with the name= option.
217 The name serves as the reference, and in addition, it will show up in the display at runtime.
220 PARAMETER EXAMPLE DESCRIPTION
221 ----------------------------------------------------------------------------
222 name inet0 Use inet0 to later refer to this port
223 mac hardware value can be: hardware, random or a literal MAC address
224 rx desc 256 number of descriptors to allocate for reception
225 tx desc 256 number of descriptors to allocate for transmission
226 promiscuous yes enable promiscuous mode
227 strip crc yes enable CRC stripping
229 lsc no While lsc is disabled for drivers known to not provide support,
230 this option explicitely overrides these settings.
231 rx_ring dpdk_ring_name use DPDK ring as an interface (receive side)
232 tx_ring dpdk_ring_name use DPDK ring as an interface (transmit side)
235 Variables can be defined in the configuration file using the $varname=value syntax.
236 Variables defined on the command line (-w varname=value) take precedence and do not
237 create conflicts with variables defined in the configuration file. Variables are
238 used in the configuration file using the $varname syntax: each instance of $varname
239 is replaced by its associated value. This is typically useful if the same parameter
240 must be used at several places. For instance, you might want to have multiple load
241 balancers, all transmitting to the same set of worker cores.
242 The list of worker cores could then be defined once in a variable:
248 Then, a load balancer definition would use the variable:
258 And the section defining the worker cores would be:
268 The default value of some options can be overridden using the [defaults] section:
271 PARAMETER EXAMPLE DESCRIPTION
272 -----------------------------------
273 mempool size 16K number of mbufs per task, relevant when task receives from a port.
274 this is the n argument provided to rte_mempool_create()
275 qinq tag 0xa888 Set qinq tag for all tasks. The result of adding this option is the
276 same as adding qinq tag= to each task
277 memcache size 128 number of mbufs cached per core, default is 256 this is the cache_size
278 argument provided to rte_mempool_create()
281 The following parameters are supported:
284 PARAMETER EXAMPLE DESCRIPTION
285 -------------------------------------------------
286 name BNG Name of the configuration, which will be shown in the title box at runtime.
287 start time 10 Time in seconds after which average statistics will be started.
289 duration time 30 Runtime duration in seconds, counted after start time.
290 This is typically useful to automate testing using
291 different parameters: PROX automatically exits when the
292 runtime duration has elapsed. Initialization and start time
293 are not included in this runtime duration.
294 For example, if start time is set to 10 and duration time is set to 30,
295 the total execution time (after initialization) will be 40 seconds.
296 Default value is 0, which means infinity and prevents PROX from automatically exiting.
297 shuffle yes When this parameter is set to yes, the order of mbufs
298 within mempools is randomized to simulate a system that has
299 been warmed up. Default value is no.
300 gre cfg /path/to/file.csv Path to CSV file that provides QinQ-to-GRE mapping.
301 Default value is gre_table.csv in same directory as
302 configuration file. Fields are GRE key and QinQ value (computed as SVLAN * 4096 + CVLAN).
303 pre cmd ls Arbitrary system commands to run while reading cfg. This option can occur multiple times.
304 user cfg /path/to/file.csv Path to CSV file that provides QinQ-to-User mapping.
305 Default value is user_table.csv in same directory as configuration file.
306 Fields are SVLAN, CVLAN and User-Id.
307 next hop cfg /path/to/file.csv Path to CSV file that provides Next-Hop details.
308 Default value is next_hop.csv in same directory as configuration file.
309 Fields are Next-Hop index (as returned by LPM lookup),
310 Out-Port index, Next-Hop IP (unused), Next-Hop MAC and MPLS label.
311 ipv4 cfg /path/to/file.csv Path to CSV file that provides IPv4 LPM routing table.
312 Default value is ipv4.csv in same directory as configuration file.
313 Fields are IPv4 subnet (in CIDR notation) and Next-Hop index.
314 dscp cfg /path/to/file.csv Path to CSV file that provides mapping for QoS classification,
315 from DSCP to Traffic Class and Queue.
316 Default value is dscp.csv in same directory as configuration file.
317 Fields are DSCP (0-63), Traffic Class (0-3) and Queue (0-3).
318 ipv6 tunnel cfg /path/to/file.csv Path to CSV file that provides lwAFTR binding table.
319 Default value is ipv6_tun_bind.csv in same directory as configuration file.
320 Fields are lwB4 IPv6 address, next hop MAC address towards lwB4,
321 IPv4 Public address and IPv4 Public Port Set.
322 acl cfg /path/to/file.csv Path to CSV file that provides ACL rules.
323 Default value is rules.csv in same directory as configuration file.
324 Fields are SVLAN value & mask, CVLAN value & mask, IP protocol value & mask,
325 source IPv4 subnet (in CIDR notation), destination IPv4 subnet (in CIDR notation),
326 source port range, destination port range, and action (drop, allow, rate limit).
331 Cores can be configured by means of a set of [core #] sections, where # represents either:
333 an absolute core number: e.g. on a 10-core, dual socket system with hyper-threading, cores are numbered from 0 to 39;
334 a core number, the letter 's', and a socket number: this allows selecting per-socket cores, independently from their interleaved numbering;
335 a core number and the letter 'h': this allows selecting the hyper-thread sibling of the specified core;
336 a dash-separated range of core numbers;
337 a comma-separated list of core numbers;
338 any combination of the above;
339 or a variable whose value complies with the above syntax.
340 The socket and hyper-thread syntax makes it easier to use the same configuration file on several platforms,
341 even if their core numbering differs (e.g. interleaving rule or number of cores per socket).
343 Each core can be assigned with a set of tasks, each running one of the implemented packet processing modes.
345 The following parameters are supported:
346 .. image:: images/prox_core.png
348 :alt: SampleVNF supported topology
355 DPDK must be installed prior to running make in the PROX directory.
356 The README file shipped with PROX describes what versions of DPDK are supported,
357 and if any patches are needed for the chosen DPDK version.
359 The following packages need to be installed. (Example for destributions that are using rpm)
362 sudo yum install net-tools wget gcc unzip libpcap-devel ncurses-devel libedit-devel pciutils lua-devel kernel-devel
365 The following instructions are here to help customers to start using PROX.
366 It's by no means a complete guide, for detailed instructions on how to install and use
367 DPDK please refer to its documentation.
368 Your mileage may vary depending on a particular Linux distribution and hardware in use.
370 Edit grub default configuration:
375 Add the following to the kernel boot parameters
378 default_hugepagesz=1G hugepagesz=1G hugepages=8
380 Rebuild grub config and reboot the system:
383 grub2-mkconfig -o /boot/grub2/grub.cfg
386 Verify that hugepages are available
393 Hugepagesize: 1048576 kB
400 umount `awk '/hugetlbfs/ { print $2 }' /proc/mounts` >/dev/null 2>&1
401 mount -t hugetlbfs nodev /mnt/huge/
403 Add the following to the end of ~/.bashrc file
406 export RTE_SDK=/root/dpdk
407 export RTE_TARGET=x86_64-native-linuxapp-gcc
408 export RTE_UNBIND=$RTE_SDK/tools/dpdk_nic_bind.py
410 Re-login or source that file
418 git clone http://dpdk.org/git/dpdk
421 make install T=$RTE_TARGET
426 lsmod | grep -w "^uio" >/dev/null 2>&1 || sudo modprobe uio
432 lsmod | grep -w "^igb_uio" >/dev/null 2>&1 || sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio.ko
434 Discover network devices available on the system:
437 lspci | grep Ethernet
439 Prior launching PROX, ports that are to be used by it must be bound to the igb_uio driver.
441 The following command will bind all Intel® Ethernet Converged Network Adapter X710 ports to igb_uio:
444 lspci | grep X710 | cut -d' ' -f 1 | sudo xargs -I {} python2.7 $RTE_UNBIND --bind=igb_uio {}
446 The following command will bind all Intel® 82599 10 Gigabit Ethernet Controller ports to igb_uio:
449 lspci | grep 82599 | cut -d' ' -f 1 | sudo xargs -I {} python2.7 $RTE_UNBIND --bind=igb_uio {}
451 COMPILING AND RUNNING PROX
452 --------------------------
454 Download and extract the PROX archive
457 wget https://01.org/sites/default/files/downloads/intel-data-plane-performance-demonstrators/dppd-prox-v021.zip
458 unzip dppd-prox-v021.zip
466 The set of sample configuration files can be found in:
471 PROX generation sample configs are in:
476 To launch PROX one may use the following command as an example, assuming the current directory is where you've just built PROX:
479 ./build/prox -f ./config/handle_none.cfg
Code Review / samplevnf.git / blob