1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
3 .. http://creativecommons.org/licenses/by/4.0
11 Ftrace is an internal tracer designed to find what is going on inside the kernel. It can be used
12 for debugging or analyzing latencies and performance issues that take place outside of user-space.
13 Although ftrace is typically considered the function tracer, it is really a frame work of several
14 assorted tracing utilities.
16 One of the most common uses of ftrace is the event tracing.
19 - For KVMFORNFV, Ftrace is preferred as it is in-built kernel tool
20 - More stable compared to other debugging tools
25 +-----------------------------+-----------------------------------------------+
27 | **Release** | **Features** |
29 +=============================+===============================================+
30 | | - Ftrace Debugging tool is not implemented in |
31 | Colorado | Colorado release of KVMFORNFV |
33 +-----------------------------+-----------------------------------------------+
34 | | - Ftrace aids in debugging the KVMFORNFV |
35 | Danube | 4.4-linux-kernel level issues |
36 | | - Option to diable if not required |
37 +-----------------------------+-----------------------------------------------+
40 Implementation of Ftrace
41 -------------------------
42 Ftrace uses the debugfs file system to hold the control files as
43 well as the files to display output.
45 When debugfs is configured into the kernel (which selecting any ftrace
46 option will do) the directory /sys/kernel/debug will be created. To mount
47 this directory, you can add to your /etc/fstab file:
51 debugfs /sys/kernel/debug debugfs defaults 0 0
53 Or you can mount it at run time with:
57 mount -t debugfs nodev /sys/kernel/debug
59 Some configurations for Ftrace are used for other purposes, like finding latency or analyzing the system. For the purpose of debugging, the kernel configuration parameters that should be enabled are:
63 CONFIG_FUNCTION_TRACER=y
64 CONFIG_FUNCTION_GRAPH_TRACER=y
66 CONFIG_DYNAMIC_FTRACE=y
68 The above parameters must be enabled in /boot/config-4.4.0-el7.x86_64 i.e., kernel config file for ftrace to work. If not enabled, change the parameter to ``y`` and run.,
73 grub2-mkconfig -o /boot/grub2/grub.cfg
76 Re-check the parameters after reboot before running ftrace.
80 The below is a list of few major files in Ftrace.
84 This is used to set or display the current tracer that is configured.
86 ``available_tracers:``
88 This holds the different types of tracers that have been compiled into the kernel. The tracers listed here can be configured by echoing their name into current_tracer.
92 This sets or displays whether writing to the tracering buffer is enabled. Echo 0 into this file to disable the tracer or 1 to enable it.
96 This file holds the output of the trace in a human readable format.
100 This is a mask that lets the user only trace on specified CPUs. The format is a hex string representing the CPUs.
104 It holds event tracepoints (also known as static tracepoints) that have been compiled into the kernel. It shows what event tracepoints exist and how they are grouped by system.
110 Here is the list of current tracers that may be configured based on usage.
124 Function call tracer to trace all kernel functions.
128 Similar to the function tracer except that the function tracer probes the functions on their entry whereas the function graph tracer traces on both entry and exit of the functions.
132 This is the "trace nothing" tracer. To remove tracers from tracing simply echo "nop" into current_tracer.
139 To list available tracers:
140 [tracing]# cat available_tracers
141 function_graph function wakeup wakeup_rt preemptoff irqsoff preemptirqsoff nop
144 [tracing]# echo function > current_tracer
145 [tracing]# cat current_tracer
149 [tracing]# cat trace | head -10
152 [tracing]# echo 0 > tracing_on
154 To Start/restart tracing:
155 [tracing]# echo 1 > tracing_on;
161 Ftrace is part of KVMFORNFV D-Release. Kvmfornfv currently uses 4.4-linux-Kernel as part of
162 deployment and runs cyclictest for testing purpose generating latency values (max, min, avg values).
163 Ftrace (or) function tracer is a stable kernel inbuilt debugging tool which tests kernel in real
164 time and outputs a log as part of the code. These output logs are useful in following ways.
167 - Helps in Kernel code Optimization and
168 - Can be used to better understand the kernel Level code flow
169 - Log generation for each test run if enabled
170 - Choice of Disabling and Enabling
172 Ftrace logs for KVMFORNFV can be found `here`_:
175 .. _here: http://artifacts.opnfv.org/kvmfornfv.html
177 Ftrace Usage in KVMFORNFV Kernel Debugging:
178 -------------------------------------------
179 Kvmfornfv has two scripts in /ci/envs to provide ftrace tool:
184 Enabling Ftrace in KVMFORNFV
185 ----------------------------
187 The enable_trace.sh script is triggered by changing ftrace_enable value in test_kvmfornfv.sh script which is zero by default. Change as below to enable Ftrace and trigger the script,
195 - Ftrace is enabled before
197 Details of enable_trace script
198 ------------------------------
200 - CPU Coremask is calculated using getcpumask()
201 - All the required events are enabled by,
202 echoing "1" to $TRACEDIR/events/event_name/enable file
208 $TRACEDIR = /sys/kernel/debug/tracing/
209 sudo bash -c "echo 1 > $TRACEDIR/events/irq/enable"
210 sudo bash -c "echo 1 > $TRACEDIR/events/task/enable"
211 sudo bash -c "echo 1 > $TRACEDIR/events/syscalls/enable"
213 The set_event file contains all the enabled events list
215 - Function tracer is selected. May be changed to other avaliable tracers based on requirement
219 sudo bash -c "echo function > $TRACEDIR/current_tracer
221 - When tracing is turned ON by setting ``tracing_on=1``, the ``trace`` file keeps getting append with the traced data until ``tracing_on=0`` and then ftrace_buffer gets cleared.
231 - Once tracing is diabled, disable_trace.sh script is triggered.
233 Details of Disable_trace Script
234 -------------------------------
235 In disable trace script the following are done:
237 - The trace file is copied and moved to /tmp folfer based on timestamp.
238 - The current tracer file is set to ``nop``
239 - The set_event file is cleared i.e., all the enabled events are disabled
240 - Kernel Ftarcer is diabled/unmounted
243 Publishing Ftrace logs:
244 -----------------------
245 The generated trace log is pushed to `artifacts`_ of Kvmfornfv project by releng team, which is done by a script in JJB of releng. The `trigger`_ in the script is.,
249 echo "Uploading artifacts for future debugging needs...."
250 gsutil cp -r $WORKSPACE/build_output/log-*.tar.gz $GS_LOG_LOCATION > $WORKSPACE/gsutil.log 2>&1
252 .. _artifacts: https://artifacts.opnfv.org/
254 .. _trigger: https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/kvmfornfv/kvmfornfv-upload-artifact.sh;h=56fb4f9c18a83c689a916dc6c85f9e3ddf2479b2;hb=HEAD#l53
257 .. include:: pcm_utility.userguide.rst