ftrace

Trace Compass supports the raw textual format of various ftrace plugins.

Supported ftrace features

Event tracing

ftrace traces with kernel events are kernel traces and all analyses available with other kernel traces in Trace Compass are made available for those traces: Kernel resources and threads analyses, I/O, memory, critical path, etc.

function graph

The function graph plugin of trace allows to record the function entry and exits of various kernel functions. With this kind of trace, we can obtain a callstack / flame chart / flame graph for the various threads on the machine.

Generating a trace

There are two ways to generate the trace in the human readable raw format. Using ftrace via the debugfs filesystem or using the trace-cmd command-line tool.

Debugfs filesystem

Mount the debugfs filesystem using the following command:

 # mount -t debugfs nodev /sys/kernel/debug

The filesystem could be mounted elsewhere.

Before starting the tracer, traced events must be enabled by echoing 1 in the correct files. For example, enabling sched_switch, sched_wakeup and syscall tracing is done with the following commands:

 # echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
 # echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
 # echo 1 > /sys/kernel/debug/tracing/events/syscalls/enable

Recording of a trace is started by echoing 1 in tracing_on:

 # echo 1 > /sys/kernel/debug/tracing/tracing_on

Recording is stopped by echoing 0 in the same file. The trace can be obtained in raw format in the trace file:

 # cat /sys/kernel/debug/tracing/trace > myFtraceFile.txt

trace-cmd

Recording of a trace can be started by the following command:

 # trace-cmd record -e sched_switch -e sched_wakeup

The recorded trace would contain only sched_switch and sched_wakeup events. Outputting the recorded trace is done using the following command:

 # trace-cmd report -R

The -R argument is needed to get the raw format.

To obtain a function graph with ftrace, tracing can be done with the following command. As the function graph can be quite big, using the -F flag allows to trace the calls only for the command being traced, here ls. The resulting trace will contain symbols. To resolve the symbols, the kernel symbols should be exported from the command line and configured in Trace Compass.

 # sudo trace-cmd record -p function_graph -F ls
 # sudo trace-cmd report -R
 # cat /proc/kallsyms > kallsyms

See the trace-cmd documentation for more tracing options.