Using LTTng

The Linux Trace Toolkit project was started by Karim Yaghmour as a means of tracing kernel activity and was one of the first trace tools generally available for the Linux kernel. Later, Mathieu Desnoyers took up the idea and re-implemented it as the next generation trace tool, LTTng. It was then expanded to cover user space traces as well as the kernel. The project website is at http://lttng.org/ and contains a comprehensive user manual.

LTTng consists of three components:

In addition to those, you will need a trace viewer such as Babeltrace (http://www.efficios.com/babeltrace) or the Eclipse Trace Compaas plug-in to display and filter the raw trace data on the host or target.

LTTng requires a kernel configured with CONFIG_TRACEPOINTS, which is enabled when you select Kernel hacking | Tracers | Kernel Function Tracer.

The description that follows refers to LTTng version 2.5; other versions may be different.

LTTng and the Yocto Project

You need to add these packages to the target dependencies, for example, in conf/local.conf:

IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust"

If you want to run Babeltrace on the target, also append the package babeltrace.

LTTng and Buildroot

You need to enable the following:

For user space trace tracing, enable this:

There is a package called lttng-babletrace for the target. Buildroot builds the host babeltrace automatically and places in output/host/usr/bin/babeltrace.

Using LTTng for kernel tracing

LTTng can use the set of ftrace events described above as potential trace points. Initially, they are disabled.

The control interface for LTTng is the lttng command. You can list the kernel probes using the following:

# lttng list --kernel
Kernel events:
-------------
      writeback_nothread (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      writeback_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      writeback_exec (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[...]

Traces are captured in the context of a session which, in this example, is called test:

# lttng create test
Session test created.
Traces will be written in /home/root/lttng-traces/test-20150824-140942
# lttng list
Available tracing sessions:
  1) test (/home/root/lttng-traces/test-20150824-140942) [inactive]

Now enable a few events in the current session. You can enable all kernel tracepoints using the --all option but remember the warning about generating too much trace data. Let's start with a couple of scheduler-related trace events:

# lttng enable-event --kernel sched_switch,sched_process_fork

Check that everything is set up:

# lttng list test
Tracing session test: [inactive]
    Trace path: /home/root/lttng-traces/test-20150824-140942
    Live timer interval (usec): 0

=== Domain: Kernel ===

Channels:
-------------
- channel0: [enabled]

    Attributes:
      overwrite mode: 0
      subbufers size: 26214
      number of subbufers: 4
      switch timer interval: 0
      read timer interval: 200000
      trace file count: 0
      trace file size (bytes): 0
      output: splice()

    Events:
      sched_process_fork (loglevel: TRACE_EMERG (0)) (type: tracepoint) [enabled]
      sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint) [enabled]

Now start tracing:

# lttng start

Run the test load and then stop tracing:

# lttng stop

Traces for the session are written to the session directory, lttng-traces/<session>/kernel.

You can use the Babeltrace viewer to dump the raw trace data in text format, in this case, I ran it on the host computer:

$ babeltrace  lttng-traces/test-20150824-140942/kernel

The output is too verbose to fit on this page, so I will leave it as an exercise for you, the reader, to capture and display a trace in this way. The text output from eBabeltrace does have the advantage that it is easy to search for strings using grep and similar commands.

A good choice for a graphical trace viewer is the Trace Compass plug-in for Eclipse, which is now part of the Eclipse IDE for C/C++ Developers bundle. Importing the trace data into Eclipse is characteristically fiddly. Briefly, you need to follow these steps:

  1. Open the tracing perspective.
  2. Create a new project by selecting File | New | Tracing project.
  3. Enter a project name and click Finish.
  4. Right-click on the New Project option in the Project Explorer menu and select Import.
  5. Expand Tracing and then select Trace Import.
  6. Browse to the directory containing the traces (for example, test-20150824-140942), tick the box to indicate which sub-directories you want (it might be the kernel) and click Finish.
  7. Now, expand the project and, within that, expand Traces[1] and, within that, double-click on kernel.
  8. You should see the trace data shown in the following screenshot:
    Using LTTng for kernel tracing

In the preceding screenshot, I have zoomed in on the control flow view to show state transitions between dropbear and a shell, and also some activity of the lttng daemon.