Once we know how to enable tracing, let's explore how to make use of it:

1. You can look at the captured trace in human-readable format via the trace and trace_pipe files, with the latter blocking on read and consuming the data.
2. The function tracer provides the following output:

$ cat /sys/kernel/debug/tracing/trace_pipe
sh-394 [003] ...1 46.205203: mutex_unlock <-
 tracing_set_tracer
sh-394 [003] ...1 46.205215: __fsnotify_parent <-
vfs_write
sh-394 [003] ...1 46.205218: fsnotify <-vfs_write
sh-394 [003] ...1 46.205220: __srcu_read_lock <-
 fsnotify
sh-394 [003] ...1 46.205223: preempt_count_add <-
 __srcu_read_lock
sh-394 [003] ...2 46.205226: preempt_count_sub <-
 __srcu_read_lock
sh-394 [003] ...1 46.205229: __srcu_read_unlock <-
 fsnotify
sh-394 [003] ...1 46.205232: __sb_end_write <-
 vfs_write
sh-394 [003] ...1 46.205235: preempt_count_add <-
 __percpu_counter_add
sh-394 [003] ...2 46.205238: preempt_count_sub <-
 __percpu_counter_add
sh-394 [003] d..1 46.205247: gic_handle_irq <-
 __irq_usr
<idle>-0 [002] d..2 46.205247: ktime_get <-
 cpuidle_enter_state

The format for the function tracer output is as follows:

task-PID [cpu-nr] irqs-off need-resched hard/softirq preempt-depth 
delay-timestamp function  

4. The graphical function tracer output is as follows:

$ cat /sys/kernel/debug/tracing/trace_pipe
3)   ==========> |
3)               |  gic_handle_irq() {
2)   ==========> |
2)               |  gic_handle_irq() {
3)   0.637 us    |    irq_find_mapping();
2)   0.712 us    |    irq_find_mapping();
3)               |    handle_IRQ() {
2)               |    handle_IRQ() {
3)               |      irq_enter() {
2)               |      irq_enter() {
3)   0.652 us    |        rcu_irq_enter();
2)   0.666 us    |        rcu_irq_enter();
3)   0.591 us    |        preempt_count_add();
2)   0.606 us    |        preempt_count_add();  

The format for the graphical function tracer output is as follows:

cpu-nr) timestamp | functions