The static tracing functionality is exposed via the debugfs filesystem. The functionality offered by the interface includes:

Listing events

You can see a list of available tracepoints exposed via sysfs and ordered in subsystem directories with the following:

# ls /sys/kernel/debug/tracing/events/
    
asoc                 ipi                  regmap
    
block                irq                  regulator
cfg80211             jbd                  rpm
clk                  jbd2                 sched
cma                  kmem                 scsi
coda                 libata               signal
compaction           mac80211             skb
cpufreq_interactive  migrate              sock
drm                  module               spi
enable               napi                 sunrpc
ext3                 net                  swiotlb
ext4                 nfs                  task
fence                nfs4                 thermal
filelock             oom                  timer
filemap              pagemap              udp
ftrace               power                v4l2
gpio                 printk               vmscan
header_event         random               workqueue
header_page          raw_syscalls         writeback
i2c                  rcu
  

Or in the available_events file with the <subsystem>:<event> format using the following commands:

#  grep 'net'  /sys/kernel/debug/tracing/available_events
net:netif_rx_ni_entry
net:netif_rx_entry
net:netif_receive_skb_entry
net:napi_gro_receive_entry
net:napi_gro_frags_entry
net:netif_rx
net:netif_receive_skb
net:net_dev_queue
net:net_dev_xmit
net:net_dev_start_xmit  

Describing events

Each event has a specific printing format that describes the information included in the log event, as follows:

# cat /sys/kernel/debug/tracing/events/net/netif_receive_skb/format
name: netif_receive_skb
ID: 579
format:
    field:unsigned short common_type;       offset:0;       size:2;  signed:0;
    field:unsigned char common_flags;       offset:2;       size:1; signed:0;
    field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
    field:int common_pid;   offset:4;       size:4; signed:1;
    
    field:void * skbaddr;   offset:8;       size:4; signed:0;
    field:unsigned int len; offset:12;      size:4; signed:0;
    field:__data_loc char[] name;   offset:16;      size:4; signed:0;
    
print fmt: "dev=%s skbaddr=%p len=%u", __get_str(name), REC->skbaddr, REC->len
 

Enabling and disabling events

 You can enable or disable events in the following ways:

• By echoing 0 or 1 to the event enable file:

# echo 1 > 
/sys/kernel/debug/tracing/events/net/netif_receive_skb/enable  

• By subsystem directory, which will enable or disable all the tracepoints in the directory/subsystem:

# echo 1 > /sys/kernel/debug/tracing/events/net/enable  

• By echoing the unique tracepoint name into the set_event file:

# echo netif_receive_skb >> 
/sys/kernel/debug/tracing/set_event  

Note the append operation >> is used not to clear events.

• Events can be disabled by appending an exclamation mark to their names:

# echo '!netif_receive_skb' >> 
/sys/kernel/debug/tracing/set_event  

• Events can also be enabled/disabled by subsystem:

# echo 'net:*' > /sys/kernel/debug/tracing/set_event 

• To disable all events:

# echo > /sys/kernel/debug/tracing/set_event  

• You can also enable tracepoints from boot by passing a trace_event=<comma separated event list> kernel command line-argument.

Adding events to the tracing buffer

To see the tracepoints appear on the tracing buffer, turn tracing on:

# echo 1 > /sys/kernel/debug/tracing/tracing_on  

Tracepoint events are integrated into the ftrace subsystem so that if you enable a tracepoint when a tracer is running, it will show up in the trace. This behavior is shown in the following commands:

# cd /sys/kernel/debug/tracing
# echo 1 > events/net/netif_receive_skb/enable
# echo netif_receive_* > set_ftrace_filter
# echo function > current_tracer
# cat trace
          <idle>-0     [000] ..s2    66.498472: netif_receive_skb_internal <-napi_gro_receive
          <idle>-0     [000] ..s2    66.498483: netif_receive_skb: dev=eth0 skbaddr=d86c2600 len=328