eBPF Tetragon Compilation and Debugging Guide
This article provides a introduction to compiling and debugging with the eBPF-based Tetragon tool. It covers configuring Vagrant and VirtualBox, methods for capturing and analyzing eBPF events using Tetragon and Tetra, as well as compiling Tetragon in Docker and debugging in Kubernetes clusters.
Debugging Dependencies
The Tetragon GitHub documentation specifies that debugging is conducted within VirtualBox. Therefore, before starting debugging, you need to install Vagrant and VirtualBox.
The Tetragon source code provides a Vagrant configuration. When starting VirtualBox via Vagrant, it will automatically install all dependencies and tools. For details, refer to the Tetragon source code.
Starting the Virtual Machine
Start and log in to the VirtualBox virtual machine with the following commands:
vagrant up
vagrant sshOn some older versions of Vagrant, vagrant up may throw an error. In such cases, you can try removing the disk configuration and retrying:
diff --git a/Vagrantfile b/Vagrantfile
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -1,6 +1,5 @@
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/impish64"
- config.vm.disk :disk, size: "50GB"
config.vm.provision :dockerLocal Compilation
- First, install the dependencies for LLVM and libbpf:
make tools-install- Then, compile the BPF programs and Go programs:
LD_LIBRARY_PATH=$(realpath ./lib) makeThis command will compile all executable programs and test programs. To improve efficiency during debugging, you can compile only tetra and tetragon:
LD_LIBRARY_PATH=$(realpath ./lib) make tetra tetragonYou will then see the locally generated tetragon and tetra programs. The purpose of these two programs will be explained later.
ls tetra*
tetra tetragon tetragon-aligncheckerLocal Debugging
Starting Tetragon
When starting, you need to specify the path of the eBPF .o files and the Tetragon library:
sudo LD_LIBRARY_PATH=$(realpath ./lib) ./tetragon --bpf-lib bpf/objsOnce Tetragon starts, it will load the eBPF programs and wait for listener connections on the specified port (default is localhost:54321). After a listener connects, Tetragon will forward the detected events to the listener.
The Tetragon startup log looks like this:
time="2022-05-29T10:37:56Z" level=info msg="Starting tetragon" version=v0.8.0-106-g5c3fd60
time="2022-05-29T10:37:56Z" level=info msg="config settings"
config="map[bpf-lib:bpf/objs btf: cilium-bpf: config-dir: config-file: debug:false enable-cilium-api:false enable-export-aggregation:false enable-k8s-api:false enable-process-ancestors:true enable-process-cred:false enable-process-ns:false export-aggregation-buffer-size:10000 export-aggregation-window-size:15s export-allowlist: export-denylist: export-file-compress:false export-file-max-backups:5 export-file-max-size-mb:10 export-file-rotation-interval:0s export-filename: export-rate-limit:-1 force-small-progs:false ignore-missing-progs:false kernel: log-format:text log-level:info metrics-server: netns-dir:/var/run/docker/netns/ process-cache-size:65536 procfs:/proc/ run-standalone:false server-address:localhost:54321 verbose:0]"
time="2022-05-29T10:37:56Z" level=info msg="Available sensors" sensors=
time="2022-05-29T10:37:56Z" level=info msg="Registered tracing sensors" sensors="kprobe sensor, tracepoint sensor"
time="2022-05-29T10:37:56Z" level=info msg="Registered probe types" types="tracepoint sensor, kprobe sensor"
time="2022-05-29T10:37:56Z" level=info msg="Disabling Kubernetes API"
time="2022-05-29T10:37:56Z" level=info msg="Disabling Cilium API"
time="2022-05-29T10:37:56Z" level=info msg="Starting process manager" enableCilium=false enableEventCache=false enableProcessCred=false enableProcessNs=false
time="2022-05-29T10:37:56Z" level=info msg="Exporter configuration" enabled=false fileName=
time="2022-05-29T10:37:56Z" level=info msg="Using metadata file" metadata=
time="2022-05-29T10:37:56Z" level=info msg="Loading sensor" name=__main__
time="2022-05-29T10:37:56Z" level=info msg="Loading kernel version 5.13.19"
time="2022-05-29T10:37:56Z" level=info msg="Starting gRPC server" address="localhost:54321"
time="2022-05-29T10:37:56Z" level=info msg="tetragon, map loaded." map=execve_map path=/sys/fs/bpf/tcpmon/execve_map sensor=__main__
time="2022-05-29T10:37:56Z" level=info msg="tetragon, map loaded." map=execve_map_stats path=/sys/fs/bpf/tcpmon/execve_map_stats sensor=__main__
time="2022-05-29T10:37:56Z" level=info msg="tetragon, map loaded." map=names_map path=/sys/fs/bpf/tcpmon/names_map sensor=__main__
time="2022-05-29T10:37:56Z" level=info msg="tetragon, map loaded." map=tcpmon_map path=/sys/fs/bpf/tcpmon/tcpmon_map sensor=__main__
time="2022-05-29T10:37:56Z" level=info msg="BPF prog was loaded" label=tracepoint/sys_exit prog=bpf/objs/bpf_exit.o
time="2022-05-29T10:37:56Z" level=info msg="BPF prog was loaded" label=kprobe/wake_up_new_task prog=bpf/objs/bpf_fork.o
time="2022-05-29T10:37:56Z" level=info msg="Load probe" Program=bpf/objs/bpf_execve_event_v53.o Type=execve
time="2022-05-29T10:37:57Z" level=info msg="Read ProcFS /proc/ appended 74/218 entries"
time="2022-05-29T10:37:57Z" level=warning msg="Procfs execve event pods/ identifier error" error="open /proc/0/cgroup: no such file or directory"
time="2022-05-29T10:37:57Z" level=info msg="BPF prog was loaded" label=tracepoint/sys_execve prog=bpf/objs/bpf_execve_event_v53.o
time="2022-05-29T10:37:57Z" level=info msg="Loaded BPF maps and events for sensor successfully" sensor=__main__
time="2022-05-29T10:37:57Z" level=info msg="Listening for events..."From the log, you can see the Tetragon startup config parameters (second line), the Tetragon version, and more.
Changing the Default Log Level
From the config parameters, you can see that the default log level is info. To change the log level to debug, add the –log-level=debug parameter at startup:
sudo LD_LIBRARY_PATH=$(realpath ./lib) ./tetragon --bpf-lib bpf/objs --log-level=debugStarting the Listener: Tetra
Tetra is a CLI debugging tool for Tetragon. You can see the specific parameters and usage of Tetra by directly running tetra:
Tetragon CLI
Usage:
tetra [flags]
tetra [command]
Available Commands:
bugtool Produce a tar archive with debug information
getevents Print events
help Help about any command
sensors Manage sensors
stacktrace-tree Manage stacktrace trees
status Print health status
tracingpolicy Manage tracing policies
version Print version
Flags:
-d, --debug Enable debug messages
-h, --help help for tetra
--server-address string gRPC server address (default "localhost:54321")
Use "tetra [command] --help" for more information about a command.You can retrieve eBPF events sent by Tetragon using tetra getevents. By default, Tetra outputs in JSON format, for example:
$ ./tetra getevents | jq "."
{
"process_exec": {
"process": {
"exec_id": "OjI0Njk4Mjk4ODYxNDI6MTI0NDE=",
"pid": 12441,
"uid": 0,
"cwd": "/run/containerd/io.containerd.runtime.v2.task/k8s.io/25ecffd7e2ee833292350
0bd930eaebe2725bb643e7e9e2206b611f8666abac0/",
"binary": "/usr/local/sbin/runc",
"arguments": "--root /run/containerd/runc/k8s.io --log /run/containerd/io.containerd.runtime.v2.task/k8s.io/c7c9f733b618c7b46f0bd86e054fc5d940fbd7eb627cf009d6d9cd0584c846a6/log.json --log-format json exec --process /tmp/runc-process789407863 --detach --pid-file /run/containerd/io.containerd.runtime.v2.task/k8s.io/c7c9f733b618c7b46f0bd86e054fc5d940fbd7eb627cf009d6d9cd0584c846a6/4c9ea3805e0d2a23de4c0cdb983c52db832689aa821d3363ba8954b1b76359d9.pid c7c9f733b618c7b46f0bd86e054fc5d940fbd7eb627cf009d6d9cd0584c846a6",
"flags": "execve clone",
"start_time": "2022-05-29T10:54:19.452Z",
"auid": 4294967295,
"parent_exec_id": "OjMzMzgwMDAwMDAwOjI0NzM=",
"refcnt": 1
},
"parent": {
"exec_id": "OjMzMzgwMDAwMDAwOjI0NzM=",
"pid": 2473,
"uid": 0,
"cwd": "/run/containerd/io.containerd.runtime.v2.task/k8s.io/25ecffd7e2ee8332923500bd930eaebe2725bb643e7e9e2206b611f8666abac0",
"binary": "/usr/local/bin/containerd-shim-runc-v2",
"arguments": "-namespace k8s.io -id 25ecffd7e2ee8332923500bd930eaebe2725bb643e7e9e2206b611f8666abac0 -address /run/containerd/containerd.sock",
"flags": "procFS auid",
"start_time": "2022-05-29T10:13:43.002Z",
"auid": 0,
"parent_exec_id": "OjIyODIwMDAwMDAwOjExOTE=",
"refcnt": 4294967265
}
},
"time": "2022-05-29T10:54:19.452Z"
}Using tetra getevents –output compact provides a more user-friendly output:
./tetra getevents --output compact
🚀 process /usr/sbin/iptables -w 5 -W 100000 -S KUBE-KUBELET-CANARY -t mangle
💥 exit /usr/sbin/iptables -w 5 -W 100000 -S KUBE-KUBELET-CANARY -t mangle 0
🚀 process /usr/sbin/ip6tables -w 5 -W 100000 -S KUBE-KUBELET-CANARY -t mangle
💥 exit /usr/sbin/ip6tables -w 5 -W 100000 -S KUBE-KUBELET-CANARY -t mangle 0 Compiling in Docker
To compile in Docker, use the following commands. By default, Docker images will be packaged. Refer to the Makefile for details.
# Build Tetragon agent and operator images
LD_LIBRARY_PATH=$(realpath ./lib) make LOCAL_CLANG=0 image image-operator
# Bootstrap the cluster
contrib/localdev/bootstrap-kind-cluster.sh
# Install Tetragon
contrib/localdev/install-tetragon.sh --image cilium/tetragon:latest --operator cilium/tetragon-operator:latestThis process may take some time because it will install the kind tool and set up a single-node Kubernetes cluster using kind.
After compiling, Tetragon will be deployed in the Kubernetes cluster. You can check this with the kubectl command:
kubectl get pods -n kube-systemDebugging in the Kubernetes Cluster
After deploying Tetragon in the Kubernetes cluster, you can check the event reports with the following command. The output log will be quite extensive (in JSON format), and you can use the jq command for log filtering.
kubectl logs -n kube-system ds/tetragon -c export-stdout -freference tetragon.
