Customization of TorizonOS
#TorizonOS #virtualization #qemu #linux #eBPF
During the last weeks, I was tasked with the preparation of the environment for an upcoming project. TorizonOS is going to be at the center of this project, custom drivers will have to be developed for custom hardware. We decided that it will be helpful to run TorizonOS on virtualized hardware.
I was able to compile TorizonOS from source and prepare a filesystem image by adding the recipes to our internal fork of Infrabase https://github.com/EDGEMTech/infrabase/ (a build system which uses bitbake but with our own much simpler layers)
I did get it to boot in about an afternoon - this is mainly because the kernel used by TorizonOS has nearly all virtualization options enabled, support for virtblk, etc
But there was a pesky problem, the setsuid permission bit was missing on the /usr/bin/sudo utility.
At first, I thought that the permission bit might be set by ostree on first boot. Started the investigation by reading the source code, but due to the complexity and shear amount of components - I've decided to write a tool that uses the eBPF facilities of Linux to see what process sets permissions.
I've read about this method before, but never used it, always got away by using the venerable strace utility instead. Unfortunately, strace is not well suited for this scenario mainly because it works on per-process basis. Here I didn't know what performed the setting of the permissions and extended attributes.
The BPF subsystem is powerful it can be used for debugging, securing/auditing and profiling at the kernel level, check the docs here: https://docs.kernel.org/bpf/index.html
By using libbpf-bootstrap https://github.com/libbpf/libbpf-bootstrap and the documentation of libbpf https://docs.ebpf.io/ I came up with a simple program that can trace the chmod(2), fchmod(2), fchmodat(2) and other system calls. The tool was implemented rather quickly and I ran it on my Ubuntu 22.04 development host but it took some time to integrate into TorizonOS Yocto...
Toradex do provide some articles focused on their Yocto distribution https://developer.toradex.com/linux-bsp/os-development/build-yocto/
It's a good start but you're on your own for the rest of the journey - to build that tracing tool - the journey was rather perilous... 😀
Running the tracing tool required the modification of the kernel configuration by applying some specific .scc configuration fragments, understanding how to enable systemd units, compiling the necessary build time dependencies using native recipes. At the end of it, the result is this layer : https://github.com/EDGEMTech/meta-edgemtech-bpf
I basically split up the libraries used by libpbf-bootstrap into single recipes and created a main recipe for the tracing tool 'fperm-trace', there are still some parts that need some 'deeper thinking' - but it's totally usable as-is.
This experience helped us gain a deeper understanding of different components used by TorizonOS and their configuration. In the end, the tracing tool helped to understand that file permissions and extended are not set on first boot... We were able to deploy the file system correctly to the filesystem image. TorizonOS now runs in Qemu and is able to reach 'Greenboot status' and Docker starts correctly.
Stay tuned for more news about the internals of TorizonOS in the future.