Fetching the kernel over HTTP with U-Boot on the RPI3

To make a long story short when working with the Raspberry Pi 3 constantly removing and inserting the SD card is not practical.

A good solution is network booting, hosting the root filesystem and kernel on the development host and retrieving them over the network - the SD card will only contain a bootloader.

This provides an effective way to switch between different kernels and also allows work with various user space environments: GNU, busybox/buildroot, alpine/busybox, without touching the SD card.

Another benefit is the improvement of the speed of the 'build, deploy, test, fix' cycle. It's not the first time that I am configuring such a setup on a RaspberryPi 3. Only this time, I am documenting it.. 😀

This article is more of a development journal - and will be split into multiple parts. The first part explains how to configure U-Boot to fetch Linux over HTTP, subsequent parts will focus on how to configure the kernel to mount a remote root filesystem. Both the kernel and the rootfs will be hosted on my development host - which is my work laptop running Ubuntu Jammy 22.04

U-Boot to the rescue

By default, the Raspberry Pi 3 uses a custom bootloader and it is perfectly sufficient for most use cases. However, this use case requires a more 'feature rich' bootloader because the kernel needs to be fetched over the network. U-Boot is a popular bootloader that supports many architectures, it can fetch a kernel image over the network. Currently only HTTP and TFTP protocols are supported, HTTP support being the most recent addition. It was added back in 2022 and there has been many fixes since then. On the Rpi 3 U-Boot acts as a third-stage bootloader, the first stage, which is provided by the vendor lives in the bootcode.bin file on the SD Card, the bootloader then runs start.elf which reads config.txt and the device tree binary file, control is then given to U-Boot.

Setting up the project directory:

mkdir -p ~/rpi3/{dependencies,etc,volume,out}
cd ~/rpi3/
git init
git submodule add --depth 1 https://github.com/raspberrypi/firmware dependencies/rpi_firmware/src
git submodule add --depth 1 https://github.com/u-boot/u-boot dependencies/u-boot/src

mkdir creates the file hierarchy for this project. Then git downloads the pre-compiled binaries of the current Raspberry Pi Linux kernel, kernel modules, compiled device trees, user space libraries, and the bootloader/GPU firmware and also U-Boot. Submodules can be used as a practical way to manage dependencies.

U-boot

Nowadays, configuring U-Boot for the Raspberry Pi 3 B plus is easy. Be sure to install the required build dependencies, by following the official documentation https://docs.u-boot.org/en/latest/build/gcc.html#configuration

Copy-paste the 'apt-get' commands from the documentation, it will install the GCC compiler for the aarch64 architecture and the necessary libraries required to compile U-Boot.

The next step is configuring U-Boot:

cd rpi3/dependencies/uboot
make rpi_3_b_plus_defconfig

This generates a .config file based on the default configuration for the Raspberry Pi 3 B+. If you're using another model, use: du -a configs | grep rpi_ to list the available default configurations for the various models of the Raspberry Pi.

Fetching the kernel over HTTP requires the use of the 'wget' monitor command, it can be enabled by setting the CMD_WGET Kconfig option.

This can be done with 'menuconfig', a ncurses configuration tool, by typing 'make menuconfig' and selecting the correct options, the .config file will be modified accordingly.

Save the generated '.config' somewhere out of code tree:

cp .config ~/rpi3/etc/uboot_rpi_3_b_plus.config

Now that U-Boot is configured, simply run make to build it: CROSS_COMPILE=aarch64-linux-gnu- make all

The compiler generates a uboot.bin file in the current directory

Preparing the SD-Card

Most SD cards are pre-formatted with a single FAT-32 partition, normally the Raspberry Pi 3 uses a two partition setup, one 512 Mb FAT32 boot partition and a EXT4 partition for the rootfs. Adjusting the partition layout can be achieved with the fdisk(8) utility: sudo fdisk -l

This will output the list of available disk drives, ignore the '/dev/loopXX' devices of which there are many on Ubuntu... USB card readers usually use '/dev/sdX' so one can filter the output: sudo fdisk -l | grep sd.:

If your system has SATA disks, locate the SD card by looking at the size.

sudo fdisk /dev/sdd starts fdisk(8) in interactive mode - to manipulate the partition table of the disk use the following commands:

d 1
n 1
p
1
2048
+512M
t
0b
w

It removes the first partition, creates another primary partition of 512 Mb, sets the type to FAT32 and writes the changes to the disk and exists. Check if the partition table is correct by running fdisk -l /dev/sdd this use case requires a single FAT32 partition, sfdisk(8) can be used instead to achieve the above result from a script or build system.

Next, format the partition by running: sudo mkfs.fat /dev/sdd1, mount the drive with: sudo mount /dev/sdd1 volume and copy the necessary files to the SD card:

sudo cp -r dependencies/rpi_firmware/src/boot/{bootcode.bin,start.elf,fixup.dat,overlays,bcm2710-rpi-3-b-plus.dtb} volume
sudo cp -r dependencies/u-boot/src/u-boot.bin volume

Create the config.txt file which is read and parsed by start.elf secondary stage bootloader:

cat > etc/bootloader_config.txt <<heredoc
kernel=u-boot.bin
arm64_bit=1
core_freq=250
device_tree=bcm2710-rpi-3-b-plus.dtb
heredoc

Copy the configuration to the SD Card: sudo cp etc/bootloader_config.txt volume/config.txt

To understand the startup sequence and the contents of config.txt, I found these notes very useful: https://github.com/mhomran/u-boot-rpi3-b-plus

The next step is to create a script for U-Boot to avoid typing monitor commands on the keyboard on each boot:

cat > etc/uboot_script.txt <<heredoc
setenv autoload 0
dhcp
setenv serverip 172.22.22.57
wget \${kernel_addr_r} /kernel/linux5.10_rpi
booti \${kernel_addr_r} - \${fdt_addr}
heredoc

This script interrupts the normal loading sequence, acquires an address for the Ethernet card using the DHCP client. It then performs a HTTP GET request to fetch the Linux image and store at the address specified by 'kernel_addr_r'.

The request URL will look like this: 'http://172.22.22.57/linux5.10_rpi'

Compile the script and copy it to the SD card:

./dependencies/u-boot/src/tools/mkimage -T script -d etc/uboot_script boot.scr
sudo cp boot.scr volume/

Finally, un-mount the volume: sudo umount volume

Plug in the Ethernet cable to the Raspberry Pi and insert the card - it will attempt to fetch the kernel and fail.

Server setup

At this stage, we have a working U-boot setup that makes use of the 'wget' monitor command to fetch the kernel and then boot it with the 'booti' command great! But a HTTP server needs to serve the kernel when the bootloader makes a HTTP GET request.

Servers are supposed to have a static IP address, it would be wise to make a DHCP reservation. Setting a static IP on the Ethernet interface used by the web server is another option, To avoid 'hard to debug' conflicts it must be an IP address outside of the DHCP range of the router you're using. Both solutions have a major disadvantage, the U-Boot script will have to be modified when working in a different location. Because the network configuration will most certainly be different from one location to the next.

A solution that doesn't require patching U-Boot, is to use a dedicated USB network adapter with a DHCP server listening on it. On Ubuntu, this requires a deep dive into 'systemd' and 'systemd-networkd' and then choosing the right DHCP server and configuring it... Another solution could be passing the IP address as a DHCP option, is it possible without patching U-Boot ? - I Need to investigate this by checking the source code/documentation, so it will be the focus of a subsequent article.

For this guide - I will use NGINX, a famous HTTP server that I am familiar with. It can be installed like so: sudo apt-get install nginx

I also like to disable the 'systemd' unit file for NGINX sudo systemctl disable nginx because I tend to run NGINX on-demand, by hand or from a script. It is also possible to run multiple instances of NGINX but this requires to specify a custom HTTP port in the configuration

Create the NGINX config file:

cat > etc/nginx_rpi.conf <<heredoc
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
   worker_connections 768;
}
http {
   sendfile on;
   tcp_nopush on;
   types_hash_max_size 2048;
   include /etc/nginx/mime.types;
   default_type application/octet-stream;
   access_log /var/log/nginx/rpi_access.log;
   error_log /var/log/nginx/rpi_error.log;
   server {
       listen 80 default_server;
       root /home/etag/rpi3/out/kernel;
       server_name _;
       location /kernel {
           try_files \$uri \$uri/ =404;
       }
   }
}
heredoc

A good idea is to create the webroot in the project file hierarchy, serving the contents of the 'out/kernel' directory. Create the directory: mkdir -p out/kernel and then Start NGINX: sudo nginx -p . -c etc/nginx_config.conf

It will start NGINX with the custom configuration - it requires setting the prefix with '-p' option. Reboot the Pi, while running: tail -f /var/log/nginx/rpi_access.log

You should see a 'GET /kernel/linux5.10_rpi' request being logged.

Conclusion

To integrate this into an existing build system, scripts will need to be written but before writing them we need to make it work on other networks without modifying the U-Boot boot script each time..

The next article will focus on configuring and compiling the Raspberry Pi stable kernel, to make it mount the rootfs over the network - so stay tuned!