Hello Embedded World – booting a minimal Linux with Busybox on RISC-V, from source

hello-embedded-world-–-booting-a-minimal-linux-with-busybox-on-risc-v,-from-source

Last time we saw how to boot Ubuntu for RISC-V on the QEMU virt board and set up a development environment for C and RISC-V assembly. That was fun and all, but nothing compares to compiling our own Linux kernel and userspace utilities and get that to boot on a virtual (or physical) RISC-V board. So we’re gonna do it today!

The RISC-V docs actually outlines a process for doing so, but unfortunately, it is very brief and skips a lot of details and thus may not be suitable for readers new to the embedded world. At least it took me a lot of fumbling, extra Googling and experimentation to finally get it working. This article thus attempts to bridge the gaps in the official docs so it would be easier to follow for a newcomer. Let’s go!

Preparation

A proper Linux environment. If on Windows / macOS, run a full-blown Linux VM with a hypervisor like VirtualBox or VMware. WSL2 on Windows may or may not work, and will not be supported in this article.

The reference distribution is Ubuntu 22.04. You may have to adapt the instructions and commands accordingly if on another Linux distribution. Or for minimal hassle, run an Ubuntu VM anyway.

It is assumed you are already well-versed in Linux commands and administration. If you get an error like gpg2: command not found halfway, you’ll be expected to figure out to sudo apt install gnupg2 instead of complaining 😉

Setting up the host, cross-compiling Linux and BusyBox for RISC-V

Main article: Running 64- and 32-bit RISC-V Linux on QEMU

Refresh repository metadata:

$ sudo apt update

Install build dependencies for Linux and BusyBox:

$ sudo apt install autoconf automake autotools-dev curl libmpc-dev libmpfr-dev libgmp-dev 
                 gawk build-essential bison flex texinfo gperf libtool patchutils bc 
                 zlib1g-dev libexpat-dev git

You’ll also need to install qemu-system for emulating the RISC-V virt board:

$ sudo apt install -y qemu-system

Linux

Head over to kernel.org and download the latest stable kernel, or any other sufficiently recent kernel of your choosing. For example, at the time of writing (2022-08-14), the latest stable kernel is 5.19.1:

$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.19.1.tar.xz

Fetch also the corresponding kernel signature. For something as important as an OS kernel, it’s best to verify its authenticity and integrity:

$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.19.1.tar.sign

Decompress the kernel tarball (but do not extract the archive yet):

$ unxz linux-5.19.1.tar.xz

https://kernel.org/category/signatures.html explains how to verify the signature of the kernel tarball. First install gnupg2:

$ sudo apt install gnupg2

Now import the keys for Linus Torvalds and Greg Kroah-Hartman, creator of Linux and lead kernel developer (respectively):

$ gpg2 --locate-keys torvalds@kernel.org gregkh@kernel.org

Trust the imported keys. Replace the hashes shown below based on the output you got from the previous command:

$ gpg2 --tofu-policy good 38DBBDC86092693E
$ gpg2 --tofu-policy good 79BE3E4300411886

Now verify:

$ gpg2 --trust-model tofu --verify linux-5.19.1.tar.sign

Expected output:

gpg: assuming signed data in 'linux-5.19.1.tar'
gpg: Signature made Thu Aug 11 11:22:54 2022 UTC
gpg:                using RSA key 647F28654894E3BD457199BE38DBBDC86092693E
gpg: Good signature from "Greg Kroah-Hartman " [full]
gpg: gregkh@kernel.org: Verified 1 signatures in the past 0 seconds.  Encrypted
     0 messages.

Now unpack the tarball:

$ tar xvf linux-5.19.1.tar

And enter the source tree:

$ pushd linux-5.19.1/

In order to cross-compile for RISC-V, we need a cross-compiler. Install gcc-riscv64-linux-gnu:

$ sudo apt install -y gcc-riscv64-linux-gnu

Now configure the kernel for RISC-V:

$ make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- defconfig

And build it (this can take a while):

$ make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- -j$(nproc)

Now we can leave the source tree:

$ popd

BusyBox

Head over to busybox.net for the BusyBox source code. The latest release at the time of writing (2022-08-14) is 1.35.0.

Fetch the compressed tarball:

$ wget https://busybox.net/downloads/busybox-1.35.0.tar.bz2

And the SHA256 hash, to verify its integrity:

$ wget https://busybox.net/downloads/busybox-1.35.0.tar.bz2.sha256

There also appears to be a signature file for verifying the tarball signature, but we’ll not cover it here.

Verify the checksum:

$ sha256sum -c busybox-1.35.0.tar.bz2.sha256

Expected output:

busybox-1.35.0.tar.bz2: OK

Unpack the archive:

$ tar xvf busybox-1.35.0.tar.bz2

Now enter the source tree:

$ pushd busybox-1.35.0/

Configure and build for RISC-V – make sure the resulting binary is statically linked:

$ CROSS_COMPILE=riscv64-linux-gnu- LDFLAGS=--static make defconfig
$ CROSS_COMPILE=riscv64-linux-gnu- LDFLAGS=--static make -j$(nproc)

We can now leave the source tree:

$ popd

Preparing the virtual disk, rootfs

Before we can boot our virt board, we need to prepare a disk image with a root filesystem (rootfs). The rootfs will mainly be provided by BusyBox, though we’ll need to create a few additional directories for mount points, startup scripts and the like.

The simplest way to do so is with dd – let’s make a virtual disk image busybox 1GB in size:

$ dd if=/dev/zero of=busybox bs=1M count=1024

Format it with ext4 filesystem (or another supported filesystem of your choice):

$ mkfs.ext4 busybox

Create a mount point rootfs:

$ mkdir -p rootfs

Now mount our virtual disk on our newly created mount directory:

$ sudo mount busybox rootfs

We can now install Busybox on this rootfs:

$ sudo CROSS_COMPILE=riscv64-linux-gnu- LDFLAGS=--static make -C busybox-1.35.0/ install CONFIG_PREFIX=../rootfs

Create a few directories for mounting key filesystems like procfs, sysfs and devtmpfs for BusyBox to boot correctly:

$ sudo mkdir -p rootfs/proc rootfs/sys rootfs/dev

Make sure /etc/fstab exists to silence a warning on poweroff:

$ sudo touch rootfs/etc/fstab

Create a directory /etc/init.d for startup scripts:

$ sudo mkdir -p rootfs/etc/init.d

BusyBox runs a script /etc/init.d/rcS on system startup. Let’s fill it in and make it executable:

$ sudo bash -c "cat > rootfs/etc/init.d/rcS" << EOF
#!/bin/sh

echo "Hello Embedded World!"
echo "Hello RISC-V World!"
mount -t proc proc /proc
mount -t sysfs sysfs /sys
ip addr add 10.0.2.15/24 dev eth0
ip link set dev eth0 up
ip route add default via 10.0.2.2 dev eth0
EOF
$ sudo chmod +x rootfs/etc/init.d/rcS

Unmount our virtual disk:

$ sudo umount rootfs

Now onto the exciting stuff!

Booting our RISC-V virt board

Let’s run our emulator:

$ qemu-system-riscv64 
    -nographic 
    -machine virt 
    -kernel linux-5.19.1/arch/riscv/boot/Image 
    -append "root=/dev/vda ro console=ttyS0" 
    -drive file=busybox,format=raw,id=hd0 
    -device virtio-blk-device,drive=hd0 
    -netdev user,id=eth0 
    -device virtio-net-device,netdev=eth0

Most of the options will look familiar to you if you followed our last article, so we’ll just cover what’s new:

  • -kernel linux-5.19.1/arch/riscv/boot/Image: remember last time we specified the bootloader (Das U-Boot) here? This time we specify an actual kernel image, the one we just built. This skips the bootloader stage straight to the kernel, also know as direct kernel boot. It’s required in this case since our BusyBox rootfs is not a bootable image
  • -append "root=/dev/vda ro console=ttyS0": here we append some kernel command-line options. For example, our rootfs at /dev/vda (as seen from within the VM) is mounted read-only (ro), and we specify the console

Since the BusyBox userspace is extremely lightweight, it should boot fully within about 1 second. Here’s what you should see:

...
Hello Embedded World!
Hello RISC-V World!

Please press Enter to activate this console.

Press Enter as prompted. You should drop into a root shell.

Let’s play around. View the list of running processes:

# ps aux

Example output:

PID   USER     TIME  COMMAND
    1 0         0:00 init
    2 0         0:00 [kthreadd]
    3 0         0:00 [rcu_gp]
    4 0         0:00 [rcu_par_gp]
    5 0         0:00 [netns]
    6 0         0:00 [kworker/0:0-eve]
    7 0         0:00 [kworker/0:0H-ev]
    8 0         0:00 [kworker/u2:0-ev]
    9 0         0:00 [mm_percpu_wq]
   10 0         0:00 [rcu_tasks_trace]
   11 0         0:00 [ksoftirqd/0]
   12 0         0:00 [rcu_sched]
   13 0         0:00 [migration/0]
   14 0         0:00 [kworker/0:1-eve]
   15 0         0:00 [cpuhp/0]
   16 0         0:00 [kdevtmpfs]
   17 0         0:00 [inet_frag_wq]
   18 0         0:00 [khungtaskd]
   19 0         0:00 [oom_reaper]
   20 0         0:00 [writeback]
   21 0         0:00 [kcompactd0]
   22 0         0:00 [kblockd]
   23 0         0:00 [ata_sff]
   24 0         0:00 [rpciod]
   25 0         0:00 [kworker/0:1H-ev]
   26 0         0:00 [xprtiod]
   27 0         0:00 [kswapd0]
   28 0         0:00 [kworker/u2:1-ev]
   29 0         0:00 [nfsiod]
   30 0         0:00 [uas]
   31 0         0:00 [mld]
   32 0         0:00 [ipv6_addrconf]
   39 0         0:00 [jbd2/vda-8]
   40 0         0:00 [ext4-rsv-conver]
   48 0         0:00 -/bin/sh
   49 0         0:00 init
   50 0         0:00 init
   51 0         0:00 init
   52 0         0:00 ps aux

That’s a really lightweight system! You can even run top to view processes and CPU usage in real time:

# top

Press q to quit.

Let’s see how much memory we’re using versus what’s available:

# free -m

Example output:

              total        used        free      shared  buff/cache   available
Mem:            108          10          95           0           3          95
Swap:             0           0           0

Again, extremely lightweight – the VM has about 128MB memory available, and we only used 10MB. Compare that with Ubuntu 22.04 server on RISC-V that fails to boot with 512MB memory due to insufficient memory 😉

Check how much disk space is used / available:

# df -Th

Example output:

Filesystem           Type            Size      Used Available Use% Mounted on
/dev/root            ext4          973.4M      1.7M    904.5M   0% /
devtmpfs             devtmpfs       53.0M         0     53.0M   0% /dev

We allocated 1GB for our virtual disk, and the BusyBox rootfs takes less than 2MB.

Let’s also test the network. Try pinging the host:

# ping -c5 10.0.2.2

Example output:

PING 10.0.2.2 (10.0.2.2): 56 data bytes
64 bytes from 10.0.2.2: seq=0 ttl=255 time=16.085 ms
64 bytes from 10.0.2.2: seq=1 ttl=255 time=3.775 ms
64 bytes from 10.0.2.2: seq=2 ttl=255 time=3.810 ms
64 bytes from 10.0.2.2: seq=3 ttl=255 time=3.770 ms
64 bytes from 10.0.2.2: seq=4 ttl=255 time=4.322 ms

--- 10.0.2.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 3.770/6.352/16.085 ms

Great! Let’s see if we can reach servers from the Internet. Here’s a public IP address for google.com: 142.250.204.46:

# ping -c5 142.250.204.46

Example output:

PING 142.250.204.46 (142.250.204.46): 56 data bytes
64 bytes from 142.250.204.46: seq=0 ttl=255 time=15.262 ms
64 bytes from 142.250.204.46: seq=1 ttl=255 time=8.313 ms
64 bytes from 142.250.204.46: seq=2 ttl=255 time=8.136 ms
64 bytes from 142.250.204.46: seq=3 ttl=255 time=11.132 ms
64 bytes from 142.250.204.46: seq=4 ttl=255 time=8.752 ms

--- 142.250.204.46 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 8.136/10.319/15.262 ms

Let’s also view information on the CPU through procfs:

# cat /proc/cpuinfo

Output:

processor   : 0
hart        : 0
isa     : rv64imafdc
mmu     : sv57

So our emulated board has a single RISC-V CPU core with a single hardware thread (hart), and the CPU core supports the RV64IMAFDC ISA specification, where “IMAFD” can be simplified to just “G” to give RV64GC (“G” for general extensions, “C” for compressed instructions). You can read more about the RISC-V ISA specification on GitHub, which is highly modular with a minimal base ISA plus many optional extensions.

Play around a bit more, then power down the board:

# poweroff

That’s it – congratulations! You’ve successfully compiled your own Linux kernel and minimal BusyBox userspace, and booted it on a virtual RISC-V virt board with QEMU.

Next steps

If this article had you craving for more, here are a few things you could try to further your adventure. The list is by no means exhaustive:

  • If you played around a bit more, you might’ve noticed that DNS resolution does not work inside the board. Try to figure out why and fix it
  • Try following this article (or the official RISC-V docs) with a real, physical RISC-V SoC. After all, physical hardware is the real deal
  • If you manage to boot your own embedded Linux on physical RISC-V hardware, try doing something useful with it, such as making it an IoT project as part of your smart home
Total
5
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
dynamic-postgresql-credentials-using-hashicorp-vault-(with-php-symfony-&-go-examples)

Dynamic PostgreSQL credentials using HashiCorp Vault (with PHP Symfony & Go examples)

Next Post
hoisting-in-javascript

Hoisting in Javascript

Related Posts