Why Use A Container When There Is chroot?#
I cannot build Yocto “Walnascar” natively on my Fedora 42 - it is not supported [1]. So I was looking for a way to “isolate” a supported distribution (Fedora 41 for that matter) inside Fedora 42.
I had done this a number of times before: using a subdirectory on one
OS as the root directory for another OS, combining a few features like
chroot and bind mounts. Also, I am always looking for
opportunities to explain people the world -
setting up a chroot jail in front the audience would not only
explain what chroot is, but also what a root directory is.
After a number of attempts to contain myself (speak: Docker and systemd-nspawn), I came to the conclusion that a virtual machine, however lightweight, was still too heavy for my task.
Read on to see how easy it is to setup a “container” without having to
install any additional baggage than the contained OS itself. Jump
directly to chroot: De-Containerization (Minimal busybox Root Filesystem) to skip my case
against Docker and read the full explanation of everything. Jump to
Repeat, In A Real Distro Jail (Fedora 41) if you don’t want any explanation
but only want the pure commandlines for a Fedora 41 chroot.
Container Attempts#
Docker#
Docker was my first choice. Colleagues are using it a lot to do build stuff, and are happy with its convenience. Convenience - e.g. gazillions of images to download from Docker Hub - might be the primary reason why Docker is so popular.
Installed docker, pulled fedora:41 from Docker Hub. Read docs, managed to get a
shell. Further steps would have been to mount my home directory into
it, to solve my original problem - build Yocto.
While digging around I found the alarmistic article “Docker considered harmful” which is really worth a read. I mean, it’s probably me - I am really sceptical about any hype, and it takes very little (for example, an article titled “Docker considered harmful” 😉) for me to run away.
systemd-nspawn#
Since systemd took place, everything just works. systemd
always used Linux cgroups
[2] and namespaces to run
services at different isolation levels. systemd-nspawn (here)
is a logical step further. It is It has been created by the
systemd team to test systemd itself, and is comparable to
Docker in many ways.
After my Docker experience, I was still in a mood for
self-modernization. I decided to give systemd-nspawn a try, partly
because its documentation compares it with chroot, only on
steroids.
Long story short: network setup for nspawn containers works best when
the host system’s network is managed by systemd-networkd. My
machine runs GNOME, which works best together with GNOME’s
NetworkManager, which again is perfectly integrated with
wpa_supplicant. systemd-networkd has no idea what
wpa_supplicant is. Doing everything by hand is a cool learning
experience, but does not help me get the job (remember: Yocto) done.
Conclusion#
I’m sure I’d have been able to build bloody Yocto going the containerization route. On the other hand, all I wanted was to have a shell in an isolated environment, and from there access to my home directory outside that environment.
If I went down that route though, I’d have to think a lot more.
Learn about the containerization technology, especially:
How does the technology want me to configure access to my outside home directory
If I access my outside home directory, how would that go together with UID isolation?
How about network access? I don’t want any isolation at that level. If you are building a cluster of microservices, each one running in a different container, you’d probably want network virtualization/isolation - I don’t, however.
But again: my only problem is that I want to build Yocto on Fedora
42 which is not supported by Yocto. I know what chroot is, I know
what a bind mount is, I know what Linux is - so I should be able to
solve my problem in a minimal and hype-less way.
chroot: De-Containerization (Minimal busybox Root Filesystem)#
I’ll begin with a minimal busybox root filesystem. The likelihood
that this particular approach will let me build Yocto is relatively
low again, but it lets me present the fundamentals without confusing
you with distro specific things. If you already know enough you may
well jump ahead and read the section where I repeat the whole
thing with Fedora 41 instead of
busybox.
Conventions
I the remainder, I’ll not use sudo to execute commands as root.
Rather, I’ll use
the hash sign (
#) as prompt when the command is run as rootthe dollar sign (
$) when the command is run as normal user
To help you distinguish commands that I run inside the jail from those that are run on the host system, I try to annotate code blocks accordingly (“Inside jail”, “Outside jail”).
chroot Jail#
chroot is a system call (documentation) that changes
the root directory of the calling process to its single parameter
which has to be a directory. In effect, it redirects path traversal to
start from that directory. This is best explained using an example,
using a shell command chroot with the same name (documentation).
Below is a functional (although rather pointless) root filesystem,
implemented using the neat busybox (Github, Wikipedia).
$ tree ~/Machines/Simple-Busybox
/home/jfasch/Machines/Simple-Busybox
└── bin
├── busybox
├── ls -> /bin/busybox
└── sh -> /bin/busybox
The chroot command is used to execute a command inside the
“jail”. The root directory of that process, and all of its
descendants, is the jail,
/home/jfasch/Machines/Simple-Busybox. Here we enclose an instance
of /bin/sh (actually
/home/jfasch/Machines/Simple-Busybox/bin/sh) into the “container”
/home/jfasch/Machines/Simple-Busybox.
# chroot ~/Machines/Simple-Busybox /bin/sh
# # <-- inside jail
# pwd # <-- actually /home/jfasch/Machines/Simple-Busybox
/
# ls # <-- busybox ls
bin
# ls /bin
busybox ls sh
proc#
proc is a virtual filesystem which implements, among other things,
a view into the kernel’s process list. It is usually mounted in the
root filesystem under /proc, and is used by tools like ps and
lsof to retrieve the information they need. Busybox ps, as an
example, fails without a populated /proc [3]:
# busybox ps
PID USER TIME COMMAND
To make proc available inside the jail, we create a mountpoint
/proc (inside), and mount a proc instance on it. On the host
(err: from outside the jail, as root):
# pwd
/home/jfasch/Machines
# mkdir Simple-Busybox/proc
# mount -t proc proc Simple-Busybox/proc
Retry busybox ps inside, and see a the full process list.
# busybox ps
PID USER TIME COMMAND
1 0 0:12 /usr/lib/systemd/systemd --switched-root --system --deserialize=52 rhgb
2 0 0:00 [kthreadd]
3 0 0:00 [pool_workqueue_]
4 0 0:00 [kworker/R-rcu_g]
...
Note
chroot has nothing to do with isolation: from inside the
jail, we see all processes that were started outside the jail
too.
devtmpfs (And /dev/shm)#
Another virtual filesystem, of type devtmpfs, is usually mounted
at /dev/. For our purposes, it provides special files like
/dev/null which are used occasionally in less trivial root
filesystems. (Our simple Busybox root doesn’t.) Make that available
much like we did with /proc
# mkdir Simple-Busybox/dev
# mount -t devtmpfs dev Simple-Busybox/dev
There is another filesystem type, tmpfs, which is a plain RAM
based filesystem (no persistence). An instance of it is usually
mounted at /dev/shm, to hold POSIX IPC artifacts like semaphores
and shared memory. Yocto uses POSIX IPC heavily; lets create it while
we are at it.
# mkdir Simple-Busybox/dev/shm
# mount -t tmpfs shm Simple-Busybox/dev/shm
Bind Mounts#
To access files outside the jail, one would have to navigate past the
root of the jail upwards in the hierarchy. This is not possible -
which is the entire point of chroot. Instead, bind mounts are
used to make outside content visible inside.
Home Directory#
To make my home directory visible inside the jail, we create a
mountpoint /home/jfasch inside, and mount the outside
/home/jfasch onto it.
# mkdir -p Simple-Busybox/home/jfasch
# chown jfasch:jfasch Simple-Busybox/home/jfasch
# mount --bind /home/jfasch Simple-Busybox/home/jfasch
Network//etc/resolv.conf#
No isolation is cool - I can use the host network inside the jail (simply because the concept “host” does not exist).
# busybox ping 142.251.39.36
PING 142.251.39.36 (142.251.39.36): 56 data bytes
64 bytes from 142.251.39.36: seq=0 ttl=114 time=36.571 ms
64 bytes from 142.251.39.36: seq=1 ttl=114 time=21.538 ms
...
One minor problem though that is easily solved: DNS names. When you
instead say ping www.google.com, the command fails because DNS
name resolution fails.
# busybox ping www.google.com
ping: bad address 'www.google.com'
The so-called resolver is a bunch of routines inside the C library
that do DNS name resolution. The resolver is configured with a file
/etc/resolv.conf on the “host”. To make that file available inside
the jail, again a mountpoint needs to be created (yes, a file can also
serve as a mountpoint), and then /etc/resolv.conf can be bind
mounted into the jail.
# touch Simple-Busybox/etc/resolv.conf
# mount --bind /etc/resolv.conf Simple-Busybox/etc/resolv.conf
Verify all is well,
# busybox ping www.google.com
PING www.google.com (142.250.180.228): 56 data bytes
64 bytes from 142.250.180.228: seq=0 ttl=114 time=24.807 ms
64 bytes from 142.250.180.228: seq=1 ttl=114 time=28.564 ms
...
Working As Non root Inside A Jail (And UID Non-Isolation)#
Now, inside the jail, we see /home/jfasch exactly as it is there
on the outside. My outside UID/GID is 1000/1000, and this is what we
see inside.
Note
chroot has nothing to do with isolation
# ls -l /home/jfasch/
...
drwxr-xr-x 1 1000 1000 68 Jul 1 19:43 Desktop
drwxr-xr-x 1 1000 1000 0 Jun 12 13:01 Documents
drwxr-xr-x 1 1000 1000 1130 Jul 20 17:04 Downloads
drwxr-xr-x 1 1000 1000 1726 Jul 17 08:28 My-Projects
...
We’d now be prepared to
Change into the jail
Work on the outside home directory from inside (for example to build Yocto)
Its just that I do not want to work in my home directory as
root. Two options exist.
Option 1: Simply Use Numeric IDs Inside#
Simply specify my UID/GID to chroot --userspec=, and be myself inside, without
any further action. Problem solved: I am not root anymore (note
the $ prompt from the inside shell), but otherwise there are no
names for me and my group - just numbers.
# chroot --userspec=1000:1000 Simple\(Busybox\) /bin/sh
$ # <-- now inside, note the non-root prompt "$"
$ ls -l /home/jfasch
...
drwxr-xr-x 1 1000 1000 68 Jul 1 19:43 Desktop
drwxr-xr-x 1 1000 1000 0 Jun 12 13:01 Documents
drwxr-xr-x 1 1000 1000 1130 Jul 20 17:04 Downloads
drwxr-xr-x 1 1000 1000 1726 Jul 17 08:28 My-Projects
...
This is fine for me once I automate the entire “chroot into jail and build Yocto there” workflow - I’d not be intersted in any names then.
Option 2: Create User Record Inside#
Note
Again, be aware that there is no isolation in a chroot jail. Any UID/GID that is referenced by a user record inside is exactly the UID/GID from outside.
For convenience only, lets now create names jfasch inside, and
define a home directory that we can easily chdir to.
Currently, at this point, we are using our simple Busybox jail to create user and group. A jail filled with a real distro is a little different; see below jjj “wrap-up fedora 41 jail” for the “real distro” case.
Prepare empty user/group files (our simple root does not have those)
Outside jail## mkdir Simple\(Busybox\)/etc # touch Simple\(Busybox\)/etc/group # touch Simple\(Busybox\)/etc/passwd
Use inside (Busybox) tools to create user and group. As I said, Fedora and other distros are a little different (Busybox has
addgroup, Fedora hasgroupadd, for example).Outside jail## chroot Simple\(Busybox\) busybox addgroup -g 1000 jfasch # chroot Simple\(Busybox\) busybox adduser -s /bin/sh -G jfasch -u 1000 -D jfasch
I can now go into jail in one step, and it would feel like freedom. It’s still Busybox which is not quite usable for a Yocto build. In the next step, let’s see how a jail can be setup from a real distro.
# chroot Simple\(Busybox\)/ /bin/busybox su - jfasch
~ $ # <-- indise jail
~ $ cd ~/My-Projects/FH-ENDLESS/Yocto/
~/My-Projects/FH-ENDLESS/Yocto $ ls -l
total 12
drwxr-xr-x 1 jfasch jfasch 15972 Jul 28 21:58 DOWNLOAD
drwxr-xr-x 1 jfasch jfasch 1060 Jul 28 21:32 SSTATE
drwxr-xr-x 1 jfasch jfasch 8 Apr 10 09:46 build
-rw-r--r-- 1 jfasch jfasch 373 Jan 20 2025 common-bblayers.conf
-rw-r--r-- 1 jfasch jfasch 373 Jan 15 2025 common-local.conf
drwxr-xr-x 1 jfasch jfasch 122 Jan 20 2025 meta-endless
drwxr-xr-x 1 jfasch jfasch 472 Oct 22 2024 meta-raspberrypi
drwxr-xr-x 1 jfasch jfasch 548 Oct 22 2024 poky
drwxr-xr-x 1 jfasch jfasch 28 Nov 15 2024 qemuarm64
drwxr-xr-x 1 jfasch jfasch 92 Jul 28 21:22 qemux86-64
drwxr-xr-x 1 jfasch jfasch 120 Jul 29 09:09 raspberry3-build
Repeat, In A Real Distro Jail (Fedora 41)#
Busybox is small and simple, which is why I used it above to illustrate all the pieces involved. Busybox is used in hardcore Embedded Linux systems to achive a small footprint, but it is for sure not capable to run a Yocto build [4].
What follows is an annotated shell-command-like transcript of what I did to solve my very original problem: a Yocto build on Yocto-unsupported Fedora 42. It has much of what I did above, only more condensed.
Fedora 41 Jail#
Lets now setup a real distro’s root filesystem. Like the Busybox in
chroot Jail, but bigger. All the mountpoints, like
/proc, are already there, for example.
We “parameterize” the root directory, if someone wants to make all this into a script.
# JAILDIR=/home/jfasch/Machines/fedora-41
Fedora package list to install. This somehow gathered by trial and error.
# minimal install
PACKAGES="dnf fedora-release glibc glibc-langpack-en glibc-langpack-de gcc g++ cmake util-linux iputils"
# for Yocto's bitbake itself
PACKAGES="$PACKAGES python3"
# these had to be installed on the host, as bitbake complained
PACKAGES="$PACKAGES chrpath diffstat lz4 patch rpcgen"
# these were discovered, on the host, as needed somewhere deep inside the build
PACKAGES="$PACKAGES perl-FindBin perl-STD"
# these are needed on top of the minimal chroot install, also complained about by bitbake at some point
PACKAGES="$PACKAGES bunzip2 bzip2 cmp cpio diff file git hostname pzstd tar unzstd wget which zstd"
# more perl crap needed
PACKAGES="$PACKAGES perl-Thread-Queue perl-File-Compare perl-open"
# one packaging/wic error near the end of yocto build, saying that something couldn't be converted to "codepage 850"
PACKAGES="$PACKAGES glibc-gconv-extra"
Populate the root directory …
# dnf -y --releasever=41 --best --setopt=install_weak_deps=False --installroot=$JAILDIR --use-host-config install $PACKAGES
Create Environment#
Let’s now repeat the steps from above, one by one.
Mount /proc (see proc):
# mount -t proc proc $JAILDIR/proc/
Mount devtmpfs and /dev/shm (see devtmpfs (And /dev/shm)):
# mount -t devtmpfs dev $JAILDIR/dev/
# mount -t tmpfs shm $JAILDIR/dev/shm
Mount my home directory (see Home Directory):
# mkdir $JAILDIR/home/jfasch
# chown jfasch:jfasch $JAILDIR/home/jfasch
# mount --bind /home/jfasch $JAILDIR/home/jfasch
Mount resolver config (see Home Directory):
# touch $JAILDIR/etc/resolv.conf
# mount --bind /etc/resolv.conf $JAILDIR/etc/resolv.conf
Give myself a name inside the jail (see Option 2: Create User Record Inside):
# chroot $JAILDIR /usr/sbin/groupadd -g 1000 jfasch
# chroot $JAILDIR /usr/sbin/useradd --home-dir /home/jfasch --gid jfasch --no-create-home --shell /bin/bash jfasch
# chroot $JAILDIR su - jfasch
[jfasch@laptop ~]$
Et voila: Fedora 41 inside Fedora 42!
Finally: Build Yocto#
Finally we are prepared to run a Yocto build in a jail the we have set up for that purpose.
[jfasch@laptop ~]$ cd ~/My-Projects/FH-ENDLESS/Yocto/
[jfasch@laptop Yocto]$ . poky/oe-init-build-env raspberry3-build/
[jfasch@laptop raspberry3-build]$ bitbake endless-image-fulldev
... CPU fan getting loud ...
Conclusion#
Please don’t consider this article a flame against any container technologies. Containers are cool when there are problems to be solved bigger than building Yocto using a different distro than the one you are just running. Need isolation, need a full (well, half) OS boot and services started inside - this is what containers are there for.
Sure a full distro container can solve my trivial Yocto problem too. In my opinion, though, this feels like using a sledgehammer to crack a nut. If you feel the same, then this article is for you, and I hope that the information was helpful. Send comments (here) if you liked it and/or have suggestions!
Footnotes