https://www.faschingbauer.meJörg Faschingbauer - Posts in embedded2024-03-21T09:11:51.960899+00:00ABloghttps://www.faschingbauer.me/blog/2022/09/cross-raspi.htmlBuilding A Cross Toolchain For The Raspberry Pi, Using crosstool-ng2022-09-23T00:00:00+02:00Jörg Faschingbauer<section id="building-a-cross-toolchain-for-the-raspberry-pi-using-crosstool-ng">
<nav class="contents local" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#why-this-article-why-not-use-a-prebuilt-toolchain" id="id13">Why This Article? Why Not Use A Prebuilt Toolchain?</a></p></li>
<li><p><a class="reference internal" href="#installing-crosstool-ng" id="id14">Installing <code class="docutils literal notranslate"><span class="pre">crosstool-ng</span></code></a></p>
<ul>
<li><p><a class="reference internal" href="#prerequisites" id="id15">Prerequisites</a></p></li>
<li><p><a class="reference internal" href="#installing-from-cloned-git-repository" id="id16">Installing From Cloned Git Repository</a></p></li>
<li><p><a class="reference internal" href="#installing-from-release-tarball" id="id17">Installing From Release Tarball</a></p></li>
<li><p><a class="reference internal" href="#path-and-basic-test" id="id18"><code class="docutils literal notranslate"><span class="pre">PATH</span></code>, And Basic Test</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#configure-toolchain-build" id="id19">Configure Toolchain Build</a></p>
<ul>
<li><p><a class="reference internal" href="#base-configuration-for-the-raspberry" id="id20">Base Configuration For The Raspberry</a></p></li>
<li><p><a class="reference internal" href="#important-glibc-version" id="id21">Important: <code class="docutils literal notranslate"><span class="pre">glibc</span></code> Version</a></p>
<ul>
<li><p><a class="reference internal" href="#the-error" id="id22">The Error</a></p></li>
<li><p><a class="reference internal" href="#the-fix" id="id23">The Fix</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#important-gcc-version" id="id24">Important: GCC Version</a></p>
<ul>
<li><p><a class="reference internal" href="#id4" id="id25">The Error</a></p></li>
<li><p><a class="reference internal" href="#id5" id="id26">The Fix</a></p></li>
</ul>
</li>
</ul>
</li>
<li><p><a class="reference internal" href="#toolchain-build-install" id="id27">Toolchain Build/Install</a></p></li>
<li><p><a class="reference internal" href="#test" id="id28">Test</a></p>
<ul>
<li><p><a class="reference internal" href="#simplest-c-stdio-h" id="id29">Simplest: C, <code class="docutils literal notranslate"><span class="pre"><stdio.h></span></code></a></p></li>
<li><p><a class="reference internal" href="#c-iostream" id="id30">C++, <code class="docutils literal notranslate"><span class="pre"><iostream></span></code></a></p></li>
<li><p><a class="reference internal" href="#c-thread" id="id31">C++, <code class="docutils literal notranslate"><span class="pre"><thread></span></code></a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#packaging-distributing-the-toolchain" id="id32">Packaging/Distributing The Toolchain</a></p></li>
<li><p><a class="reference internal" href="#afterword" id="id33">Afterword</a></p></li>
</ul>
</nav>
<section id="why-this-article-why-not-use-a-prebuilt-toolchain">
<h2><a class="toc-backref" href="#id13" role="doc-backlink">Why This Article? Why Not Use A Prebuilt Toolchain?</a></h2>
<ul class="simple">
<li><p>There <a class="reference external" href="https://github.com/raspberrypi/tools">used to be</a> a
toolchain for the Raspberry Pi, made by the <a class="reference external" href="https://www.raspberrypi.com/software/">Raspberry Pi OS</a> creators. Sadly, this
repository has been dead since years.</p></li>
<li><p>One popular toolchain, <a class="reference external" href="https://snapshots.linaro.org/gnu-toolchain/">Linaro</a>, is widely
recommended for the Raspberry. It does not work for C++ though - its
<code class="docutils literal notranslate"><span class="pre">libstdc++</span></code> has an unsatisfied version dependency on
<code class="docutils literal notranslate"><span class="pre">libpthread</span></code>.</p></li>
<li><p>A number of other Raspberry toolchains exist out there, but I either
didn’t get them to work at all (see the <code class="docutils literal notranslate"><span class="pre">glibc</span></code> <a class="reference internal" href="../2022/09/cross-raspi.html#glibc-version"><span class="std std-ref">version
dependency</span></a>), or didn’t get them to work for C++.</p></li>
</ul>
<p>Prebuilt toolchains may work for C, but I failed to find one that
works for C++. The amount of work that I invested in finding one was,
in my opinion, totally wasted. Building a toolchain yourself is not at
all hard, provided that you bring basic Linux understanding (which you
should if you program for it).</p>
<p>In my opinion (please given me <a class="reference internal" href="../../about/myself/contact.html"><span class="doc">feedback</span></a>), this article describes the most
close-to-correct way to build a toolchain for a target that does not
come with a toolchain. Although I describe the situation for the
Raspberry (because that was my primary intention), it will sure work
for <em>any</em> target.</p>
</section>
<section id="installing-crosstool-ng">
<h2><a class="toc-backref" href="#id14" role="doc-backlink">Installing <code class="docutils literal notranslate"><span class="pre">crosstool-ng</span></code></a></h2>
<p>Below is a writeup of how I installed <code class="docutils literal notranslate"><span class="pre">crosstool-ng</span></code> (running
Fedora, I chose to install from a released tarball) . Please read
through their <a class="reference external" href="https://crosstool-ng.github.io/docs/install/">installation documentation</a> for your particular
situation.</p>
<section id="prerequisites">
<h3><a class="toc-backref" href="#id15" role="doc-backlink">Prerequisites</a></h3>
<p><code class="docutils literal notranslate"><span class="pre">crosstool-ng</span></code> requires one to install a number of packages
first. Thankfully, they do thorough testing in Docker, and the list of
dependencies can be taken from the Docker file for your OS (see <a class="reference external" href="https://github.com/crosstool-ng/crosstool-ng/tree/master/testing/docker">here</a>).</p>
<div class="literal-block-wrapper docutils container" id="id7">
<div class="code-block-caption"><span class="caption-text">Dependencies for Fedora</span></div>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">root@host # </span>dnf<span class="w"> </span>install<span class="w"> </span>autoconf<span class="w"> </span>gperf<span class="w"> </span>bison<span class="w"> </span>file<span class="w"> </span>flex<span class="w"> </span>texinfo<span class="w"> </span>help2man<span class="w"> </span>gcc-c++<span class="w"> </span>libtool<span class="w"> </span>make<span class="w"> </span>patch<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>ncurses-devel<span class="w"> </span>python3-devel<span class="w"> </span>perl-Thread-Queue<span class="w"> </span>bzip2<span class="w"> </span>git<span class="w"> </span>wget<span class="w"> </span>which<span class="w"> </span>xz<span class="w"> </span>unzip<span class="w"> </span>rsync<span class="w"> </span>diffutils<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>meson<span class="w"> </span>ninja-build
</pre></div>
</div>
</div>
<div class="literal-block-wrapper docutils container" id="id8">
<div class="code-block-caption"><span class="caption-text">Dependencies for Ubuntu</span></div>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">root@host # </span>apt<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>gcc<span class="w"> </span>g++<span class="w"> </span>gperf<span class="w"> </span>bison<span class="w"> </span>flex<span class="w"> </span>texinfo<span class="w"> </span>help2man<span class="w"> </span>make<span class="w"> </span>libncurses5-dev<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>python3-dev<span class="w"> </span>autoconf<span class="w"> </span>automake<span class="w"> </span>libtool<span class="w"> </span>libtool-bin<span class="w"> </span>gawk<span class="w"> </span>wget<span class="w"> </span>bzip2<span class="w"> </span>xz-utils<span class="w"> </span>unzip<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>patch<span class="w"> </span>libstdc++6<span class="w"> </span>rsync<span class="w"> </span>git<span class="w"> </span>meson<span class="w"> </span>ninja-build
</pre></div>
</div>
</div>
</section>
<section id="installing-from-cloned-git-repository">
<h3><a class="toc-backref" href="#id16" role="doc-backlink">Installing From Cloned Git Repository</a></h3>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>At one point, <code class="docutils literal notranslate"><span class="pre">ct-ng</span></code> (as installed from the release tarball)
failed to build toolchains because it tried to download a version
of <code class="docutils literal notranslate"><span class="pre">zlib</span></code> that did not exist anymore:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">[EXTRA] Retrieving 'zlib-1.2.12'</span>
<span class="go">[ERROR] zlib: download failed</span>
</pre></div>
</div>
<p>The issue had already been resolved in the upstream repository, but
no tarball release had been made. So building <code class="docutils literal notranslate"><span class="pre">crosstool-ng</span></code> from
source is in order.</p>
</div>
<p>The steps are basically the same as installing from the release
tarball, except that some massaging (yes, autotools) in the source
directory is necessary prior to building.</p>
<p>Clone repository, and massage source tree:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@host$ </span>git<span class="w"> </span>clone<span class="w"> </span>https://github.com/crosstool-ng/crosstool-ng
<span class="gp">me@host$ </span><span class="nb">cd</span><span class="w"> </span>crosstool-ng/
<span class="gp">me@host$ </span>sh<span class="w"> </span>./bootstrap
<span class="gp">me@host$ </span>./configure<span class="w"> </span>--prefix<span class="o">=</span>/home/jfasch/cross
<span class="gp">me@host$ </span>make
<span class="gp">me@host$ </span>make<span class="w"> </span>install
</pre></div>
</div>
</section>
<section id="installing-from-release-tarball">
<h3><a class="toc-backref" href="#id17" role="doc-backlink">Installing From Release Tarball</a></h3>
<p>As complained above, releases might be out of date. Anyway, the
procedure is slightly simpler that building from the upstream repo
source. Download latest release <a class="reference external" href="https://crosstool-ng.github.io/download/">here</a>); current version as of
this writing is <code class="docutils literal notranslate"><span class="pre">1.25.0</span></code>.</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@host$ </span>wget<span class="w"> </span>http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.25.0.tar.xz
<span class="gp">me@host$ </span>tar<span class="w"> </span>xf<span class="w"> </span>crosstool-ng-1.25.0.tar.xz
<span class="gp">me@host$ </span><span class="nb">cd</span><span class="w"> </span>crosstool-ng-1.25.0/
<span class="gp">me@host$ </span>./configure<span class="w"> </span>--prefix<span class="o">=</span>/home/jfasch/cross
<span class="gp">me@host$ </span>make
<span class="gp">me@host$ </span>make<span class="w"> </span>install
</pre></div>
</div>
</section>
<section id="path-and-basic-test">
<h3><a class="toc-backref" href="#id18" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">PATH</span></code>, And Basic Test</a></h3>
<p>Add <code class="docutils literal notranslate"><span class="pre">ct-ng</span></code> to <code class="docutils literal notranslate"><span class="pre">PATH</span></code>,</p>
<div class="literal-block-wrapper docutils container" id="id9">
<div class="code-block-caption"><span class="caption-text">You may want to add this to <code class="docutils literal notranslate"><span class="pre">~/.bashrc</span></code></span></div>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>me@host$<span class="w"> </span><span class="nb">export</span><span class="w"> </span><span class="nv">PATH</span><span class="o">=</span><span class="nv">$HOME</span>/cross/bin:<span class="nv">$PATH</span>
</pre></div>
</div>
</div>
<p>Test if all is well,</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@host$ </span>ct-ng<span class="w"> </span><span class="nb">help</span>
<span class="go">This is crosstool-NG version 1.25.0</span>
<span class="go">Copyright (C) 2008 Yann E. MORIN <yann.morin.1998@free.fr></span>
<span class="go">This is free software; see the source for copying conditions.</span>
<span class="go">There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A</span>
<span class="go">PARTICULAR PURPOSE.</span>
<span class="go">...</span>
</pre></div>
</div>
</section>
</section>
<section id="configure-toolchain-build">
<h2><a class="toc-backref" href="#id19" role="doc-backlink">Configure Toolchain Build</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">ct-ng</span></code> toolchain build configuration works a lot like the Linux
kernel’s configuration system (a little less elaborate though). There
are a number of base configurations to choose from (we need a
Raspberry config), and, having chosen one, you can tune it to your
needs.</p>
<section id="base-configuration-for-the-raspberry">
<h3><a class="toc-backref" href="#id20" role="doc-backlink">Base Configuration For The Raspberry</a></h3>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@host$ </span>ct-ng<span class="w"> </span>list-samples<span class="p">|</span>grep<span class="w"> </span>rpi
<span class="go">[G...] aarch64-rpi3-linux-gnu</span>
<span class="go">[G...] aarch64-rpi4-linux-gnu</span>
<span class="go">[G...] armv7-rpi2-linux-gnueabihf</span>
<span class="go">[G...] armv8-rpi3-linux-gnueabihf</span>
<span class="go">[G...] armv8-rpi4-linux-gnueabihf</span>
</pre></div>
</div>
<p>Mine is a Pi 4, still running the (still default) 32 bit version, so I
create a <code class="docutils literal notranslate"><span class="pre">.config</span></code> file for that,</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@host $ </span>ct-ng<span class="w"> </span>armv8-rpi4-linux-gnueabihf
<span class="gp">me@host $ </span>ls<span class="w"> </span>-l<span class="w"> </span>.config
<span class="go">-rw-r--r--. 1 jfasch jfasch 23179 Sep 23 08:18 .config</span>
</pre></div>
</div>
</section>
<section id="important-glibc-version">
<span id="glibc-version"></span><h3><a class="toc-backref" href="#id21" role="doc-backlink">Important: <code class="docutils literal notranslate"><span class="pre">glibc</span></code> Version</a></h3>
<section id="the-error">
<h4><a class="toc-backref" href="#id22" role="doc-backlink">The Error</a></h4>
<p>Executables built by your toolchain won’t work if the toolchain links
against a C library whose version is higher than what is present on
the target. Continuing with the above default configuration, and just
saying <code class="docutils literal notranslate"><span class="pre">ct-ng</span> <span class="pre">build</span></code> will likely result in an error
<a class="footnote-reference brackets" href="#glibc-default-version" id="id2" role="doc-noteref"><span class="fn-bracket">[</span>3<span class="fn-bracket">]</span></a>.</p>
<p>A minimal <code class="docutils literal notranslate"><span class="pre">int</span> <span class="pre">main()</span> <span class="pre">{</span> <span class="pre">return</span> <span class="pre">0;</span> <span class="pre">}</span></code> program, compiled for the
target, will complain:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@target $ </span>./a.out
<span class="go">./a.out: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.34' not found (required by ./a.out)</span>
</pre></div>
</div>
<p>Background is that <code class="docutils literal notranslate"><span class="pre">glibc</span></code> employs <a class="reference external" href="https://refspecs.linuxfoundation.org/LSB_3.0.0/LSB-PDA/LSB-PDA.junk/symversion.html">ELF symbol versioning</a>
a lot. Key point is that an executable which links against version,
say, <cite>2.34</cite> (which is the <code class="docutils literal notranslate"><span class="pre">ct-ng</span></code> default), will require the symbol
<code class="docutils literal notranslate"><span class="pre">GLIBC_2.34</span></code> in the C library, and fail to load if the target
<code class="docutils literal notranslate"><span class="pre">glibc</span></code> version is, say, <code class="docutils literal notranslate"><span class="pre">2.28</span></code> <a class="footnote-reference brackets" href="#my-situation" id="id3" role="doc-noteref"><span class="fn-bracket">[</span>1<span class="fn-bracket">]</span></a>. That version
obviously cannot have <code class="docutils literal notranslate"><span class="pre">GLIBC_2.34</span></code>, so the executable cannot load.</p>
<p>Find out location of <code class="docutils literal notranslate"><span class="pre">glibc</span></code>,</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@target $ </span>ldd<span class="w"> </span>/usr/bin/ls
<span class="go"> ...</span>
<span class="go"> libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6e06000)</span>
<span class="go"> ...</span>
</pre></div>
</div>
<p>Simply <em>invoke</em> (sic!) the library,</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@target $ </span>/lib/arm-linux-gnueabihf/libc.so.6
<span class="go">GNU C Library (Debian GLIBC 2.28-10+rpt2+rpi1+deb10u1) stable release version 2.28.</span>
<span class="go">...</span>
</pre></div>
</div>
</section>
<section id="the-fix">
<h4><a class="toc-backref" href="#id23" role="doc-backlink">The Fix</a></h4>
<p>So, it’s <code class="docutils literal notranslate"><span class="pre">2.28</span></code>. Back on the <em>host</em> again, tune configuration:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@host $ </span>ct-ng<span class="w"> </span>menuconfig
</pre></div>
</div>
<p>In <code class="docutils literal notranslate"><span class="pre">C-library</span> <span class="pre">-></span> <span class="pre">Version</span> <span class="pre">of</span> <span class="pre">glibc</span></code>, choose <code class="docutils literal notranslate"><span class="pre">2.28</span></code>.</p>
<img alt="blog/2022/09/glibc-version.png" src="blog/2022/09/glibc-version.png" style="width: 477.0px; height: 255.5px;" />
<p>Minor quirk: version <code class="docutils literal notranslate"><span class="pre">2.28</span></code> does not build without warnings, so one
has to disable <code class="docutils literal notranslate"><span class="pre">-Werror</span></code> during its build,</p>
<img alt="blog/2022/09/werror.png" src="blog/2022/09/werror.png" style="width: 477.0px; height: 255.5px;" />
</section>
</section>
<section id="important-gcc-version">
<h3><a class="toc-backref" href="#id24" role="doc-backlink">Important: GCC Version</a></h3>
<p>Not only is the Pi’s <code class="docutils literal notranslate"><span class="pre">glibc</span></code> version rather outdated; GCC is also a
bit behind - it’s <code class="docutils literal notranslate"><span class="pre">8.3.0</span></code>.</p>
<section id="id4">
<h4><a class="toc-backref" href="#id25" role="doc-backlink">The Error</a></h4>
<aside class="system-message">
<p class="system-message-title">System Message: INFO/1 (<span class="docutils literal">/home/jfasch/My-Projects/jfasch-home/blog/2022/09/cross-raspi.rst</span>, line 256); <em><a href="#id4">backlink</a></em></p>
<p>Duplicate implicit target name: “the error”.</p>
</aside>
<p>Using the latest supported GCC version (<code class="docutils literal notranslate"><span class="pre">12.2.0</span></code> as of this writing)
as a cross compiler, A less trivial C++ program fails to start:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">./a.out: /lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by ./a.out)</span>
<span class="go">./a.out: /lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by ./a.out)</span>
</pre></div>
</div>
<p>It turns out that the program makes use of the <code class="docutils literal notranslate"><span class="pre">std::filesystem</span></code>
library that has been added in C++17. Apparently that library uses
some more modern functionality from GCC’s C++ support library,
<code class="docutils literal notranslate"><span class="pre">libgcc_s.so.1</span></code>. That functionality is obviously not contained in
<code class="docutils literal notranslate"><span class="pre">libgcc_s.so.1</span></code> as of GCC <code class="docutils literal notranslate"><span class="pre">8.3.0</span></code></p>
</section>
<section id="id5">
<h4><a class="toc-backref" href="#id26" role="doc-backlink">The Fix</a></h4>
<aside class="system-message">
<p class="system-message-title">System Message: INFO/1 (<span class="docutils literal">/home/jfasch/My-Projects/jfasch-home/blog/2022/09/cross-raspi.rst</span>, line 273); <em><a href="#id5">backlink</a></em></p>
<p>Duplicate implicit target name: “the fix”.</p>
</aside>
<p>Downgrade the toolchain’s GCC to something like <code class="docutils literal notranslate"><span class="pre">8.3.0</span></code> (<code class="docutils literal notranslate"><span class="pre">8.5.0</span></code>
is just a minor release, so it should work).</p>
<p>In <code class="docutils literal notranslate"><span class="pre">C</span> <span class="pre">compiler</span> <span class="pre">-></span> <span class="pre">Version</span> <span class="pre">of</span> <span class="pre">gcc</span></code>, choose <code class="docutils literal notranslate"><span class="pre">8.5.0</span></code>:</p>
<img alt="blog/2022/09/gcc-version.png" src="blog/2022/09/gcc-version.png" style="width: 400.5px; height: 264.5px;" />
</section>
</section>
</section>
<section id="toolchain-build-install">
<h2><a class="toc-backref" href="#id27" role="doc-backlink">Toolchain Build/Install</a></h2>
<p>This is as simple as saying,</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@host $ </span>ct-ng<span class="w"> </span>build
</pre></div>
</div>
<p>By default <a class="footnote-reference brackets" href="#ct-prefix" id="id6" role="doc-noteref"><span class="fn-bracket">[</span>2<span class="fn-bracket">]</span></a>, the toolchain is installed in
<code class="docutils literal notranslate"><span class="pre">$HOME/x-tools/armv8-rpi4-linux-gnueabihf/</span></code> (you may want to add
<code class="docutils literal notranslate"><span class="pre">$HOME/x-tools/armv8-rpi4-linux-gnueabihf/bin/</span></code> to <code class="docutils literal notranslate"><span class="pre">$PATH</span></code>).</p>
<p>For build system configuration (see for example
<a class="reference internal" href="../../trainings/material/soup/linux/toolchain/cmake/cross.html"><span class="doc">CMake: Cross Build</span></a>),</p>
<table class="docutils align-default">
<tbody>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">PATH</span></code> (best in <code class="docutils literal notranslate"><span class="pre">~/.bashrc</span></code>)</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">$HOME/x-tools/armv8-rpi4-linux-gnueabihf/bin</span></code></p></td>
</tr>
<tr class="row-even"><td><p>Sysroot; in CMake toolchain files (see
<a class="reference internal" href="../../trainings/material/soup/linux/toolchain/cmake/cross.html"><span class="doc">CMake: Cross Build</span></a>,
“Toolchain Files”), this is</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">CMAKE_FIND_ROOT_PATH</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">CMAKE_SYSROOT</span></code></p></li>
</ul>
</td>
<td><p><code class="docutils literal notranslate"><span class="pre">$HOME/x-tools/armv8-rpi4-linux-gnueabihf/armv8-rpi4-linux-gnueabihf</span></code></p></td>
</tr>
</tbody>
</table>
</section>
<section id="test">
<h2><a class="toc-backref" href="#id28" role="doc-backlink">Test</a></h2>
<p>I tested the toolchain from a number of viewpoints that I describe
below.</p>
<section id="simplest-c-stdio-h">
<h3><a class="toc-backref" href="#id29" role="doc-backlink">Simplest: C, <code class="docutils literal notranslate"><span class="pre"><stdio.h></span></code></a></h3>
<p>Check for availability of <code class="docutils literal notranslate"><span class="pre">libc</span></code>.</p>
<div class="literal-block-wrapper docutils container" id="id10">
<div class="code-block-caption"><span class="caption-text"><a class="reference download internal" download="" href="../../_downloads/930da294b589e3c0ee18e29e9212dc93/c-stdio.c"><code class="xref download docutils literal notranslate"><span class="pre">c-stdio.c</span></code></a></span></div>
<div class="highlight-c notranslate"><div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"Hello World</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@host $ </span>~/x-tools/armv8-rpi4-linux-gnueabihf/bin/armv8-rpi4-linux-gnueabihf-gcc<span class="w"> </span>c-stdio.c
<span class="gp">me@host $ </span>scp<span class="w"> </span>a.out<span class="w"> </span>me@target:
<span class="gp">me@host $ </span>ssh<span class="w"> </span>me@target<span class="w"> </span>./a.out
<span class="go">Hello World</span>
</pre></div>
</div>
</section>
<section id="c-iostream">
<h3><a class="toc-backref" href="#id30" role="doc-backlink">C++, <code class="docutils literal notranslate"><span class="pre"><iostream></span></code></a></h3>
<p>Check for availability of <code class="docutils literal notranslate"><span class="pre">libstdc++</span></code>.</p>
<div class="literal-block-wrapper docutils container" id="id11">
<div class="code-block-caption"><span class="caption-text"><a class="reference download internal" download="" href="../../_downloads/1f44756998952fcb268db788f7cf730b/c%2B%2B-iostream.cpp"><code class="xref download docutils literal notranslate"><span class="pre">c++-iostream.cpp</span></code></a></span></div>
<div class="highlight-c++ notranslate"><div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><iostream></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span>
<span class="p">{</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s">"Hello World"</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@host $ </span>~/x-tools/armv8-rpi4-linux-gnueabihf/bin/armv8-rpi4-linux-gnueabihf-g++<span class="w"> </span>c++-iostream.cpp
<span class="gp">me@host $ </span>scp<span class="w"> </span>a.out<span class="w"> </span>me@target:
<span class="gp">me@host $ </span>ssh<span class="w"> </span>me@target<span class="w"> </span>./a.out
<span class="go">Hello World</span>
</pre></div>
</div>
</section>
<section id="c-thread">
<h3><a class="toc-backref" href="#id31" role="doc-backlink">C++, <code class="docutils literal notranslate"><span class="pre"><thread></span></code></a></h3>
<p>Check for availability of <code class="docutils literal notranslate"><span class="pre">libpthread</span></code>.</p>
<div class="literal-block-wrapper docutils container" id="id12">
<div class="code-block-caption"><span class="caption-text"><a class="reference download internal" download="" href="../../_downloads/59cbba0b5d6de2183a7a48be92ed635e/c%2B%2B-thread.cpp"><code class="xref download docutils literal notranslate"><span class="pre">c++-thread.cpp</span></code></a></span></div>
<div class="highlight-c++ notranslate"><div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><thread></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><iostream></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span>
<span class="p">{</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="kr">thread</span><span class="w"> </span><span class="n">t</span><span class="p">([](){</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s">"Hello Thread"</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="n">t</span><span class="p">.</span><span class="n">join</span><span class="p">();</span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="s">"Bye"</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">me@host $ </span>~/x-tools/armv8-rpi4-linux-gnueabihf/bin/armv8-rpi4-linux-gnueabihf-g++<span class="w"> </span>-pthread<span class="w"> </span>c++-thread.cpp
<span class="gp">me@host $ </span>scp<span class="w"> </span>a.out<span class="w"> </span>me@target:
<span class="gp">me@host $ </span>ssh<span class="w"> </span>me@target<span class="w"> </span>./a.out
<span class="go">Hello Thread</span>
<span class="go">Bye</span>
</pre></div>
</div>
</section>
</section>
<section id="packaging-distributing-the-toolchain">
<h2><a class="toc-backref" href="#id32" role="doc-backlink">Packaging/Distributing The Toolchain</a></h2>
<p>The toolchain is fully relocatable. Although
<code class="docutils literal notranslate"><span class="pre">$HOME/x-tools/armv8-rpi4-linux-gnueabihf/</span></code> contains the cross
toolchain <em>together</em> with the <a class="reference internal" href="../../trainings/material/soup/linux/toolchain/cross/basics.html"><span class="doc">sysroot</span></a>, even if you
move it around it will remain self-contained.</p>
<p>Pack it into an archive,</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>tar<span class="w"> </span>-C<span class="w"> </span>~/x-tools<span class="w"> </span>-J<span class="w"> </span>-c<span class="w"> </span>-f<span class="w"> </span>raspberry-pi-toolchain.tar.xz<span class="w"> </span>armv8-rpi4-linux-gnueabihf/
</pre></div>
</div>
<p>And share it with your colleagues.</p>
</section>
<section id="afterword">
<h2><a class="toc-backref" href="#id33" role="doc-backlink">Afterword</a></h2>
<p>I hope I could help you with this article. Share it if I could, or, if
I couldn’t, feel free to send me an <a class="reference internal" href="../../about/myself/contact.html"><span class="doc">email</span></a> explaining why.</p>
<p>Either way: cross development is not easy, given the huge number of
dependencies between target libraries. This is a rather complex and
fragile thing, and there are sure better ways to do cross development
than the one that I describe here. Ideally, a target vendor (in our
case, the guys who create <a class="reference external" href="https://www.raspberrypi.com/software/">Raspberry Pi OS</a>) would release an SDK that
developers could install, and that contained <em>all</em> that is necessary
to do cross development. This would be the toolchain and associated
basic runtime libraries (described here), and the entire <em>sysroot</em> of
the target.</p>
<p>Creating such an SDK is the job of higher level tools like <a class="reference external" href="https://www.yoctoproject.org/">Yocto</a> or <a class="reference external" href="https://buildroot.org/">Buildroot</a> - but this is another story.</p>
<p class="rubric">Footnotes</p>
<aside class="footnote-list brackets">
<aside class="footnote brackets" id="my-situation" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="#id3">1</a><span class="fn-bracket">]</span></span>
<p>That describes the situation that I encountered;
yours might be a little different.</p>
</aside>
<aside class="footnote brackets" id="ct-prefix" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="#id6">2</a><span class="fn-bracket">]</span></span>
<p>You change the installation prefix by setting the
<code class="docutils literal notranslate"><span class="pre">CT_PREFIX</span></code> environment variable prior to building,</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span><span class="nv">CT_PREFIX</span><span class="o">=</span>/some/other/location<span class="w"> </span>ct-ng<span class="w"> </span>build
</pre></div>
</div>
</aside>
<aside class="footnote brackets" id="glibc-default-version" role="doc-footnote">
<span class="label"><span class="fn-bracket">[</span><a role="doc-backlink" href="#id2">3</a><span class="fn-bracket">]</span></span>
<p>As of this writing, <code class="docutils literal notranslate"><span class="pre">ct-ng</span></code> chooses the
GNU C Library version <code class="docutils literal notranslate"><span class="pre">2.34</span></code>. Your
situation might be different, the
principle remains the same.</p>
</aside>
</aside>
</section>
</section>
Why This Article? Why Not Use A Prebuilt Toolchain?2022-09-23T00:00:00+02:00https://www.faschingbauer.me/blog/2011/01/porting-to-linux-there-s-always-a-better-way.htmlPorting to Linux (There’s Always A Better Way)2011-01-07T00:00:00+01:00Jörg Faschingbauer<section id="porting-to-linux-there-s-always-a-better-way">
<aside class="sidebar">
<p class="sidebar-title">Contents</p>
<nav class="contents local" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#processes-are-threads" id="id1">Processes are Threads</a></p></li>
<li><p><a class="reference internal" href="#signals-aren-t-messages" id="id2">Signals Aren’t Messages</a></p></li>
<li><p><a class="reference internal" href="#message-queues-aren-t-message-queues" id="id3">Message Queues aren’t Message Queues</a></p></li>
<li><p><a class="reference internal" href="#semaphores-and-mutexes" id="id4">Semaphores and Mutexes</a></p></li>
<li><p><a class="reference internal" href="#timers" id="id5">Timers</a></p></li>
<li><p><a class="reference internal" href="#polling" id="id6">Polling</a></p></li>
<li><p><a class="reference internal" href="#event-dispatching" id="id7">Event Dispatching</a></p></li>
<li><p><a class="reference internal" href="#interrupts" id="id8">Interrupts</a></p></li>
<li><p><a class="reference internal" href="#realtime" id="id9">Realtime</a></p></li>
</ul>
</nav>
</aside>
<p>Over the past months I met a couple of people who are relatively new
to Linux. For one, I’ve been to Germany doing courses on Linux -
naturally the audience in such a course is new to Linux. For another,
I have one and a half customers here in Austria who are mainly doing
microcontroller work and who don’t have a clue of Linux.</p>
<p>Many of these people have code which they have written and maintained
over a long time, and which has a value. The code runs happily on bare
metal, or on a minimal OS like Segger embOS, Enea OSE, VxWorks, or
even Windows CE. No way rewriting it just because of Linux.</p>
<p>Well, there’s no need to. It’s just not trivial to choose the right
mechanisms. Linux (and Unix in general) offers so many choices to mess
things up right from the beginning. And it’s quite likely that things
are in fact messed up, which is what I learned from the people I met.</p>
<p>“Good Old Device Firmware” is quite limited in what it can use.</p>
<ul class="simple">
<li><p>It is a statically linked blob of executable code.</p></li>
<li><p>It operates in a single address space (there’s generally no idea
what an address space is, so there is only one).</p></li>
<li><p>It has interrupts hammering on it.</p></li>
<li><p>If there is an OS, then chances are that there are multiple
processes that are scheduled by the OS. Often processes communicate
by dumping messages into each other’s message queues.</p></li>
<li><p>Locking mechanisms. In the presence of interrupts you need to
disable them if need be. If processes are used, there are generally
semaphores available in one or more flavors.</p></li>
<li><p>It’s these simple concepts where the confusion starts. For example,
in Linux/Unix a process is probably not what you want - you rather
want a thread instead. When you search for the term “message queue”
you will find <em>several</em> mechanisms, <em>none of which</em> are what you
want. You’ll notice sooner or later that you chose the wrong
concept, but it’s maybe too late to revert - causing your Linux port
to fail over the years. (In my opinion it’s never too late to
revert, but that’s a different kind of story.)</p></li>
</ul>
<p>In the remainder I’ll try to give a list of recipes for porting to
Linux. I hope that it will help you speed up the porting project, by
avoiding severe mistakes right at the beginning. The list cannot be
complete - it’s just a blog post after all. But I do hope it will give
you a bigger picture than you already have, and that it provides you
with the starting points for your own research (which you will have to
do unless you want to hire me as a consultant :-).</p>
<section id="processes-are-threads">
<h2><a class="toc-backref" href="#id1" role="doc-backlink">Processes are Threads</a></h2>
<p>In the embedded OS world, you use the word “process” to refer to an
entity that is scheduled by the OS. There is no such thing as separate
address spaces and memory protection in such systems (these OS’s are
initially designed for cheap MMU-less processors).</p>
<p>So, what people generally want is <em>threads</em>. <em>Processes</em> in Unix refer
to address space separation and memory protection, so don’t get misled
only by the term. The Unix way of multithreading is POSIX threads
(pthreads for short). On Linux, <code class="docutils literal notranslate"><span class="pre">man</span> <span class="pre">pthreads</span></code> will tell you
more. Better yet, buy yourself a copy of a <a class="reference external" href="http://oreilly.com/catalog/9781565921153">PThreads Programming</a> book. Read that book
twice. Threading has lots of pitfalls, and it is crucial to understand
these. (I have seen people create five threads to solve a simple
data-acquisition-and-network-communication problem where they didn’t
know what a mutex was.)</p>
</section>
<section id="signals-aren-t-messages">
<h2><a class="toc-backref" href="#id2" role="doc-backlink">Signals Aren’t Messages</a></h2>
<p>Don’t use Unix signals for communication. They have very few in common
with what’s called a <em>signal</em> in RTOSs. (In the RTOS world, <em>signals</em>
are often called <em>messages</em>.)</p>
<aside class="sidebar">
<p class="sidebar-title">References</p>
<ul class="simple">
<li><p><a class="reference internal" href="../../trainings/material/soup/linux/sysprog/sysprog_signals/slides.html"><span class="doc">Slide material: UNIX signals</span></a></p></li>
<li><p><a class="reference internal" href="../../trainings/material/soup/linux/sysprog/sysprog_signals/screenplay.html"><span class="doc">Demo code: UNIX signals</span></a></p></li>
</ul>
</aside>
<p>In Unix, signals are commonly used to tear down a process (<em>not</em> a
thread), giving it a chance to perform proper cleanup before
exit. There are other uses of signals as well, such as notification of
memory protection violation (the dreaded “segmentation fault”) and
other programming errors.</p>
<p>The problem with signal handling is that signals arrive in a special
context which you probably know as “interrupt context” in your
OS. Consequently, your options are very limited in a signal
handler. You cannot use most of the functions that are available from
the C runtime library!</p>
<p>If you still feel that you need to use signals, then perform the
following steps.</p>
<ul class="simple">
<li><p>Think twice. Why do you need signals? There’s probably a better way.</p></li>
<li><p>Read <code class="docutils literal notranslate"><span class="pre">man</span> <span class="pre">-s</span> <span class="pre">7</span> <span class="pre">signal</span></code> for more. <strong>Pay special attention to the
section about async-signal-safe functions</strong> (note that none of the
<code class="docutils literal notranslate"><span class="pre">pthread_</span></code> functions appears in the list).</p></li>
<li><p>Try to defer processing from the signal handler into the regular
course of your application. For example, you can use the “self pipe”
trick (Google sure knows about it.)</p></li>
<li><p>Forget about asynchronous signal delivery and use one of the``
sigwait``, <code class="docutils literal notranslate"><span class="pre">sigwaitinfo</span></code>, and <code class="docutils literal notranslate"><span class="pre">sigtimedwait</span></code> system calls. This
way you suspend the execution of one thread until a signal is
caught, thus turning signal handling into a <em>synchronous</em>
approach. You’d have to dedicate an entire thread to signal handling
though.</p></li>
<li><p>An alternative way of synchronous signal handling is to use the new
<code class="docutils literal notranslate"><span class="pre">signalfd</span></code> system call. Its semantics are the same as
<code class="docutils literal notranslate"><span class="pre">sigwaitinfo</span></code>, only you use a file descriptor as an “event
source”. You can embed this file descriptor among other event
sources in an event driven application, using <code class="docutils literal notranslate"><span class="pre">select</span></code>, <code class="docutils literal notranslate"><span class="pre">poll</span></code>,
or <code class="docutils literal notranslate"><span class="pre">epoll</span></code>. See below for more.</p></li>
</ul>
</section>
<section id="message-queues-aren-t-message-queues">
<h2><a class="toc-backref" href="#id3" role="doc-backlink">Message Queues aren’t Message Queues</a></h2>
<aside class="sidebar">
<p class="sidebar-title">References</p>
<ul class="simple">
<li><p><a class="reference internal" href="../../trainings/material/soup/linux/sysprog/sysprog_ipc/slides.html"><span class="doc">Slide material: POSIX message queues</span></a></p></li>
<li><p><a class="reference internal" href="../../trainings/material/soup/linux/sysprog/sysprog_ipc/screenplay.html"><span class="doc">Demo code: POSIX message queues</span></a></p></li>
</ul>
</aside>
<p>Chances are that the threads (err, processes) of your OS communicate
via messages queues over which messages (err, signals) are sent. A
naive Google search (“Linux message queues”) will lead you to POSIX
and System V message queues, both of which are inter process
communication (IPC) mechanisms. This is probably not what you want, as
your messages need not be transferred across different address
spaces. Intra process message queues are normally built on top of
pthread primitives. You can find a sample such implementation <a class="reference external" href="http://jf-linuxtools.git.sourceforge.net/git/gitweb.cgi?p=jf-linuxtools/jf-linuxtools;a=blob;f=pthread/base/message-queue.h">here</a>.</p>
</section>
<section id="semaphores-and-mutexes">
<h2><a class="toc-backref" href="#id4" role="doc-backlink">Semaphores and Mutexes</a></h2>
<aside class="sidebar">
<p class="sidebar-title">References</p>
<ul class="simple">
<li><p><a class="reference internal" href="../../trainings/material/soup/linux/sysprog/sysprog_ipc/slides.html"><span class="doc">Slide material: POSIX semaphores</span></a></p></li>
<li><p><a class="reference internal" href="../../trainings/material/soup/linux/sysprog/sysprog_ipc/screenplay.html"><span class="doc">Demo code: POSIX semaphores</span></a></p></li>
</ul>
</aside>
<p>Many of those OSs out there have semaphores as the one and only
synchronisation mechanism. Linux has semaphores - System V and POSIX
style. Again these are probably not what you want. You are likely to
initialize your semaphores with a count of one and use it as a <em>binary
semaphore</em>. This is what a pthread mutex is for instead.</p>
<p>See <code class="docutils literal notranslate"><span class="pre">man</span> <span class="pre">pthread_mutex_init</span></code> for more.</p>
<p>Confusing note for those porting from Windows: A <em>critical section</em> is
the region that is protected by a mutex, although a
<code class="docutils literal notranslate"><span class="pre">CRITICAL_SECTION</span></code> is the closest Windows pendant to a mutex.</p>
</section>
<section id="timers">
<h2><a class="toc-backref" href="#id5" role="doc-backlink">Timers</a></h2>
<p>Timers are hard, not only on Linux.</p>
<p>In most embedded OSs, timers expire as interrupts which are then
handled by the application. Sometimes there is the possibility to let
the OS send you a message (err, signal) on timer expiration.</p>
<p>There are multiple APIs for timers, most of which (<code class="docutils literal notranslate"><span class="pre">setitimer</span></code>,
<code class="docutils literal notranslate"><span class="pre">timer_create</span></code>) require you to use signals (Unix signals this
time) - see above for the drawbacks to this approach. Unfortunately a
straightforward Google search will lead you to these APIs.</p>
<p>An alternative is to simply defer the execution of a thread for a
specified amount of time, using <code class="docutils literal notranslate"><span class="pre">nanosleep</span></code>. Another alternative is
to use <code class="docutils literal notranslate"><span class="pre">timerfd_create</span></code> and friends in conjunction with <code class="docutils literal notranslate"><span class="pre">select</span></code>
or <code class="docutils literal notranslate"><span class="pre">poll</span></code>, and embed timers into an event driven application. (See
below for event loops based on file descriptors. See <a class="reference external" href="http://jf-linuxtools.git.sourceforge.net/git/gitweb.cgi?p=jf-linuxtools/jf-linuxtools;a=blob;f=linux-events/events/timer.h">here</a>
for a sample implementation of such a thing.)</p>
<p>You can always build your own structures around whatever timer
mechanism and emulate the behavior of your OS on Linux. This might
require quite a bit of understanding of Linux programming, though.</p>
</section>
<section id="polling">
<h2><a class="toc-backref" href="#id6" role="doc-backlink">Polling</a></h2>
<p>Polling for something to happen is generally the most stupid thing to
do, not only on Linux. The are cases (screwed hardware for example)
where there is no other way, but otherwise there’s always a better
way.</p>
<p>I have seen people set a flag in one process (err, thread), and poll
for it every millisecond from another thread. I know, a millisecond is
an eternity in nowadays’ processors - but I can imagine that there are
more flags of that sort being polled for, in a moderately complex
program. After all, it’s <em>events</em> that these flags communicate. Some
OSs have an “event flag” mechanism which tries to achieve exactly
that, albeit a bit more intelligently - for example without the need
for polling, and without losing events through race conditions.</p>
<p>There’s a big wealth of mechanisms in Linux to communicate events. Use
message queues (see above), for example. Build your own “event flag”
mechanism by using PThreads primitives (but not before you have read
the book).</p>
</section>
<section id="event-dispatching">
<h2><a class="toc-backref" href="#id7" role="doc-backlink">Event Dispatching</a></h2>
<p>On Unix, <em>everything is a file</em>. Entire disks are represented as
files, partitions are, network sockets are, arbitrary devices are (for
example, serial/UART lines). Regular files are, naturally. On Linux,
even timers and Unix signals can be represented as files, as I
mentioned previously. Every open file is represented as a <em>file
descriptor</em>. File descriptors can be read from and written to, using
the <code class="docutils literal notranslate"><span class="pre">read</span></code> and <code class="docutils literal notranslate"><span class="pre">write</span></code> system calls, regardless of their type.</p>
<p>For people who are new to Unix, this may come as a surprise: you can
read from a serial line as if it were a network connection, and you
can read from a network connection as if it were a regular
file. (Anyone tried that on Windows?)</p>
<p>This fact alone is something you can take advantage of in your porting
project.</p>
<p>Now what has this got to do with <em>event dispatching</em>? Consider the
following types of <em>events</em> that regularly happen on an OS:</p>
<ul class="simple">
<li><p>Network connection has data to be read.</p></li>
<li><p>Network connection can be written to without blocking (or, “local
TCP buffer just got emptied” or so).</p></li>
<li><p>Same with serial IO (open <code class="docutils literal notranslate"><span class="pre">/dev/ttyS0</span></code> or so just like a regular
file), as with any kind of stream connection.</p></li>
<li><p>A timer has expired.</p></li>
<li><p>A signal has arrived.</p></li>
<li><p>An arbitrary event happened (check out the <code class="docutils literal notranslate"><span class="pre">eventfd</span></code> system call;
I didn’t mention it).</p></li>
<li><p>Even USB events are delivered via files. (It’s just a bit more
complicated because USB is a bit more complicated; there’s <a class="reference external" href="http://www.libusb.org/">libusb</a> there to handle this.).</p></li>
<li><p>Most drivers for hardware devices deliver the device interrupts as
events through file descriptors (you open <code class="docutils literal notranslate"><span class="pre">/dev/blah</span></code> just like a
regular file), nicely telling you “Hey, I’ve got to tell you
something!”.</p></li>
</ul>
<p>There is a set of system calls that are used to wait for events on
<em>multiple</em> file descriptor, namely <code class="docutils literal notranslate"><span class="pre">select</span></code>,`` poll``, and
<code class="docutils literal notranslate"><span class="pre">epoll</span></code>. Note that these system call put the calling process (err,
thread) to sleep until something happens - the name <code class="docutils literal notranslate"><span class="pre">poll</span></code> has
nothing to do with polling as we know it.</p>
<p>So, depending on your application, you might find it relatively easy
to use either of these system calls. Build a bit of an infrastructure
around it, like registering and unregistering callbacks, and you will
be able to comfortably use these extremely exciting mechanisms. (Or
take <a class="reference external" href="http://jf-linuxtools.git.sourceforge.net/git/gitweb.cgi?p=jf-linuxtools/jf-linuxtools;a=blob;f=events/dispatcher.h">my own infrastructure</a>. Or
take anybody else’s, mine’s not the only - it’s just the best.)</p>
<p>One last note: look at the desktop environment of your choice. It’s
made up of a hundred or so processes (no, not threads). Under Gnome,
which is what I use, there’s a process called`` gnome-panel``, for
example. Try out <code class="docutils literal notranslate"><span class="pre">strace</span> <span class="pre">-p</span> <span class="pre"><pid</span> <span class="pre">of</span> <span class="pre">gnome-panel>;</span></code>, direct the mouse
over the panel, and see what happens: <code class="docutils literal notranslate"><span class="pre">poll</span></code> all over.</p>
</section>
<section id="interrupts">
<h2><a class="toc-backref" href="#id8" role="doc-backlink">Interrupts</a></h2>
<p>One of the primary goals of an OS is to shield you from
hardware. There a clear separation between <em>user space</em> and <em>kernel
space</em>. User space uses system calls to talk to the kernel. No
interrupts in user space. Basta. (Don’t even think about using
signals!)</p>
<p>If you have hardware that nobody else has, then chances are you will
have to deal with it yourself and write a driver for it. That’s about
the only case where you’ll get in touch with interrupts and have to
learn kernel programming. But this is a completely different story,
and a different world. Definitely fun.</p>
<p>Nevertheless, I suggest you stay away from kernel programming just
because it is fun. Debugging is much harder there. A buggy user space
program crashes, possibly leaving you with a core dump that you can
examine with a debugger (well, GDB). Everything else will remain
running. A buggy kernel crashes, leaving nothing running. This is what
<em>I</em> call hard.</p>
<p>On Linux, there are a couple of hardware interfaces exported to
userspace. I already mentioned <a class="reference external" href="http://www.libusb.org/">userspace USB</a>. There is a similar thing to implement USB
<em>devices</em> in user space, called <a class="reference external" href="http://www.linux-usb.org/gadget">GadgetFS</a>.</p>
<p>For others, look inside the kernel documentation, part of the kernel
source, in the <code class="docutils literal notranslate"><span class="pre">Documentation/</span></code> directory. (Usually, the kernel
source is installed in <code class="docutils literal notranslate"><span class="pre">/usr/src/linux/</span></code>.) Or simply ask Google.</p>
<p>For example, check out the userspace I2C and SPI interfaces described
in <code class="docutils literal notranslate"><span class="pre">Documentation/i2c/dev-interface</span></code> and
<code class="docutils literal notranslate"><span class="pre">Documentation/spi/spidev</span></code>.</p>
</section>
<section id="realtime">
<h2><a class="toc-backref" href="#id9" role="doc-backlink">Realtime</a></h2>
<p>Well, realtime … what’s that? To put it bluntly, Linux is realtime
capable.</p>
<p>By default, Linux schedules processes (“scheduled entities”, so to
say - threads as well as processes) <em>in a fair way</em>. This is, everyone
gets its fair share of CPU resources. As such, there are no guarantees
given as to when a process is scheduled, and whether its deadlines are
met.</p>
<p><strong>However:</strong></p>
<p>You can make any process realtime-scheduled, by calling
<code class="docutils literal notranslate"><span class="pre">sched_setscheduler()</span></code>. You can make threads realtime-scheduled,
selectively, by creating them with the appropriate attributes (see
<code class="docutils literal notranslate"><span class="pre">pthread_attr_setschedparam()</span></code>, and read the book :-).</p>
<p>Realtime-processes are scheduled <em>immediately</em> when they become
runnable (unless a higher priority process is runnable as well, of
course). And by immediate I mean immediate - and that’s what <em>I</em> call
realtime. No fairness involved anymore, definitely not nice
anymore. And therefore potentially harmful.</p>
<p>Basically, there’s two realtime scheduling policies to choose from:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">SCHED_FIFO</span></code>. The process runs until it puts itself to sleep
(waiting for something to happen), or until it is interrupted by a
process which has an even higher priority.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">SCHED_RR</span></code>. Processes in the same priority level alternate in a
round robin manner, with very short timeslices. As with
<code class="docutils literal notranslate"><span class="pre">SCHED_FIFO</span></code>, they are interrupted by higher priority processes.</p></li>
</ul>
<p>Check out the man pages, and read the book. No joke. Doing realtime
scheduling is crying for trouble. Even more so if code is involved
where you don’t know 100% exactly what it does. It might go into an
infinite loop, at best. At worst, it will trigger race conditions only
under certain circumstances, or bring up deadlock
situations. Naturally, this kind of errors does not show up during
in-house testing where you are able to inspect and fix them, but
rather months later, at your customer’s site.</p>
<p>That’s it been for now. I could possibly add a couple more items to
this list, right now. Deployment comes to mind (What’s a firmware
image in Linux? Will I be minimal? Do I use <a class="reference external" href="http://busybox.net/">BusyBox</a> and <a class="reference external" href="http://www.uclibc.org/">uClibc</a>?),
build issues (Shared libraries? Static? What, how, why?). I could add
another couple more items if I you give me a minute.</p>
<p>If you have read through this list, you’ll see what I mean: take care
to choose the right mechanisms. Don’t start coding immediately because
there’s no time left in the project. Thinking twice and investigating
will save you from spending even more time later in the project. (But
what am I telling you?)</p>
</section>
</section>
Processes are Threads2011-01-07T00:00:00+01:00https://www.faschingbauer.me/blog/2010/07/beagleboard-how-to-repair-nand-also-known-as-40w.htmlBeagleboard: How to Repair NAND (also known as 40W)2010-07-30T00:00:00+02:00Jörg Faschingbauer<section id="beagleboard-how-to-repair-nand-also-known-as-40w">
<aside class="sidebar">
<p class="sidebar-title">Contents</p>
<nav class="contents local" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#basics" id="id1">Basics</a></p></li>
<li><p><a class="reference internal" href="#getting-x-loader-and-u-boot" id="id2">Getting X-Loader and U-Boot</a></p></li>
<li><p><a class="reference internal" href="#creating-a-bootable-mmc-sd-card" id="id3">Creating a Bootable MMC/SD card</a></p></li>
<li><p><a class="reference internal" href="#repairing-the-nand" id="id4">Repairing the NAND</a></p></li>
<li><p><a class="reference internal" href="#writing-x-loader-to-nand" id="id5">Writing X-Loader to NAND</a></p></li>
<li><p><a class="reference internal" href="#writing-u-boot-to-nand" id="id6">Writing U-Boot to NAND</a></p></li>
</ul>
</nav>
</aside>
<p>It happened again. Friends of mine have an alias,</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">alias faschingbauer="rm -rf ~"</span>
</pre></div>
</div>
<p>in their <code class="docutils literal notranslate"><span class="pre">.bashrc</span></code> for a couple of years now, and now I created a
creative variation thereof.</p>
<p>I used to have my beagleboard running happily off an SD card, using a
root filesystem which I had updated natively from a <a class="reference external" href="http://mirrors.kernel.org/gentoo/releases/arm/autobuilds/current-stage3/armv4tl-softfloat-linux-gnueabi/">Gentoo ARM stage3</a>. This
is decadent, so I prepared a Busybox-only initramfs linked into the
kernel image (cross-built of course, as opposed to the stage3), with
the plans to use this as an alternative boot image residing in NAND
flash.</p>
<p>It happened to me that, during finding out the correct arguments of
U-Boot’s <code class="docutils literal notranslate"><span class="pre">nand</span> <span class="pre">erase</span></code> command, I hit the
return key early - expecting that a sole</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">nand erase</span>
</pre></div>
</div>
<p>would give me further instructions. U-Boot took it literally though
and erased NAND, just like I told it to. A reboot showed the fatal
consequences. “40W” was the only thing I saw over the serial terminal
(115200/8N1 by the way).</p>
<p>Here in the remainder I write down how to recover from such a
fauxpas. The internet has a lot of explanations, but none of them is a
walk-through from the beginning to the end. You’ll have to do a couple
of things, and most of the instructions you’ll find don’t explain
much. This is not what I like, so I researched a bit and write the
annotated collected instructions down right here. You might find it
useful - I surely will once I’ll brick my beagle again.</p>
<section id="basics">
<h2><a class="toc-backref" href="#id1" role="doc-backlink">Basics</a></h2>
<p>The OMAP has a small boot ROM inside. This is the first of the chain
of boot loaders, and it’s where the CPU core jumps to when it’s done
with whatever CPU cores do when they are powered on). The boot ROM is
programmed to load a second stage boot loader (the “X-Loader” as it is
called) into the processor’s SRAM. X-Loader is then responsible for
loading another stage - U-Boot in the Linux case - into the external
DRAM. It’s the latter two stages - X-Loader and U-Boot - that I
faschingbauered away from NAND, and that I’ll recover now.</p>
<p>They key to all that is the ability of the boot ROM to boot from a
MMC/SD card when the user button is pressed during its operation. We
will prepare one such card, put the boot files on it, and write these
to NAND.</p>
</section>
<section id="getting-x-loader-and-u-boot">
<h2><a class="toc-backref" href="#id2" role="doc-backlink">Getting X-Loader and U-Boot</a></h2>
<p>The beagleboard guys wrote up a page with <a class="reference external" href="http://code.google.com/p/beagleboard/wiki/BeagleboardRevC3Validation">board validation
instructions</a>. This
is where I have most of the information from. There they give links to
the images that I want. Take care that you select the ones that match
your board revision; the files they use on that page are slightly
seasoned. I have a recent C4, and I found it valuable to have a recent
U-Boot version that enables power on the USB host controller. This was
a change they made from C3 to C4, for example - take care.</p>
<p>Anyway, <a class="reference external" href="http://www.angstrom-distribution.org/demo/beagleboard/">Angstrom</a> have the
most recent stuff. Download X-Loader and U-Boot from there, and
additionally another bootloader stage, <code class="docutils literal notranslate"><span class="pre">MLO</span></code>, which is used during
the MMC boot. The filenames change, just browse the directory a choose
something that sounds like <code class="docutils literal notranslate"><span class="pre">x-load.bin.ift</span></code>, <code class="docutils literal notranslate"><span class="pre">u-boot*</span></code>,
<code class="docutils literal notranslate"><span class="pre">MLO*</span></code>. Save them away for later use. I saved them as
<code class="docutils literal notranslate"><span class="pre">x-load.bin.ift</span></code> and <code class="docutils literal notranslate"><span class="pre">u-boot.bin</span></code>, and it’s these names that I
refer to later on.</p>
</section>
<section id="creating-a-bootable-mmc-sd-card">
<h2><a class="toc-backref" href="#id3" role="doc-backlink">Creating a Bootable MMC/SD card</a></h2>
<p>For basic card configuration I refer you to <a class="reference external" href="http://processors.wiki.ti.com/index.php/MMC_Boot_Format">a TI maintained page</a> which is
very concise and to the point, and where there is nothing much left to
say. I followed the instruction using fdisk, and obvoiusly the card
can be read by the boot ROM. There are reports that FAT32 does not
work on some cards, and that selecting FAT16 instead helped. You might
want to try that out in case.</p>
<p>Once the card is ready and a FAT partition has been created, you mount
that partition and put the files on it as follows.</p>
<ul class="simple">
<li><p>First comes <code class="docutils literal notranslate"><span class="pre">MLO</span></code>, as file <code class="docutils literal notranslate"><span class="pre">MLO</span></code>, no matter what name you stored
it under on your disk. It is read by the boot ROM, and that one is
very particular that the file is found in the first sectors.</p></li>
<li><p>You store the U-Boot image as file <code class="docutils literal notranslate"><span class="pre">u-boot.bin</span></code>. <code class="docutils literal notranslate"><span class="pre">MLO</span></code> expects
it under that name during MMC boot. (If you intend to use a
different U-Boot image for NAND then you store that one under a
different name which you use further down when we write NAND.)</p></li>
<li><p>The X-Loader is not used during MMC boot, so you can choose any name
you like.</p></li>
</ul>
<p>Insert the card into the MMC/SD slot of the beagle, and boot. In case
you erased your NAND only partly and there is still something
meaningful in it, be sure to hold down the user button to force the
boot ROM to boot from the card.</p>
</section>
<section id="repairing-the-nand">
<h2><a class="toc-backref" href="#id4" role="doc-backlink">Repairing the NAND</a></h2>
<p>The following is taken from the <a class="reference external" href="http://beagleboard.googlecode.com/files/reset_revc_v3.scr">reset.scr</a> U-Boot
script referred to by the <a class="reference external" href="http://code.google.com/p/beagleboard/wiki/BeagleboardRevC3Validation">beagleboard validation instructions</a>. I
took care to annotate as much as possible in order to understand the
steps.</p>
<p>First off, we initialize MMC/SD.</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">OMAP3 beagleboard.org # mmc init</span>
<span class="go">mmc1 is available</span>
</pre></div>
</div>
<p>To understand where the numbers in the remainder instructions come
from, open the kernel source file
<code class="docutils literal notranslate"><span class="pre">arch/arm/mach-omap2/board-omap3beagle.c</span></code>. I’ll refer to that file
in the remainder explanations. Near the top of the file you’ll notice
the partition table; you’ll find it insightful.</p>
</section>
<section id="writing-x-loader-to-nand">
<h2><a class="toc-backref" href="#id5" role="doc-backlink">Writing X-Loader to NAND</a></h2>
<p>First, erase the X-Loader partition. According to the partition table,
it starts at offset 0 (where the OMAP’s boot ROM expects it) and is
4*128K=0x80000 bytes long.</p>
<p>The boot ROM relies on hardware to manage error correction code (ECC),
as described in the <a class="reference external" href="http://download.micron.com/pdf/technotes/nand/tn2916.pdf">tech note</a>, so we
select it.</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">OMAP3 beagleboard.org # nandecc hw</span>
<span class="go">HW ECC selected</span>
<span class="go">OMAP3 beagleboard.org # nand erase 0 80000</span>
<span class="go">NAND erase: device 0 offset 0x0, size 0x80000</span>
<span class="go">Erasing at 0x60000 -- 100% complete.</span>
<span class="go">OK</span>
</pre></div>
</div>
<p>According to the <a class="reference external" href="http://download.micron.com/pdf/technotes/nand/tn2916.pdf">tech note</a> again,
the OMAP’s boot ROM takes care of bad blocks (unlikely because an
intermediate bootloader isn’t written to flash that often, but
nevertheless it sounds like a good idea). It checks the first four
blocks of the X-Loader partition for a valid image, so we write the
image into these four blocks. Again, the erase block size is
determined by looking in <code class="docutils literal notranslate"><span class="pre">board-omap3beagle.c</span></code> - 128K = 0x20000.</p>
<p>Read the image into memory, and write it out 4 times, in the first
four erase blocks of the partition.</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">OMAP3 beagleboard.org # fatload mmc 0 80200000 x-load.bin.ift</span>
<span class="go">reading x-load.bin.ift</span>
<span class="go">20392 bytes read</span>
<span class="go">OMAP3 beagleboard.org # nand write 80200000 0 20000</span>
<span class="go">NAND write: device 0 offset 0x0, size 0x20000</span>
<span class="go">131072 bytes written: OK</span>
<span class="go">OMAP3 beagleboard.org # nand write 80200000 20000 20000</span>
<span class="go">NAND write: device 0 offset 0x20000, size 0x20000</span>
<span class="go">131072 bytes written: OK</span>
<span class="go">OMAP3 beagleboard.org # nand write 80200000 40000 20000</span>
<span class="go">NAND write: device 0 offset 0x40000, size 0x20000</span>
<span class="go">131072 bytes written: OK</span>
<span class="go">OMAP3 beagleboard.org # nand write 80200000 60000 20000</span>
<span class="go">NAND write: device 0 offset 0x60000, size 0x20000</span>
<span class="go">131072 bytes written: OK</span>
</pre></div>
</div>
</section>
<section id="writing-u-boot-to-nand">
<h2><a class="toc-backref" href="#id6" role="doc-backlink">Writing U-Boot to NAND</a></h2>
<p>Can’t find which ECC incarnation the X-Loader uses, software seems to
be the choice (sounds like the do-nothing-and-cross-fingers approach).</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">OMAP3 beagleboard.org # nandecc sw</span>
<span class="go">SW ECC selected</span>
</pre></div>
</div>
<p>According to the kernel sources, U-Boot’s partition starts at offset
0x80000 and is 15*128K=0x1e0000 bytes long.</p>
<p>Erase NAND, load U-Boot from card, write it to the partition.</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">OMAP3 beagleboard.org # nand erase 80000 1e0000</span>
<span class="go">NAND erase: device 0 offset 0x80000, size 0x1e0000</span>
<span class="go">Erasing at 0x240000 -- 100% complete.</span>
<span class="go">OK</span>
<span class="go">OMAP3 beagleboard.org # fatload mmc 0 80200000 u-boot.bin</span>
<span class="go">reading u-boot.bin</span>
<span class="go">275928 bytes read</span>
<span class="go">OMAP3 beagleboard.org # nand write 80200000 80000 1e0000</span>
<span class="go">NAND write: device 0 offset 0x80000, size 0x1e0000</span>
<span class="go">1966080 bytes written: OK</span>
</pre></div>
</div>
<p>That’s it. You are now able to boot without the card, up to the U-Boot
prompt.</p>
<p>Take care when you flash the kernel. In case you don’t know the
correct <code class="docutils literal notranslate"><span class="pre">nand</span> <span class="pre">erase</span></code> parameters, write</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">OMAP3 beagleboard.org # help nand</span>
</pre></div>
</div>
<p>instead of <code class="docutils literal notranslate"><span class="pre">nand</span> <span class="pre">erase</span></code>. I suggest you type</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">OMAP3 beagleboard.org # nand erase 280000 400000</span>
</pre></div>
</div>
<p>just like the kernel source says for the kernel partition.</p>
</section>
</section>
Basics2010-07-30T00:00:00+02:00