Building Today’s Yocto Tomorrow

Any software expected to have a long lifetime will require maintenance which in turn means that you need to be able to build the software again in the future. So how can you ensure that in 10 years time that you will even have a system that is capable of running today’s build system? How do you know you are not missing some tiny part that will stop the whole thing from working?

In this post we will discuss some of the problems that can be encountered over time in maintaining a Yocto distribution, how to ensure that you can build Yocto again in the future, and how using CI can be helpful.

Why Would the Build Fail Over Time?

The main reason that builds start to fail over time is due to external resources moving or just disappearing.

For instance we still regularly come across recipes where the SRC_URI specifies a repo on source.codeaurora.org which is a server that no longer exists. In this case, if you are lucky, a copy of that repo will be available somewhere else. It may have been moved to github, or someone happened to make a mirror of it somewhere. In this case it is easy enough to fix the problem just by changing SRC_URI, but if it has actually been deleted and there is no mirror available then that is potentially a big problem. Luckily, if you still have a complete copy of the downloads directory from a previously successful build, then there is actually no problem.

The build will also break if the Yocto layers specified in the kas configuration file are moved or deleted. In this case, you should have local copies of these repos from a previous build as they are downloaded at the start of the kas build process. But in any case, while the build is still working, it is wise to make a backup of these repos.

Another reason why builds would stop working is that the operating system on the build machine gets updated and then the Yocto build starts throwing up errors that you never saw before. We have seen this on a yocto project that is still based on the pyro branch of yocto and found that the build cannot run on ubuntu 20.04 or later. So rather than keep a machine with ubuntu 18.04 installed on it, the build was moved to a ubuntu 18.04 docker image. In general it is better to use some form of container technology or a VM for builds anyway as you can get a reproducible build environment.

The following sections provide recommendations for ensuring that your Yocto builds can continue to be used into the future.

Use Containers and Archive Them

A given version of Yocto is designed to be built against a range of supported distributions, and so builds can fail in the future if you use a newer distribution. Therefore it is a good idea to use containers for the build environment as the container images themselves can be backed up and will provide a consistent build environment. The most common containerisation tools are Docker and Podman.

Yocto Container Images and Tooling

The Yocto Project recommends the use of ‘CROSss PlatformS’ (CROPS), which is Docker container image designed for running Yocto builds.

A popular alternative solution is to use kas-container from the kas project. Kas-container (which can use either Docker or Podman under the hood) will download a suitable docker image containing a kas build environment. You can run typical kas commands such as ‘kas build‘ and ‘kas shell‘ as ‘kas-container build‘ and ‘kas-container shell‘. As builds are performed inside the container, there is no need to set up the kas environment on the host machine.

Furthermore, by using shared volumes or bind-mounts, the build outputs can be made available outside the container and the runtime container image can be deleted whenever it is stopped, meaning that the image will provide exactly the same environment next time it is run.

Archive Container Images

Regardless to where a container image has come from the ability to back up and restore container images is important as there is no guarantee that the image will still be available when the project needs to be rebuilt.

If you are using docker to run containers, docker images can be backed up to a local tar file as follows (docker back up and restore data) :

docker image save -o images.tar image1 [image2] ...

Then these images can be recovered from the tar file as follows :

docker image load -i images.tar

Other container technologies such as podman provide similar backup/restore functionality.

When using kas-container, you can specify a specific kas container with the KAS_IMAGE_VERSION environment variable, or provide a path to a local archived container via the KAS_CONTAINER_IMAGE environment variable.

Archive the Downloads Directory

One of the first stages of a Yocto build is to fetch all the required sources specified in the SRC_URI variable in recipes and the Yocto layers specified in the repos list of a kas configuration file.

To save time on builds, Yocto saves every repo and tarball that is fetched during the build in the directory specified in the DL_DIR environment variable. If a tarball was fetched, the tarball is stored in the DL_DIR directory along with a corresponding .done file, and if it is a git repo, a clone of the repo is in a directory under the DL_DIR/git2 directory, again with a corresponding .done file. Therefore, if DL_DIR starts off empty and you build the project from scratch, DL_DIR contains a complete backup of all the sources that were fetched during the build. Subsequent builds can use the source found in DL_DIR instead of downloading from SRC_URI. Yocto will also create a DL_DIR/uninative directory for the common native libraries it uses. Note that even if a recipe has local source, e.g. in a files directory under the recipe, Yocto will create a tarball of this in DL_DIR. Therefore it is recommended that the DL_DIR directory is archived.

Use BB_GENERATE_MIRROR_TARBALLS

Where SRC_URI in a recipe says to fetch from a git repo rather than downloading a source tarball, Yocto creates a bare clone of the repo under the DL_DIR/git2 directory, but for backup purposes, it is easier to manage a single tarball rather than a clone directory. This can be achieved by asking Yocto to create tarballs from the cloned repository in the downloads directory by setting the BB_GENERATE_MIRROR_TARBALLS variable “1” in the local build configuration or kas local_conf_header. The git:// urls used in Yocto recipes will work with either the tarball or the git clone directory.

For example, the recipe file the file tool, poky/meta/recipes-devtools/file/file_5.45.bb has the following SRC_URI :

git://github.com/file/file.git;branch=master;protocol=https

Since we have BB_GENERATE_MIRROR_TARBALLS = "1", do_fetch of the file recipe will create a clone directory under downloads/git2 called github.com.file.file.git (and github.com.file.file.git.done) and a corresponding tarball in downloads called git2_github.com.file.file.git.tar.gz (and git2_github.com.file.file.git.tar.gz.done) and the above git:// url still works now even if we delete the github.com.file.file.git directory in downloads. And it will still work if you install the tarball on a local mirror.

Generate DL_DIR Contents

To get a complete set of tarballs into the downloads directory it is necessary to do a build from scratch, and the easiest way to make this happen is just to delete any sstate and start the build again. To just create the all the tarballs without doing a complete build, delete the sstate directory and run the following command which will run the do_fetch task for every recipe involved in the relevant Yocto image:

bitbake core-image-minimal --runall=fetch

Once the complete set of tarballs has been created (and their associated .done files) the entire downloads directory can be archived. If you have set BB_GENERATE_MIRROR_TARBALLS, you can exclude the git2 directory from your archive.

Archive Yocto Layers

Another important part of the backup is the list of metadata repos that are downloaded by kas (or similar tool such as repo) because these repos themselves can be moved or can disappear. This is easy to achieve with kas by using the ‘kas checkout‘ command which fetches all the external repos specified in the configuration files and checks them out to the specified commit. If you make sure that all the external Yocto layers are placed under a single directory, then backing them up is simply a matter of making a tarball of that directory. For example, the path: lines here place poky and meta-openembedded under a directory called layers :

repos:
   meta-openembedded:
     url: "https://github.com/openembedded/meta-openembedded"
     path: layers/meta-openembedded
     layers:
       meta-oe:
 poky:
     url: "https://git.yoctoproject.org/git/poky"
     path: layers/poky
     layers:
       meta:
       meta-poky:
 ...

For more complex projects with multiple configuration files and included files, it may be necessary to run kas checkout on all the top level files to ensure that all external repos are fetched.

An alternative solution is to mirror these repositories locally within your organisation and update the kas file to use the mirror versions instead.

Use Premirrors

In a larger organisation it may be appropriate to set up a local mirror server, e.g. an http server, that can serve all the tarballs/repos needed by the do_fetch tasks. In addition to removing a dependency on the internet, this can also result in faster builds. Note that this is not the same as DL_DIR and if using mirrors, the tarballs still get downloaded from the mirror server to DL_DIR

Yocto searches DL_DIR first, then it searches PREMIRRORS which is a list of servers where sources may be found, failing that it tries the SRC_URI in the recipe and as a last resort tries the MIRRORS list. The mirror server can be prepended to PREMIRRORS so it is tried first, add the following in the local_header_conf section of the kas config file :

PREMIRRORS:prepend = "git://.*/.* http://local.network.server/mirror/sources/ "

So now, if not already in DL_DIR, Yocto will convert any git:// urls it finds in a recipe to http:// urls pointing at the local server. Further lines can be added for different type of urls that may be found in recipes, e.g. for https:// urls, add this :

PREMIRRORS:prepend = "https://.*/.* http://local.network.server/mirror/sources/ "

Note that using PREMIRRORS doesn’t fix the problem of disappearing repos unless you generate the tarballs while the build is still working – then it’s ok for the repos to disappear from the internet!

Pin Source Revisions

Some recipes use SRCREV = "${AUTOREV}" to make sure that source is fetched from HEAD of the relevant repository. For instance a u-boot recipe encountered recently contained this :

 SRCBRANCH = "u-boot-compulab_v2023.04"
 SRC_URI = "git://github.com/compulab-yokneam/u-boot-compulab;protocol=https;branch=${SRCBRANCH}"
 SRCREV = "${AUTOREV}"

This caused problems as we were also patching this version of u-boot, and as HEAD moved on in the source repo, this eventually broke some of the patches, i.e. they would no longer apply cleanly or at all. This can be fixed by pinning SRCREV to a particular git revision in a .bbappend file as in the following example :

 # Fixed commit on 2023.04 branch
 SRCREV = "3584328bd07ad0af364ad6c077461a58a7746d4a"

Similarly, it is a good idea to pin the version of metadata specified in the kas file. The repos section of the kas config file indicates the url of the Yocto layer repo to fetch, and for instance the default branch to be fetched for all repos is set to “scarthgap” :

 defaults:
   repos:
     branch: "scarthgap"
 repos:
   meta-openembedded:
     url: "https://github.com/openembedded/meta-openembedded"
     path: layers/meta-openembedded
     layers:
       meta-oe:
   poky:
     ...

With this configuration you will get something from the scarthgap branch but you don’t know exactly what, so you can specify an exact commit ref by adding this under the url line :

     commit: "491671faee11ea131feab5a3a451d1a01deb2ab1"

With kas 4.6 and later, there is now a ‘kas lock‘ command which pins every repo in the repos list (if they are not pinned already) to a specific commit. For instance if the kas config file is test.yml, then the command is simply :

kas lock test.yml

The resulting lock file will be test.lock.yml in the same directory as test.yml, and this contains two commit lines which are both the HEAD of the scarthgap branch in the respective repos when they were downloaded by kas :

 header:
     version: 14
 overrides:
     repos:
         meta-openembedded:
             commit: 491671faee11ea131feab5a3a451d1a01deb2ab1
         poky:
             commit: 9b0e95db584602a13e46ed86b73930506b13e4a9

If you have already specified a commit for a repo in the configuration file, then this commit will appear in the lock file rather than using the HEAD commit. Note that the lock file should be added to your git repo. You may prefer to specify the commit yourself at the point where you have decided to archive the project.

Verify the Build

With the complete set of tarballs in build/downloads, you can force a build from these tarballs by deleting the sstate directory (SSTATE_DIR) and the build/tmp directory and setting BB_NO_NETWORK to "1". Doing so will allow you to verify that you can build without depending on sources from the internet.

BB_NO_NETWORK disables network access in bitbake fetcher modules, and effectively causes any command that tries to access the network to fail with an error. In theory, only the fetcher modules require network access and network access is disabled by default for the rest of the build.

To verify a build where you are using premirrors on a local network then you can’t use BB_NO_NETWORK and in this case you would need to limit the PREMIRRORS to just your local machine. Alternatively, instead of using BB_NO_NETWORK you can use the BB_ALLOWED_NETWORKS variable to contain only hosts that you want to access. Perhaps a better solution is to use BB_FETCH_PREMIRRORONLY which when set to "1" causes bitbake’s fetcher module to only search PREMIRRORS for files (although note that it still searches DL_DIR first). Bitbake will not search the main SRC_URI or MIRRORS.

If you’ve archived a build and are able to do a clean build successfully offline (in a container), then it’s likely that you will be able to continue building the distribution into the future.

Use CI to Spot New Issues

Being able build today’s Yocto distribution in the future, with the help of archives is helpful. However, if you plan to rebuild it, then it’s likely that you also plan to make changes. You’ll find that in order to use later versions of packages or layers, you’ll have to fix up out of date SRC_URIs, likewise you may also find it convenient to use a more modern build container. Of course, the longer the distribution is left to bit rot, the more effort will be required. Therefore, by using a CI system like GitHub Actions, you can schedule a build to run periodically, and without archives – it will then email you if a build fails, in which case you have an opportunity to deal with the problem before it gets out of control.

It’s also helpful to have a CI pipeline that will build Yocto which depends on your archives and uses BB_NO_NETWORK – thus ensuring that you can in fact build this on something that isn’t your local machine. One of the incidental benefits of using a CI system is that it documents how to build whatever artifact you are trying to produce. For instance, with GitHub actions, CI workflows are part of the git repo and are yaml files stored in the .github/workflows directory underneath the top directory. This is a useful form of backup as it automatically retains the knowledge of how to build the artifacts as part of the repo.

It’s also possible to set up a CI pipeline that is used to (re)generate an archived build, thus ensuring that despite ongoing development, you have at any point in time the confidence that you can perform an offline build without depending on the wider internet.

Assuming that the metadata revisions are pinned and the SRC_URI in all recipes are pinned, then the most likely reason for a backup CI build to fail is that a fetch has failed. If this is because a repo has been moved, then a bbappend file can be added for the offending recipe to override SRC_URI. If it is because a repo has been deleted, then you have a copy of the tarball for this repo, and you can modify the backup CI workflow to prepopulate DL_DIR with this tarball before starting the build.

Write Documentation

The documentation that goes with an archived project is important as whoever is tasked with rebuilding your project at some point in the future may have no idea about what it is or about any of the tools needed to build it, and there may not be anyone around with any helpful technical knowledge of the project.

The use of CI is valuable in that it demonstrates (through scripts) how a project can be built from scratch (and sometimes also booted and tested in the case of continuous testing).

Documentation should be provided which includes how to obtain relevant archive material, build the software from the archive, boot the software on a device and validate that it works. It should also include notes on how to recreate the archive in the future.