Yocto is an excellent distribution builder, however building any Linux distribution from scratch is resource intensive and may take a frustrating long time. This blog posts examines some of the most common techniques that can be used to reduce build times (and resource use).
Downloads
During a build, bitbake fetches source code from the internet as described by the SRC_URI variable in the individual recipes that make up an image. A basic core-image-sato image (Kirkstone) requires approximately 6 gigabytes of data from ‘upstream project releases‘ – this is a term Yocto uses for describing tarballs or other compressed archives (as opposed to source from revision control systems such as GIT). Bitbake will download this data to the path pointed to by the DL_DIR variable – by default this is the downloads directory inside the build directory (${TOPDIR}/downloads). The next time bitbake wishes to unpack the source it will first look in this directory.
If you have multiple builds on your machine, then by default each will have a downloads directory with very similar (duplicated) contents. Likewise, if you remove your build directory to test a clean build, then bitbake will need to re-download everything. In these scenarios you can reduce build times by having a machine global download directory that all builds can make use of. This is simple to achieve by setting the DL_DIR variable in your local.conf file to a path outside your build directory, e.g.
DL_DIR = "/var/cache/yocto_downloads"During a build, bitbake will also obtain source from ‘source control managers‘ – this is Yocto’s term for revision control systems such as Git and SVN. By default bitbake doesn’t store this source in the DL_DIR – however it is possible to configure Yocto to do so via the following addition to your local.conf:
BB_GENERATE_MIRROR_TARBALLS = "1"BB_GENERATE_MIRROR_TARBALLS will instruct bitbake to create an archive for each source that it checks out and place that archive in the downloads directory. For subsequent builds if the source is needed it will first look in the downloads directory for an archive before attempting to fetch it via a source control manager. A core-image-sato image (Kirkstone) requires more than 3 gigabytes of source obtained via source control managers.
Making use of DL_DIR and BB_GENERATE_MIRROR_TARBALLS can improve the time it takes to build Yocto by reducing the amount of time taken to obtain sources from the internet (effectively an offline build). It also greatly improves the reliability and reproducibility of builds – as its not that uncommon for a SRC_URI to point to a URL that is temporarily unavailable (or hosted on a slow server).
It’s also worth pointing out that you can ask bitbake to simply download all the sources needed up front, thus providing a way to populate the DL_DIR without having to build anything. This is achieved via the ‘runonly‘ bitbake option:
bitbake core-image-sato --runonly=fetchIt’s possible to test that you have downloaded all the required sources for a build by adding the BB_NO_NETWORK option option to your local.conf and then rebuilding. This disables network access and thus results in an error if source is needed that is not already downloaded.
BB_NO_NETWORK = "1"Shared State Cache
Yocto has a sophisticated mechanism for caching outputs of individual bitbake tasks and associating them with a hash which represents the task’s inputs. By keeping track of such data Yocto can improve the speed of subsequent builds by reusing previously built outputs instead of building them again. Yocto stores this data in the path pointed to by the SSTATE_DIR variable – by default this is the sstate-cache directory inside the build directory (${TOPDIR}/sstate-cache). A core-image-sato image (Kirkstone) results in more than 4 gigabytes of sstate-cache.
Much like the DL_DIR, the shared state cache can also be moved outside of the build directory to a machine wide directory that can be shared by all builds. This can significantly improve the speed of a build (and reduce disk space needed) – it is configured as follows:
SSTATE_DIR = "/var/cache/yocto_sstate"Where Yocto identifies that there is an sstate object for a given task, it will substitute the task with an alternative ‘setscene‘ variant of the task. This task will replicate the end result of the substituted task – e.g. by copying the cached build artifacts to the expected location. During a build, you may notice these setscene tasks, as shown in the following output:
Sstate summary: Wanted 2756 Local 117 Mirrors 0 Missed 2639 Current 0 (4% match, 0% complete)
NOTE: Executing Tasks
Setscene tasks: 2652 of 2756
Currently 7 running tasks (0 of 6613) 0% | |
0: swig-native-4.0.2-r0 do_deploy_source_date_epoch_setscene - 0s (pid 1092726)
1: zstd-native-1.5.2-r0 do_deploy_source_date_epoch_setscene - 0s (pid 1092732)
2: flex-native-2.6.4-r0 do_deploy_source_date_epoch_setscene - 0s (pid 1092733)
3: libical-native-3.0.14-r0 do_deploy_source_date_epoch_setscene - 0s (pid 1092735)
4: icu-native-70.1-r0 do_deploy_source_date_epoch_setscene - 0s (pid 1092734)
5: libidn2-native-2.3.2-r0 do_deploy_source_date_epoch_setscene - 0s (pid 1092740)
6: libgpg-error-native-1.44-r0 do_deploy_source_date_epoch_setscene - 0s (pid 1092762)As these tasks are not resource intensive, you’ll notice that they execute quickly – much quicker than the original task they replaced. You may also notice the text “Local 117” in the “Sstate summary: ” line of the above output – this indicates that Yocto identified 117 tasks that will be substituted with setscene tasks.
A clean build with a pre-existing sstate cache will likely complete very rapidly as most of the tasks will not need to be executed, instead Yocto will reuse the existing outputs from the sstate cache – thus highlighting the benefit of having a sstate cache shared for all builds on your local machine.
Other Peoples Sources
The Yocto features we’ve discussed so far improves build time on the basis that you keep a cache of data that you’ve obtained or created for later use. But this means that the first build of something new will always take some time (as the cache doesn’t exist yet). Yocto provides some features that allow for making use of other people’s caches. This can be especially beneficial where each engineer in a team wishes to build something that another engineer or continuous integration server has already built.
It’s possible to share a downloads directory across a network, this is achieved via a source mirror and may be beneficial if 1) you want to ensure you will always have a copy of source originally obtained from the internet, 2) you wish to have a local mirror that is on your corporate network (and thus faster) or 3) to simply to mitigate against unreliable servers. This is achieved by adding the following to your local.conf:
SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
INHERIT += "own-mirrors"This tells Yocto that, after looking in the DL_DIR, look for sources in the SOURCE_MIRROR_URL before anywhere else. The URL shown above is on the local filesystem, however it is commonly on a HTTP server. When Yocto fetches sources it does so in the following order: DL_DIR (i.e. is it already on the local machine), PREMIRRORS (i.e. is there somewhere you’d prefer Yocto to get the sources from), the upstream source (i.e. SRC_URI), MIRRORS (i.e. somewhere to look if the official SRC_URI fails). The own-mirrors bbclass is a short hand way of using SOURCE_MIRROR_URL to specify a PREMIRROR. The full contents of the own-mirrors.bbclass follows:
PREMIRRORS:prepend = " \
cvs://.*/.* ${SOURCE_MIRROR_URL} \
svn://.*/.* ${SOURCE_MIRROR_URL} \
git://.*/.* ${SOURCE_MIRROR_URL} \
gitsm://.*/.* ${SOURCE_MIRROR_URL} \
hg://.*/.* ${SOURCE_MIRROR_URL} \
bzr://.*/.* ${SOURCE_MIRROR_URL} \
p4://.*/.* ${SOURCE_MIRROR_URL} \
osc://.*/.* ${SOURCE_MIRROR_URL} \
https?://.*/.* ${SOURCE_MIRROR_URL} \
ftp://.*/.* ${SOURCE_MIRROR_URL} \
npm://.*/?.* ${SOURCE_MIRROR_URL} \
s3://.*/.* ${SOURCE_MIRROR_URL} \The contents of a mirror are exactly those of a DL_DIR (except it’s read only to Yocto), thus it can be created trivially by building a Yocto image with the ‘–runonly=fetch’ Bitbake argument and moving the DL_DIR to a webserver.
It’s worth pointing out that by default Yocto has a list of MIRRORS and PREMIRRORS – the full list can be seen in the mirrors.bbclass. Of course on a corporate network, it may be beneficial to have an office local PREMIRROR.
Other Peoples Shared State Cache
The sstate-cache is portable and thus can be shared with others in the form of an sstate mirror. Depending on the specification of your build machine and network connection – it may be quicker to download someone else’s pre-built artifacts than to build them locally.
Much like the download mirrors, Yocto will always look at the local sstate-cache first (as described by SSTATE_DIR) before using a mirror. When it does use a mirror that state will be copied to the local sstate-cache such that subsequent builds will use the local sstate cache. You can tell Yocto about a sstate cache mirror by using the SSTATE_MIRRORS option in your local.conf:
SSTATE_MIRRORS ?= "file://.* https://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"Much like the DL_DIR, you can easily create a sstate mirror by sharing the contents of your sstate-cache directory. However there is some complexity that should first be understood.
When Yocto completes a build task it creates a stamp file (usually in the tmp/stamps directory) – this speeds up incremental builds as Yocto can use these stamps to determine if a task needs to be re-run or not. The stamps include a hash (also called signature), which broadly is a representation of the tasks inputs – when the inputs change the hash changes – Yocto will detect this and know to rerun the task. Determining how to generate a hash isn’t trivial and as a result there are many different ways – these are called signature handlers and is configurable by the BB_SIGNATURE_HANDLER variable in the local.conf.
Prior to Dunfell the default handler used in Poky was OEBasicHash. However since then the OEEquivHash has been used instead – this makes use of a feature known as Hash Equivalence. It’s possible that different task inputs can result in the same output (the example given is changes to whitespace in source files) – in these cases it doesn’t make sense to rebuild the package or rebuild packages that depend on it. Hash Equivalence uses a database (hashserv.db) to keep mappings between the hashes of the task inputs and the hash of the output that is generated – this allows subsequent builds to identify that even though inputs to a task have changed there is no need to rebuild. In essence Hash Equivalence reduces build times for incremental builds by minimising the amount of work that needs to be done after changes to source.
However using OEEquivHash requires a Hash Equivalence server (bitbake-hashserv) – there are two variables that can be configured in the local.conf to configure this. The BB_HASHSERVE variable specifies which server to connect to, however if set to “auto” (which is the default in Poky), then Yocto will start a local server automatically when needed. When BB_HASHSERVE is set to “auto”, the BB_HASHSERVE_UPSTREAM variable can be used to specify a remote hash server – though much like the DL_DIR, Yocto will favour using the local server before the remote.
It’s worth noting that Yocto provides a public sstate mirror and in the past some BSP vendors have made available downloadable sstate caches (e.g. Xilinx). The following configuration will use the sstate mirror and hash equivalence server from the Yocto Project Auto Builder:
BB_SIGNATURE_HANDLER = "OEEquivHash"
BB_HASHSERVE = "auto"
BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
SSTATE_MIRRORS ?= "file://.* https://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"When we tried the above mirror we noticed the following output during a build:
Sstate summary: Wanted 2756 Local 0 Mirrors 1102 Missed 1654 Current 0 (39% match, 0% complete)Thus showing that 1102 tasks could be substituted for setscene tasks from the remote mirror.
When we initially tried using this mirror, the Hash Equivalence server was down – we quickly realised that when using the OEEquivHash it’s not possible to use an sstate mirror without a Hash Equivalence server. However we found that if we switched to OEBasicHash and set BB_HASHSERVE to “” we managed to get 577 hits from the same mirror. Finally it’s worth pointing out that the Yocto project plans to replace typhoon.yocto.io with hashserv.yocto.io in the coming days.
We also noticed that the Hash Equivalence server stores its data in a hashserv.db file, this is normally in the ${TOPDIR}/cache directory – along with other bitbake cache files. We think it makes sense to also share this across builds on a local machine. This can be achieved by adding PERSISTENT_DIR to your local.conf file:
PERSISTENT_DIR="/var/cache/yocto_persistent"Finally, when Yocto builds native packages (e.g. for the build host) it will make use of host libraries (such as glibc) – as a result sstate cache for native packages are specific to the build host. Yocto identifies the host and stores it in the NATIVELSBSTRING (type bitbake -e | grep NATIVELSBSTRING= to see it) – any native sstate objects are stored under a subdirectory of the same name in the sstate cache. To make it easier to share sstate for native tasks, the Poky distro inherits the uninative.bbclass – this results in Yocto downloading a pre-built C library and sets NATIVELSBSTRING to universal – thus isolating host specific differences.
In summary, you can improve the speed of your build by using a remote sstate-mirror and matching hash equivalence server – though the benefits are probably most noticeable where you can set up a CI system on your local or corporate network which matches your build, thus providing greater coverage.
Site Configuration
For optimal build times, we recommend a machine wide local download cache (with generated tarballs) and shared state cache (with local equivalence server). If you are working in an office (or have a high bandwidth connection to a machine somewhere) then we also recommend a source pre-mirror (with generated tarballs) as this may be quicker and more reliable. And finally if you are working on a project with others then an upstream shared sstate mirror (with equivalence server) will help to improve build times – though this may be suited to situations where you can support a CI system.
It also helps to put this configuration in a configuration file that is shared by all your builds. We’ve refer to the local.conf file in this blog post, however Yocto actually reads configuration from three files in a the following order: site.conf (organisation wide), auto.conf (for autobuilders) and local.conf (for local users). As such you can create a site wide file, e.g.:
# Machine wide download cache
DL_DIR = "/var/cache/yocto_downloads"
# Generate tarballs from fetches from revision control
BB_GENERATE_MIRROR_TARBALLS = "1"
# Attempt to get source from this mirror after looking in DL_DIR, but before SRC_URI
SOURCE_MIRROR_URL ?= "http://example.com/source-mirror"
INHERIT += "own-mirrors"
# Machine wide shared state cache
SSTATE_DIR = "/var/cache/yocto_sstate"
# Use Hash Equivalence (probably using it by default anyway)
BB_SIGNATURE_HANDLER = "OEEquivHash"
BB_HASHSERVE = "auto"
# Use upstream shared state cache mirror after looking in SSTATE_DIR
BB_HASHSERVE_UPSTREAM = "example.com:8687"
SSTATE_MIRRORS ?= "file://.* https://example.com/sstate-mirror/PATH;downloadfilename=PATH"
# Use a shared persistent cache
PERSISTENT_DIR="/var/cache/yocto_persistent"Finally it may be helpful to have a single copy of site.conf and use symbolic links in the build/conf directory to point to it.
We’ve covered a lot here, but we’ve probably only scratched the surface – we may have even got something wrong. If you have any suggestions feel free to send them our way.