The latest Long Term Support (LTS) version of Yocto was released last month with the code name “Scarthgap” – presumably named after a mountain pass in the Lake District national park. It promises 4 years of support, meaning regular point releases with security and bug fixes. In the context of increasing regulatory requirements for cyber and device security (for example the European Cyber Resilience Act), the latest LTS is an attractive starting point for any new development. However, the new release brings some changes in its handling of security vulnerabilities (CVEs) and therefore we thought we’d provide an overview of how it now works and what’s changed.
The Yocto Project has the capability, thanks to it’s cve-check class, to compare the list of software packages it’s building with a widely used database of known software vulnerabilities maintained by the National Institute of Standards and Technology (NIST). This means that when you build a Yocto distribution, it’s trivial to find out what vulnerabilities are present and if they’ve been patched. Let’s see how that’s done.
Building Yocto Scarthgap
Let’s start by checking out Scarthgap and configuring a build environment:
$ git clone --single-branch --branch scarthgap git://git.yoctoproject.org/poky
$ cd poky
$ source oe-init-build-envLet’s also add the following lines to our build/conf/local.conf directory:
BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws"
SSTATE_MIRRORS ?= "file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH"
BB_HASHSERVE = "auto"
BB_SIGNATURE_HANDLER = "OEEquivHash"This will tell Yocto to use publicly available pre-built artifacts rather than build them from scratch resulting in significantly shorter build times. Let’s now build the reference core-image-sato image:
$ bitbake core-image-sato
$ ls tmp/deploy/images/qemux86-64/
bzImage core-image-sato-qemux86-64.rootfs.cve
bzImage--6.6.23+git0+f7f00b22ef_2d01bc1d4e-r0-qemux86-64-20240502131547.bin core-image-sato-qemux86-64.rootfs.ext4
bzImage-qemux86-64.bin core-image-sato-qemux86-64.rootfs.json
core-image-sato-qemux86-64.rootfs-20240503130009.cve core-image-sato-qemux86-64.rootfs.manifest
core-image-sato-qemux86-64.rootfs-20240503130009.ext4 core-image-sato-qemux86-64.rootfs.qemuboot.conf
core-image-sato-qemux86-64.rootfs-20240503130009.json core-image-sato-qemux86-64.rootfs.spdx.tar.zst
core-image-sato-qemux86-64.rootfs-20240503130009.manifest core-image-sato-qemux86-64.rootfs.tar.bz2
core-image-sato-qemux86-64.rootfs-20240503130009.qemuboot.conf core-image-sato-qemux86-64.rootfs.testdata.json
core-image-sato-qemux86-64.rootfs-20240503130009.spdx.tar.zst modules--6.6.23+git0+f7f00b22ef_2d01bc1d4e-r0-qemux86-64-20240502131547.tgz
core-image-sato-qemux86-64.rootfs-20240503130009.tar.bz2 modules-qemux86-64.tgz
core-image-sato-qemux86-64.rootfs-20240503130009.testdata.json Detecting Vulnerabilities with cve-check
Now, here comes the fun part – lets use Yocto’s cve-check class to check for vulnerabilities. We do this by adding an INHERIT line to our conf/local.conf file and then rebuilding the core-image-sato image, as follows:
$ echo 'INHERIT += "cve-check"' >> conf/local.conf
$ bitbake core-image-sato
Loading cache: 100% | | ETA: --:--:--
Loaded 0 entries from dependency cache.
Parsing recipes: 100% |##############################################################################################################################################################################| Time: 0:00:17
Parsing of 922 .bb files complete (0 cached, 922 parsed). 1878 targets, 46 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "2.8.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "universal"
TARGET_SYS = "x86_64-poky-linux"
MACHINE = "qemux86-64"
DISTRO = "poky"
DISTRO_VERSION = "5.0"
TUNE_FEATURES = "m64 core2"
TARGET_FPU = ""
meta
meta-poky
meta-yocto-bsp = "scarthgap:3c9778fbc8ce22f3e85e13fa03d4380e1c491edc"
Checking sstate mirror object availability: 100% |###################################################################################################################################################| Time: 0:00:09
Sstate summary: Wanted 1257 Local 0 Mirrors 1253 Missed 4 Current 3446 (99% match, 99% complete)
Removing 2 stale sstate objects for arch qemux86_64: 100% |##########################################################################################################################################| Time: 0:00:00
NOTE: Executing Tasks
WARNING: nasm-native-2.16.01-r0 do_cve_check: Found unpatched CVE (CVE-2022-46456), for more information check /home/andy/projects/yocto/poky/build/tmp/work/x86_64-linux/nasm-native/2.16.01/temp/cve.log
WARNING: gnupg-2.4.4-r0 do_cve_check: Found unpatched CVE (CVE-2022-3219), for more information check /home/andy/projects/yocto/poky/build/tmp/work/core2-64-poky-linux/gnupg/2.4.4/temp/cve.log
WARNING: openssh-9.6p1-r0 do_cve_check: Found unpatched CVE (CVE-2023-51767), for more information check /home/andy/projects/yocto/poky/build/tmp/work/core2-64-poky-linux/openssh/9.6p1/temp/cve.log
WARNING: busybox-1.36.1-r0 do_cve_check: Found unpatched CVE (CVE-2023-42363 CVE-2023-42364 CVE-2023-42365 CVE-2023-42366), for more information check /home/andy/projects/yocto/poky/build/tmp/work/core2-64-poky-linux/busybox/1.36.1/temp/cve.log
WARNING: glibc-2.39+git-r0 do_cve_check: Found unpatched CVE (CVE-2010-4756), for more information check /home/andy/projects/yocto/poky/build/tmp/work/core2-64-poky-linux/glibc/2.39+git/temp/cve.log
WARNING: qemu-native-8.2.1-r0 do_cve_check: Found unpatched CVE (CVE-2019-12067 CVE-2021-20255 CVE-2023-1386), for more information check /home/andy/projects/yocto/poky/build/tmp/work/x86_64-linux/qemu-native/8.2.1/temp/cve.log
WARNING: linux-yocto-6.6.23+git-r0 do_cve_check: Found unpatched CVE (CVE-1999-0524 CVE-2008-4609 CVE-2010-4563 CVE-2019-14899 CVE-2021-3714 CVE-2021-3864 CVE-2022-0400 CVE-2022-1247 CVE-2022-38096 CVE-2022-4543 CVE-2023-3397 CVE-2023-3640 CVE-2023-4010 CVE-2023-6238 CVE-2023-6240 CVE-2023-6270 CVE-2023-6356 CVE-2023-6535 CVE-2023-6536 CVE-2023-7042 CVE-2024-0841 CVE-2024-21803 CVE-2024-23307 CVE-2024-23848 CVE-2024-24857 CVE-2024-24858 CVE-2024-24859 CVE-2024-24861 CVE-2024-24864 CVE-2024-25739 CVE-2024-25740 CVE-2024-26596 CVE-2024-26900 CVE-2024-26913), for more information check /home/andy/projects/yocto/poky/build/tmp/work/qemux86_64-poky-linux/linux-yocto/6.6.23+git/temp/cve.log
Image CVE report stored in: /home/andy/projects/yocto/poky/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0/deploy-core-image-sato-image-complete/core-image-sato-qemux86-64.rootfs-20240503154720.cve
Image CVE JSON report stored in: /home/andy/projects/yocto/poky/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0/deploy-core-image-sato-image-complete/core-image-sato-qemux86-64.rootfs-20240503154720.json
NOTE: Tasks Summary: Attempted 10123 tasks of which 9549 didn't need to be rerun and all succeeded.
Complete CVE report summary created at: /home/andy/projects/yocto/poky/build/tmp/log/cve/cve-summary
NOTE: Generating JSON CVE summary
Complete CVE JSON report summary created at: /home/andy/projects/yocto/poky/build/tmp/log/cve/cve-summary.json
Summary: There were 7 WARNING messages.There are a few elements of interest in the above output, firstly you can see that when unpatched vulnerabilities are detected they are displayed as warnings alongside their CVE numbers, here is an example of several vulnerabilities found in the busybox package:
WARNING: busybox-1.36.1-r0 do_cve_check: Found unpatched CVE (CVE-2023-42363 CVE-2023-42364 CVE-2023-42365 CVE-2023-42366), for more information check /home/andy/projects/yocto/poky/build/tmp/work/core2-64-poky-linux/busybox/1.36.1/temp/cve.logThese warnings are enabled by default but can be turned off via the CVE_CHECK_SHOW_WARNINGS variable. The build output also makes reference to the following generated files:
tmp/work/qemux86_64-poky-linux/core-image-sato/1.0/deploy-core-image-sato-image-complete/core-image-sato-qemux86-64.rootfs-20240503123354.cve
tmp/work/qemux86_64-poky-linux/core-image-sato/1.0/deploy-core-image-sato-image-complete/core-image-sato-qemux86-64.rootfs-20240503123354.json
tmp/log/cve/cve-summary
tmp/log/cve/cve-summary.jsonThe first two files, which start with the image name (core-image-sato) and have either the .cve or .json suffix, describe vulnerabilities that are known to exist in packages which have been included in the built image. This includes both patched and unpatched vulnerabilities. Note that these files are also duplicated in the tmp/deploy/images directory.
The second set of files, named cve-summary or cve-summary.json, and located in the tmp/log directory, describe the vulnerabilities that are known to exist (patched or otherwise) in packages that have been built as part of the generation of the built image. In other words, it includes not only packages present in the resulting image, but also native packages that were used to build the image. We care about this because it’s possible for issues in native packages to impact the security of the resulting image – for example code generation tools (i.e. compilers) may have bugs which introduce vulnerabilities in the built code (e.g. CVE-2008-1367 and CVE-2006-1902). Another example may be cryptographic tools generating artifacts that are not effective, perhaps because they have been based on weak randomness as was the case for CVE-2021-41117.
Whether you’re looking at the cve-summary or the image summary files, both provide the same data, either in JSON format or plain text, as shown below for text:
LAYER: meta
PACKAGE NAME: openssh
PACKAGE VERSION: 9.6p1
CVE: CVE-2023-51767
CVE STATUS: Unpatched
CVE SUMMARY: OpenSSH through 9.6, when common types of DRAM are used, might allow row hammer attacks (for authentication bypass) because the integer value of authenticated in mm_answer_authpassword does not resist flips of a single bit. NOTE: this is applicable to a certain threat model of attacker-victim co-location in which the attacker has user privileges.
CVSS v2 BASE SCORE: 0.0
CVSS v3 BASE SCORE: 7.0
VECTOR: LOCAL
VECTORSTRING: CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H
MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2023-51767And for JSON:
{
"id": "CVE-2023-51767",
"summary": "OpenSSH through 9.6, when common types of DRAM are used, might allow row hammer attacks (for authentication bypass) because the integer value of authenticated in mm_answer_authpassword does not resist flips of a single bit. NOTE: this is applicable to a certain threat model of attacker-victim co-location in which the attacker has user privileges.",
"scorev2": "0.0",
"scorev3": "7.0",
"vector": "LOCAL",
"vectorString": "CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H",
"status": "Unpatched",
"link": "https://nvd.nist.gov/vuln/detail/CVE-2023-51767"
}Every item in these files represents a vulnerability that is applicable to a package that has been built or included in the image. The various fields are obtained from the NVD database, however the ‘CVE STATUS’ or ‘status’ fields originate from Yocto and describe if the vulnerability has been patched in your build. This status will either be ‘Patched’, ‘Unpatched’ or ‘Ignored. The Yocto documentation nicely describes what these fields mean:
"The status Patched means that a patch file to address the security issue
has been applied. Unpatched status means that no patches to address the
issue have been applied and that the issue needs to be investigated.
Ignored means that after analysis, it has been deemed to ignore the issue
as it for example affects the software component on a different operating
system platform."Ignoring CVEs with CVE_CHECK_IGNORE / CVE_STATUS
There are lots of reasons why a vulnerability may be ignored, for example the vulnerability may only be applicable on other platforms, perhaps the vulnerability exists but isn’t a problem due to the configuration of a specific package – or maybe the vulnerability isn’t severe enough to justify a fix.
Yocto assumes that all vulnerabilities are unpatched unless it sees a patch that contains a CVE ID in its filename or has a tag in the patch. As for Ignored CVEs, Yocto knows about these through additional configuration – prior to Scarthgap this was achieved via the CVE_CHECK_IGNORE variable, where package or layer maintainers use it to indicate that a particular CVE doesn’t apply. To illustrate this, let’s look at a snippet of the openssh_8.9p1 recipe from Kirkstone:
# This CVE is specific to OpenSSH with the pam opie which we don't build/use here
CVE_CHECK_IGNORE += "CVE-2007-2768"
# This CVE is specific to OpenSSH server, as used in Fedora and Red Hat Enterprise Linux 7
# and when running in a Kerberos environment. As such it is not relevant to OpenEmbedded
CVE_CHECK_IGNORE += "CVE-2014-9278"
# CVE only applies to some distributed RHEL binaries
CVE_CHECK_IGNORE += "CVE-2008-3844"You’ll see above that the authors of this package have determined that some of the CVEs that apply to this particular version of OpenSSH can be ignored and thus have marked the CVE as such. They’ve also helpfully used comments to give their reasoning.
The problem with this is traceability, if you use cve-check to generate a list of CVEs for your product, how can you be confident that the ignored CVEs really should be ignored?, would you have to look over the code to get clues as to why it was marked as Ignored? Thankfully Scarthgap brings some changes in this area.
With Scarthgap the CVE_CHECK_IGNORE variable has been removed and is replaced with the CVE_STATUS variable, this provides a little more traceability. Let’s take a look at how the OpenSSH package has changed in Scarthgap:
CVE_STATUS[CVE-2007-2768] = "not-applicable-config: This CVE is specific to OpenSSH with the pam opie which we don't build/use here."
# This CVE is specific to OpenSSH server, as used in Fedora and Red Hat Enterprise Linux 7
# and when running in a Kerberos environment. As such it is not relevant to OpenEmbedded
CVE_STATUS[CVE-2014-9278] = "not-applicable-platform: This CVE is specific to OpenSSH server, as used in Fedora and Red Hat Enterprise Linux 7 and when running in a Kerberos environment"
CVE_STATUS[CVE-2008-3844] = "not-applicable-platform: Only applies to some distributed RHEL binaries."As you can see, rather than relying on the convention of developers adding their rationale in comments, the CVE_STATUS requires you to give a reason and a description. These reasons are mapped to the three top-level statuses (Patched, Unpatched, Ignored) found in the CVE files and is achieved via a mapping file in meta/conf/cve-check-map.conf, as follows:
# Possible options for CVE statuses
# used by this class internally when fix is detected (NVD DB version check or CVE patch file)
CVE_CHECK_STATUSMAP[patched] = "Patched"
# use when this class does not detect backported patch (e.g. vendor kernel repo with cherry-picked CVE patch)
CVE_CHECK_STATUSMAP[backported-patch] = "Patched"
# use when NVD DB does not mention patched versions of stable/LTS branches which have upstream CVE backports
CVE_CHECK_STATUSMAP[cpe-stable-backport] = "Patched"
# use when NVD DB does not mention correct version or does not mention any verion at all
CVE_CHECK_STATUSMAP[fixed-version] = "Patched"
# used internally by this class if CVE vulnerability is detected which is not marked as fixed or ignored
CVE_CHECK_STATUSMAP[unpatched] = "Unpatched"
# use when CVE is confirmed by upstream but fix is still not available
CVE_CHECK_STATUSMAP[vulnerable-investigating] = "Unpatched"
# used for migration from old concept, do not use for new vulnerabilities
CVE_CHECK_STATUSMAP[ignored] = "Ignored"
# use when NVD DB wrongly indicates vulnerability which is actually for a different component
CVE_CHECK_STATUSMAP[cpe-incorrect] = "Ignored"
# use when upstream does not accept the report as a vulnerability (e.g. works as designed)
CVE_CHECK_STATUSMAP[disputed] = "Ignored"
# use when vulnerability depends on build or runtime configuration which is not used
CVE_CHECK_STATUSMAP[not-applicable-config] = "Ignored"
# use when vulnerability affects other platform (e.g. Windows or Debian)
CVE_CHECK_STATUSMAP[not-applicable-platform] = "Ignored"
# use when upstream acknowledged the vulnerability but does not plan to fix it
CVE_CHECK_STATUSMAP[upstream-wontfix] = "Ignored"Of course, if the provided categories are not sufficient they can be easily updated. You may decide that certain vulnerabilities are impractical to exploit or perhaps not severe enough to bother fixing, in which case you could add new relevant categories.
The beautiful part about CVE_STATUS is that the additional rationale propagates right through to the cve-check generated files, for example, earlier in this post we discovered vulnerability CVE-2023-51767 in OpenSSH which was shown as unpatched. Let’s pretend that this isn’t an issue for us, and so we’ll add the following to our conf/local.conf:
CVE_STATUS[CVE-2023-51767] = “not-applicable-platform: Not relevant to our DRAM”Let’s see what happens when we retest for vulnerabilities:
LAYER: meta
PACKAGE NAME: openssh
PACKAGE VERSION: 9.6p1
CVE: CVE-2023-51767
CVE STATUS: Ignored
CVE DETAIL: not-applicable-platform
CVE DESCRIPTION: Not relevant to our DRAM
CVE SUMMARY: OpenSSH through 9.6, when common types of DRAM are used, might allow row hammer attacks (for authentication bypass) because the integer value of authenticated in mm_answer_authpassword does not resist flips of a single bit. NOTE: this is applicable to a certain threat model of attacker-victim co-location in which the attacker has user privileges.
CVSS v2 BASE SCORE: 0.0
CVSS v3 BASE SCORE: 7.0
VECTOR: LOCAL
VECTORSTRING: CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H
MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2023-51767As you can see the additional context and justification for ignoring the CVE has made it through to the generated CVE list, this can be really helpful when reviewing a product’s security.
As you can imagine there will be a number of CVEs known to the Yocto community that are known to be not applicable and can be ignored. Helpfully, these have been placed in conf/distro/include/cve-extra-exclusions.inc and can be included from your local.conf (or similar), the header of this file describes it’s purpose:
# This file contains a list of CVE's where resolution has proven to
# be impractical or there is no reasonable action the Yocto Project
# can take to resolve the issue. It contains all the information we
# are aware of about an issue and analysis about why we believe it
# can't be fixed/handled. Additional information is welcome through
# patches to the file.CVE_STATUS_GROUPS
Whilst reviewing CVEs for your platform, you may find that you choose to ignore multiple CVEs for the same reason, perhaps you’ve decided not to patch vulnerabilities that require physical access – in such a scenario Yocto provides the CVE_STATUS_GROUPS variable which allows you to provide the reason string in a single place. Let’s see how this works by adding the following to our conf/local.conf file:
CVE_STATUS_GROUPS += "CVE_STATUS_IGNORED_PHYSICAL"
CVE_STATUS_IGNORED_PHYSICAL = "CVE-2021-33656 CVE-2022-20423"
CVE_STATUS_IGNORED_PHYSICAL[status] = "not-applicable-config: agreed to ignore CVEs requiring physical access on 5/5/24"This time we’ve created a CVE_STATUS group named ‘CVE_STATUS_IGNORED_PHYSICAL’ and informed Yocto about it by adding it to the CVE_STATUS_GROUPS variable. We also defined CVE_STATUS_IGNORED_PHYSICAL by associating it with multiple vulnerabilities and provided a status reason and description.
Improving NVD Database Fetch
Yocto is aware of CVEs thanks to its use of the NVD’s public APIs, during a build it uses this API to populate a local sqlite3 database – however this can take a while (30 minutes or more). Thankfully there are ways to improve or control the process. By default Yocto doesn’t use an API key which results in larger delays between API requests – it’s trivial to obtain an API key from NVD and tell Yocto about this via the NVDCVE_API_KEY variable to improve download time.
For incremental builds, Yocto will use incrementally update the local database by using features of the NVD API that allow for requesting changes between a date range. Yocto uses the CVE_DB_UPDATE_INTERVAL variable to determine how often an update should be attempted – by default this is 24 hours but can be increased if this isn’t necessary (the NVD database updates once per day). If the database hasn’t been updated in over 120 days then Yocto will perform a full update, if you’d like to do a full update earlier than that (instead of incremental) then the CVE_DB_INCR_UPDATE_AGE_THRES can be used which was introduced in Scarthgap. You may wonder why you’d want to do this – this mailing list post, by the author of this feature may suggest that incremental update could result in missed updates.
We believe that the new CVE related features of Scarthgap add great value, particularly the ability to add context to CVE resolution decisions and to see that context in the generated cve-check output. This increased traceability is undoubtedly a great help for product makers trying to keep on top of vulnerabilities. In the coming days we’ll be adding support for Scarthgap to our ‘Galapagos’ CVE monitoring, reporting and notification service which will bring CVE resolution decisions directly into our reports.