With the complexity and dependence on software ever growing, it’s crucial to be able to perform over-the-air (OTA) updates to devices to provide fixes, features and patches for security vulnerabilities. However, a combination of cheap storage and complex software stacks has resulted in an increase in the size of software distributions thus requiring devices to download hundreds or thousands of megabytes for a single software update. This can be especially challenging for devices with mobile data connections or more generally for devices where internet connectivity is expensive and sometimes intermittent.
Fortunately there is a solution, offered by both SWUpdate and RAUC, and that is delta updates (adaptive updates in RAUC terminology). The rationale is that between two versions of software only a portion of that software will have actually changed – therefore it’s much more efficient to identify and download just those changed portions. The technical basis of the approaches taken by SWUpdate and RAUC are well covered in various online resources and presentations (here and here) – so in this post we’ll focus on how you can actually make use of delta updates in SWUpdate in the context of a Yocto distribution.
Let’s start by briefly explaining how SWUpdate implements delta support. Under the hood, SWUpdate makes use of zchunk – this essentially divides an input file such as a filesystem image into zstd compressed chunks and calculates the checksums of each chunk. It then outputs this to a single zchunk file (.zck) which consists of a header and the compressed chunks. Rather than including the filesystem image into the SWUpdate update file (.swu), the header of the zchunk file is included instead – incidentally this makes the update file very small as only the meta data of the filesystem image is sent.
Once the SWUpdate update file makes its way to the device, a delta handler in SWUpdate then uses zchunk to calculate checksums of the currently installed image – for typical A/B systems with read-only partitions this is likely to be the currently mounted image. SWUpdate can then compare the checksums of the current image with the checksums included in the SWUpdate update file. If there are any differences then it will download the missing chunks. The excellent thing about this approach is that when you prepare the update package, you don’t need to care about the version of software already running on the device – in other words it’s not like creating a diff with a binary diff tool and applying a patch. Instead you tell SWUpdate (via the zchunk header) what checksums should appear in which order, and SWUpdate will construct the new filesystem image from chunks of the current image and any downloaded chunks.
However, this approach means that you have to tell SWUpdate (via the sw-description file) a URL from which it can download the full zchunk file from. SWUpdate won’t download the entire file but will instead use HTTP range requests to only download the parts (i.e. chunks) of the file that are actually needed.
Now the theory is out of the way, let’s see this in action. At the time of writing, delta updates in SWUpdate work as expected – however there isn’t any integration of delta updates in the meta-swupdate layer. As a result many of the steps which follow are manual.
Install Host Tools
The first step is to install the zchunk utilities on the build machine. We downloaded the latest version from Github, built and installed it as per the instructions in the README.md. On our Ubuntu 20.04 machine we found that we needed to execute the “meson build” command with the “–prefix=/usr” argument as otherwise the zck utility failed to resolve the libzck library.
Create ZChunk Artifacts
We can now build the relevant zchunk related artifacts, as follows:
# Create ZCK file from uncompressed filesystem image
$ zck --output output.zck -u --chunk-hash-type sha256 core-image-base.ext4
# Determine the size of the ZCK header (we'll need to extract it)
$ HSIZE=`zck_read_header -v output.zck | grep "Header size" | cut -d ':' -f 2`
# Extract just the header (we'll include this in the swu package)
$ dd if=output.zck of=output.zck.header bs=1 count=$((HSIZE))This creates the zchunk file and extracts it’s header. Please note that only an uncompressed disk image can be used, this isn’t a problem as ZCK itself will compress the individual chunks.
Create SWU Package
We now need to construct a SWUpdate update file that contains the zck header instead of the disk image. However rather than create this from scratch, we’ll rely on Yocto to build one before manually modifying it, as follows:
# Extract the existing SWU package
$ mkdir unpacked && cd unpacked
$ cpio -idv < ../core-image-base.swu
# Replace the disk image with the ZCK header
$ rm core-image-base.ext4.gz
$ cp ../output.zck.header .
# Modify sw-description
# Rebuild the SWU (sw-description must be first in the archive)
$ ls -f sw-description output.zck.header | cpio -ov -H crc > ../zchunk.swuIt’s necessary to modify the sw-description to inform SWUpdate that we wish to perform a delta update. The following diff gives an indication of the changes required:
images: (
{
- filename = "core-image-base.ext4.gz";
- type = "raw";
- compressed = "zlib";
+ filename = "output.zck.header";
+ type = "delta";
device = "/dev/mmcblk0p2";
+ properties: {
+ url = "https://URL_HERE/output.zck";
+ chain = "raw";
+ source = "/dev/mmcblk0p3";
+ };
}
);As you can see we’ve changed the type from “raw” to “delta” and we’ve replaced the filename from the ext4.gz disk image to the zchunk header. We’ve then provided additional details to inform SWUpdate where it can download the full zchunk file from.
You’ll also see that it will write it’s image to mmcblk0p2 and use mmcblk0p3 as its reference image.
Yocto SWUpdate Delta and ZChunk Support
It’s necessary to configure SWUpdate to include support for the delta handler and support for the zchunk compression format (zstd) – for us this required that we modified the SWUpdate defconfig to include CONFIG_DELTA and CONFIG_ZSTD.
As SWUpdate now depends on the zchunk library we need to ensure that this package is included in the build, it’s worth pointing out that version 1.2.0 (or higher) of zchunk is required – thus depending on the version of Yocto this may need to be updated. It may be of interest that our testing was conducted with a 2021.11 version of SWUpdate.
Artifact Storage
Given that we’ve replace the filesystem image with a zchunk header file, we need to locate the full zchunk file somewhere that the device can access (as referenced from the url parameter of the sw-description file) so that missing chunks can be downloaded.
An obvious place may be Hawkbit – we uploaded an artifact and found that we can access it via it’s Management API (e.g. /rest/v1/softwaremodules/1/artifacts/1/download). However in most Hawkbit configurations HTTP requests to the Management API must include an Authorization header. Sadly the SWUpdate Delta Downloader handler doesn’t appear to offer support for this. So instead, for our testing, we located it on publicly accessible server – but of course this isn’t recommended for production due to security concerns.
Bringing it all together
Finally, we have all in the pieces in place and an update package ready to test. If you are using Hawkbit or similar then you may upload the package as you normally would. In our case we ran swupdate on the command line, see below for some of the key debug messages displayed:
$ swupdate -H raspberrypi4:1.0 -e stable,copy1 -f /etc/swupdate.cfg -i /tmp/zchunk2.swu
[TRACE] : SWUPDATE running : [network_initializer] : Software update started
[TRACE] : SWUPDATE running : [start_delta_downloader] : Starting Internal process for downloading chunks
[TRACE] : SWUPDATE running : [add_properties] : Found properties for output.zck.header:
[TRACE] : SWUPDATE running : [_parse_images] : Found Image: output.zck.header in device : /dev/mmcblk0p2 for handler delta
[TRACE] : SWUPDATE running : [install_single_image] : Found installer for stream output.zck.header delta
[TRACE] : SWUPDATE running : [install_delta] : ZCK Header read successfully from SWU, creating header from /dev/mmcblk0p3
[INFO ] : SWUPDATE running : [get_total_size] : Total bytes to be reused : 714226962
[INFO ] : SWUPDATE running : [get_total_size] : Total bytes to be downloaded : 10945313
[INFO ] : SWUPDATE running : [install_delta] : Size of artifact to be installed : 771751936
[TRACE] : SWUPDATE running : [install_single_image] : Found installer for stream output.zck.header raw
[TRACE] : SWUPDATE running : [trigger_download] : Range request : 1032757-1033347,1033876-1245079,1252472-1258732,...
[INFO ] : SWUPDATE running : [install_delta] : Total downloaded data : 10976914 bytes
[INFO ] : SWUPDATE running : [endupdate] : Swupdate was successful !As you can see from the abbreviated output SWUpdate identifies that the delta handler is to be used. It then reads the zchunk header and generates hashes from the existing image. It then describes how much data will be downloaded (which in this case is only 1.4% of the size of the new full image). It then proceeds to download the missing chunks via range requests and completes the update.
Summary
As you can see delta support in SWUpdate is available and fully functional. Though impeding its adoption is likely the lack of integration in the meta-swupdate Yocto layer and perhaps missing support for a HTTP Authorisation header in the delta download handler. If we have time or a willing customer then hopefully we can work on these missing pieces upstream.
However we believe delta updates are an excellent feature and particularly attractive to devices with mobile data connections where interruptions to downloads are problematic and where bandwidth is limited and expensive.