Earlier this year, the Embedded Linux Conference took place in Amsterdam as part of the Linux Foundation’s Open Source Summit Europe. As is tradition, we sent most of our team – to allow us to share and absorb knowledge, as well as to network with our counterparts in the ecosystem. Read on to find read about our favourite talks.
Navigating Security Tradeoffs in Embedded Linux Systems – Olivier Benjamin, Bootlin
As security becomes an increasingly critical concern in embedded systems due to new regulations (like the EU Cyber Resilience Act), reputational risk, and customer demand. Against this backdrop, this talk explores practical security strategies for embedded systems, emphasising that security exists on a spectrum rather than as a binary state. It begins by outlining how to define a threat model, identifying critical assets such as user data, intellectual property, cryptographic keys. Assessing potential attackers, their motivations and their capabilities.
Key security mechanisms covered include file system encryption (dm-crypt), which protects data at rest from physical attacks; secure boot, which ensures the bootloader and kernel are verified from a hardware root of trust to prevent unauthorised code; root filesystem verification (dm-verity), which checks the integrity of userland files using a hash tree, typically with a read-only filesystem and secure enclaves (ARM TrustZone) which isolate sensitive computations and secrets in hardware, protecting cryptographic keys, licenses, or biometric data.
See the schedule for the slides and further information.
Fail-Safe Embedded Linux: Designing for Power Resilience – Sergio Prado, Embedded Labworks
Embedded devices are often deployed in situations where the chance of unexpected power cuts are likely, which can lead to corruption of both the rootfs and application data. This talk by Sergio Prado of Embedded Labworks details the approaches that can be taken in order to reduce the probability of a power failure causing corruption to an embedded system. There is no “silver bullet”, and steps to mitigate against power failure should be considered at the hardware, operating system, and application level, and each of these approaches come with their own set of tradeoffs.
At the hardware level, combining a supercapacitor or battery backup with a suitable PMIC that supports brownout detection can be used to trigger a protective shutdown when the system detects a power loss from the main power source. Industrial grade eMMC chips should be chosen as these provide enhanced power-loss protection, and support additional power-loss protections such as Enhanced area and Write reliability.
Choosing a robust partition scheme is also important. Non-volatile data should be kept in a separate partition to volatile data, and furthermore critical and non-critical data should be separated too. By separating data in this way, we can minimise the surface over which a filesystem corruption can impact critical parts of the system. Additionally, tmpfs should be considered when working with volatile data, which will further reduce writes to the eMMC or flash. The partition scheme should be paired with a suitable filesystem depending on the storage medium in use.
At the application level, the talk discussed two main ways to mitigate against file corruption due to power loss. This includes reducing the vulnerability window by ensuring that any critical data that has been written to a file is periodically flushed from any buffers by periodically calling fsync(), as well as using techniques to ensure atomicity of file writes – this can be achieved by first writing out to a new temporary file, then renaming the temporary file over the old file (rename operations are atomic, as defined in the POSIX standard). We can further increase integrity by using checksums to reject corrupt files.
See the schedule for the slides and further information.
TPM Is No Silver Bullet: Pitfalls in Embedded Device Security – David Gstir, sigma star gmbh
The Trusted Platform Module (TPM) is standard in the PC landscape, with operating systems such as Windows requiring them to boot. Traditionally though, TPM’s haven’t been common in the embedded space – but this seems to be changing. The problem though, is that embedded devices and PCs are very different and have very different threat models. This talk provides an excellent introduction into the use of TPMs for embedded systems as well as possible alternatives.
See the schedule for the slides and further information.
Why Won’t My CPU Sleep? Debugging CPUIdle Mysteries on ARM SoCs – Dhruva Gole & Kevin Hilman
On any embedded system the desire to better optimise efficiency is almost always a goal, be it for thermal or power reasons. One of the fundamental ways to do this is to ensure that when the Linux scheduler has no tasks to run the system is put into the most appropriate idle state. These idle states come in several forms, from the generic scheduler idle task all the way advanced CPU specific sleep states which can power down parts of the CPU itself to various levels each with differing latency penalties for wake up.
The talk details the journey in properly configuring a new TI AM62 series SOC and the debugging techniques used to trace down unexpected wake up events. The talk delves into the details of how the kernel handles timer events and wake up sources. Details on how to ensure that a timer device is used so that there is a method available to wake the CPU from the powered down state when needed. Details of how the kernel interacts with the Trusted Firmware(TF-A) is explained along with debugging tips.
See the schedule for the slides and further information.
Hotplug of Non-discoverable Hardware: Status and Future Directions – Luca Ceresoli, Bootlin
This was about hot-pluggable devices in Linux and how we can achieve this in the correct way for both the
devicetree and drivers themselves. This is an issue we have seen before with customers’ hardware and have had
to work around in a custom way. We have also seen it proposed on the kernel mailing list to allow support for
some laptop docks which use i2c; another good real world example of hotpluggable non-discoverable hardware.
Currently drafts have been sent for the initial framework of allowing hotpluggable DRM devices, for example detachable DSI displays, and there is a lot more work to go. When a display detaches there needs to be graceful shutdown of any clocks and any errors cleared. Similarly when a new one attaches, the resolution, clocks and timing might all be different and need changing for any DRM devices before and after the connector.
This is not an easy problem to solve, and something not all subsystems can support. The media subsystem, cameras for example, does not allow a device in a media pipeline to be removed, or the pipeline modified after creation. This was discussed during the questions at end of the talk and is something I hope is explored further. This as a whole might be a lot of effort but we believe it will be worth it in the end, and allow for supporting this type of hardware with fewer kernel hacks and workarounds.
See the schedule for the slides and further information.
Easily Generating Debian-Based Embedded Systems – Clara Kowalsky & Felix Mößbauer, Siemens AG
Probably most people who use yocto to build images wil probably have heard of kas and kas-container by now. Kas was developed by Siemens originally to simplify the whole process of doing yocto builds by using yaml configuration files. Isar, named after the river flowing through Munich, and standing for Integration System for Automated Root filesysem generation, is equivalent to Yocto in that it is a build system that uses recipes, layers, config
files and bitbake.
Clara Kowalsky explained that kas + Isar is an attempt to get the best of both worlds between the customisability and flexibility of yocto and the convenience and solidity of the Debian distribution with its binary feeds. She showed how you can customise an image for Raspberry Pi via a menu based system to select machine, packages etc., and then went on to show how you would go about creating a custom image in a more yocto-like way, but still using the debian binary feed and also creating your own debianised packages.
Felix Mößbauer then went on to talk about the overall Isar ecosystem and future plans. He explained that although yocto and Isar are similar, they are not the same thing, but close enough so that they can contribute Isar features to yocto and vice versa. During the questions he explained that although machine definitions in Isar are slightly different from yocto, the main task in supporting a new board would be converting the BSP layer, but that if this was based on u-boot it actually would not be too hard.
See the schedule for the slides and further information.