Keeping Track of Time with Systemd

Nearly all smart devices obtain and keep track of time – it’s something that ‘just works’ and something which we often take for granted. Yet under-the-hood there is a surprising amount of complexity – software needs to obtain an accurate external source of time (e.g. NTP), it needs to handle drift and gradual synchronisation between a Real Time Clock (RTC), the system time and external sources of time. It needs to keep track of every-changing time zones, day light savings and leap-seconds. Oh and let’s not forget issues such as the Y2K bug and friends (and the upcoming 2038 issues). Whilst we rarely need to understand the detail of how this works – it is often helpful to understand the software components involved and what they do in particular scenarios. In this post we’re going take a look at how a typical embedded Linux system manages time.

Let’s start with the system clock – Linux systems provide the ‘date‘ utility (and Systemd provides a ‘timedatectl‘ utility) which allows you to read and write the current ‘wall-clock’ time, this is Linux’s view of the current system’s date and time. Most processors have some form of counter that increments at a known rate which the kernel uses to keep the system clock up to date.

Of course the problem with the system clock and underlying hardware counters is that they reset and stop counting when the device is powered off. This means that when the device is powered on, the current date/time is unknown – often reverting back to the year 2000 or 1970.

This is where the Real Time Clock (RTC) comes to play – this is the hardware component (normally powered by a coin-cell battery) that continues to count time, even when the CPU is unpowered. The utility ‘hwclock‘ is often available that allows you to read and write the RTC time. It’s recommended to store the time in the RTC as UTC – this avoids issues relating to time zone changes and daylight saving switches.

$ timedatectl
   Local time: Mon 2021-11-22 15:24:43 UTC
   Universal time: Mon 2021-11-22 15:24:43 UTC
   RTC time: Mon 2021-11-22 15:24:43
   Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
   NTP service: active
   RTC in local TZ: no
$ date
Mon Nov 22 15:24:44 UTC 2021 
$ hwclock
Mon Nov 22 15:28:00 2021  0.000000 seconds

Upon start up, the kernel (through it’s rtc_hctosys initcall) will read the hardware clock and set the system time from it. You’ll usually see something like this in dmesg:

rtc-em3027 2-0012: setting system clock to 2021-11-22T15:23:00 UTC (1637594580)

However, what happens if the time in the RTC is wrong? Perhaps the battery is dead? Well the kernel doesn’t care. This is where the systemd timesyncd service comes to the rescue. When this daemon starts it attempts to determine if the current system time (most likely obtained via the RTC) is valid. Without knowing the real world time, it does this by comparing against a couple of known timestamps. The first timestamp is the known date/time that systemd was compiled (as per a SOURCE_DATE_EPOCH environment variable) – of course, the current time can’t possibly be before the time the running code was compiled. The second timestamp is the modification date of the /var/lib/systemd/timesync/clock file – this file is touched whenever systemd gets a time update from an external time server. Thus given that time only moves forward then a valid time will be later than this timestamp. Therefore upon start if timesyncd detects that time has gone backwards then it will set the system time to a valid timestamp – you’ll see a message in the systemd journal when this happens:

Feb 02 15:29:49 affinity systemd-timesyncd[297]: System clock time unset or jumped backwards, restoring from recorded timestamp: Mon 2021-11-22 11:15:48 UTC

Though, it’s questionable how useful this behaviour is. For example if the RTC has a dead battery, would you rather revert back to 1970 time – when its obvious the date is wrong. Or revert back to some recent date that may seem to be valid? It very much depends on the use case.

Now unless your device is an atomic clock, then the RTC time and system time will drift compared with each other and against the real world time. So it’s extremely beneficial to synchronise the device with an external time source. Once again systemd can do this for us with it’s timesyncd service. This service implements a simple SNTP client that allows it to obtain the time from NTP servers periodically. Once it obtains the time it updates the system clock using the clock_adjtime systemcall. This call allows for a gradual adjustment of the system clock for small differences in time.

It’s worth pointing out the systemd doesn’t ever write time to the RTC – this is taken care of by the kernel. The kernel will automatically write out the system time to the RTC approximately every 11 minutes – however this only occurs when the kernel configuration option CONFIG_RTC_SYSTOHC is set. It also requires that userspace has unset the STA_UNSYNC flag on its call to clock_adjtime – systemd will unset this flag when it has synchronised its time with an NTP server and only when the RTC uses UTC time instead of local time. Thus the view is that there is no value in regularly syncing the RTC clock from the system clock unless the system clock is accurate.

We were curious as to why the RTC is updated every 11 minutes so we put on our kernel archaeology hats and found that the kernel has been doing this right back since version 0.99.13k of the kernel (1993) – unfortunately there doesn’t appear to be an explanation.

So in summary – systemd and the kernel will usually ‘do the right thing’ with regards to time. Though it was interesting to note that systemd will never write to the RTC and the kernel only writes to it if it believes that the system clock has been updated by an external source (e.g. NTP). It was also interesting to note that systemd will attempt to provide the last known valid time on start up – this effectively prevents the system clock from every being set to the 1970’s – however this may be a surprise to some device makers and may not be what they want – especially if they have application code that looks for old dates as a way to prompt the user to enter the time. And finally, where a device makes use of a read-only filesystems, you may wish to consider the impact this has on the /var/lib/systemd/timesync/clock file.