Juggling switch ports and setting them up with VLAN configuration can be a difficult and time-consuming task at times. Switches use a variety of interfaces such as SPI/I2C/MDIO that can be used to access registers on the switch and control the port in order to do things such as specifying which VLAN a physical port is on or setting different modes. This also means that standard tools such as IP etc can’t be used. Talking to registers in this way requires an intimate knowledge of both the hardware at hand, as well as the protocol that is being used, meaning more time is spent with datasheets. What if instead, you could interact with the ports of a switch as if they were their own device?
The distributed switch architecture (DSA) subsystem is a networking framework that, even when cascaded, provides exactly that – consistent management of switch ports, bridging, and even packet forwarding. Treating each port in the cascade of switches as a unique Linux network device (netdev), unmodified Linux tools such as ip link/addr can be used transparently on each port. This means that all of the regular network tooling that is familiar to most regular Linux users suddenly becomes able to be used in situations that previously demanded a specific setup and hardware knowledge. Created in 2008, the framework was originally designed to support the Marvell Ethernet switch family, it has since evolved to support a number of other vendors. Not much activity surrounding DSA occurred after its introduction to the kernel until around 2016 when the architecture began to receive more attention, and device tree bindings were added and refined as well as more vendor support.
Put simply, DSA is a subsystem for managing hardware switch chips, with a dedicated ‘CPU’ port passing Ethernet frames from/to the CPU. The CPU port here is the switch port that connects upstream to the Ethernet MAC on the processor. The defining characteristic of a DSA switch is that sending/receiving a packet is not done in the standard way. Rather, it is connected to a host Ethernet controller (also referred to as the ‘conduit’) through the CPU port, and it is through the driver of this that packets are transferred to and from the switch. Regular switch ports are known as ‘user’ interfaces, which, thanks to this subsystem, are also standard Linux network interfaces, and can be treated as such. DSA manages the relationship between the switch and the host Ethernet controller itself, ensuring that the Ethernet device acting as the host plays nicely with the additional functionality added by a DSA switch.
The architecture stipulates that one virtual network interface should be exposed for each front-facing switch port (swp0-3 on the software side in the above diagram), and no virtual network interface should be exposed for inward-facing ports (‘CPU’ ports, or DSA/cascade ports if used). This is made possible by the use of the DSA ‘tagging’ protocol, such that each Ethernet frame is tagged with a virtual switch, and these tags are multiplexed/demultiplexed by a hook in the RX and TX path of the host Ethernet controller. The subsystem also supports switches that are not able to insert or strip tags, but the set of features might be limited since traffic separation depends heavily on port-based VLAN IDs.
In this subsystem, tagging refers to adding a small piece of metadata added to the Ethernet frame by the switch driver before it’s handed up to the networking stack which encodes information about switch specific details. DSA handles these tags by stripping them before the packet reaches the normal networking stack, abstracting this information. On transmit, the reverse happens, where the kernel adds a tag, so that the switch knows which port to send the packet out on. All of this is abstracted from the networking stack, meaning it doesn’t have to handle or worry about this fine grained control.
The ‘distributed’ section of the acronym refers to the way the subsystem has been designed with the (entirely optional) ability to configure cascaded switches on top of each other using upstream and downstream Ethernet links between switches using ports referred to as ‘DSA’ ports. This is not shown in the above example. Switches set up in this way are treated as one big switch tree, and the topology is invisible to the user as it is controlled by the DSA framework.
From a software designer’s standpoint, you might be wondering whether some hardware should make use of the switchdev framework or a DSA framework. The answer to this question is not the same as a simple DSA vs. switchdev comparison, since they are not differing solutions to the same problem. DSA manages many features that are outside of a switchdev driver’s operation. Switchdev, on the other hand, is a framework that offloads these features to hardware, as opposed to the software approach taken by DSA, therefore it depends on the specific requirement of a network.
The primary use cases for this subsystem fall into two main categories. The first use is in embedded and IoT networking, since many system-on-chips for routers, gateways, and IoT devices integrate Ethernet switches. DSA allows Linux to control these switches seamlessly; exposing each port as a normal network interface. Other use cases can include home networking electronics, since routers, NAS devices and smart hubs often embed multiport Ethernet switches.
The DSA dataflow is much the same as a ‘normal’ Ethernet switch dataflow but with an added hook. This hook checks if DSA is in use, and if so, changes the device used by the network interface to the required receiver/sender, and then repeats the process. In the example of a reception event as shown below, the standard dataflow for reception of an Ethernet frame follows a similar path to that of a DSA-enabled switch frame but simply bypasses the device extraction that occurs in the DSA case. The processing that happens before and after the destination has been determined is identical.
Let’s take a look at some of the functions described above in more detail to provide an insight into how the tagging protocols are used in practice:
/* eth.c */
__be16 eth_type_trans(struct sk_buff *skb, struct net_device *dev)
{
/* ... */
if (unlikely(netdev_uses_dsa(dev)))
return htons(ETH_P_XDSA);
/* ... */
}
eth_type_trans() is a function called from within netif_receive_skb(), and is the Ethernet-specific transfer function. In an Ethernet data transfer, the kernel checks if the network device uses DSA (netdev_uses_dsa), and if so, proceeds to the tag receive/transmit function, where the tags are stripped/added respectively.
Here is a receiver example from the Qualcomm tag file:
/* tag_qca.c */
static struct sk_buff *qca_tag_rcv(struct sk_buff *skb, struct net_device *dev)
{
/* ... */
/* Get source port information */
port = FIELD_GET(QCA_HDR_RECV_SOURCE_PORT, hdr);
skb->dev = dsa_conduit_find_user(dev, 0, port);
if (!skb->dev)
return NULL;
/* Remove QCA tag and recalculate checksum */
skb_pull_rcsum(skb, QCA_HDR_LEN);
dsa_strip_etype_header(skb, QCA_HDR_LEN);
/* ... */
return skb;
}
Here, the device used by the network interfaces is updated to be the required destination, by traversing the DSA network to find the end user. Then, the destination – being treated as its own network device – will not trigger the DSA hook above, and the computation continues as if it were a ‘normal’ Ethernet frame, with the frame reaching the desired target.
Alternatives to DSA include the swconfig program (often used in OpenWrt), which allows the configuration of configurable Ethernet network switches. While both services function similarly, one of the key differences is that DSA has been accepted into the mainline kernel, while swconfig has not and will not be. There are slight conceptual details too, since DSA exposes every switch port as its own Linux network interface, which is then bridged together, to make use of the VLAN tagging. There are also other more ‘rough and ready’ solutions that are created by SoC vendors, but these are specific and more of a simple way to just get things working.
There is, of course, a rather steep initial learning curve when using DSA compared to ‘traditional’ configurations such as swconfig or UCI (United Configuration Interface). Critically though, DSA abstracts away a lot of the pain of setting up switch ports when fully utilized, since there is no reason for the user to interact with the host Ethernet controller. As described above, it also handles tagging/stripping silently without the need for a complicated user setup. Without this abstraction, users would need to provide much more configuration, and – notably – need to set up and interact with the host Ethernet controller. Configuration can therefore grow in complexity quite quickly, without cutting out this setup.
DSA makes controlling the data flow of Ethernet frames between switch ports – particularly cascaded switches – straightforward. If your network makes use of an independent Ethernet controller, DSA might be the switch architecture for you.