Earlier this year one of our customers presented us with an unexpected problem: they weren’t able to stream video from multiple USB UVC web cameras – instead they got a “No space left on device” error. In our quest to provide a solution we learnt a little about USB and so we thought we’d share our new found knowledge here. For this blog post we’ve reproduced the problem with a Raspberry Pi and a couple of USB web cameras which we found on Amazon.
To reproduce the problem, we’ll use GStreamer to stream MJPEG from one of our cameras at its lowest resolution:
$ gst-launch-1.0 -e v4l2src device=/dev/video0 ! "image/jpeg, width=320, height=240, framerate=25/1" ! fakesink &
$ Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClockNo problems there, whilst that’s still running let’s now start another GStreamer pipeline for the other camera, at the same resolution:
$ Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[ 1180.939251] dwc2 3f980000.usb: dwc2_do_reserve: Insufficient periodic bandwidth for periodic transfer
[ 1180.943308] dwc2 3f980000.usb: DWC OTG HCD URB Enqueue failed adding QTD. Error status -28
[ 1180.951746] uvcvideo: Failed to submit URB 0 (-28).
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed to allocate required memory.
Additional debug info:
../sys/v4l2/gstv4l2src.c(662): gst_v4l2src_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Buffer pool activation failed
Execution ended after 0:00:00.057912916
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...And it goes bang. The precise error you’ll see in userspace depends on the software you use (it’s not a GStreamer issue). The source of the error originates from the USB host controller driver (usually in response to a VIDIOC_STREAMON ioctl) and is usually in the form of an ENOSPC / -28 / “No space left on device” error. In the case of an XHCI host controller you may see a “Not enough bandwidth for new device state” error and for EHCI an “iso sched full” error.
To get a better understanding of the problem we can turn on trace debug of the UVC driver module, we can do this without rebuilding as follows:
$ echo 0xffff > /sys/module/uvcvideo/parameters/traceThis command will result in lots more debug in dmesg. This time when we run our second GStreamer pipeline we see the following relevant debug:
$ dmesg
[ 700.797547] uvcvideo: Trying format 0x47504a4d (MJPG): 320x240.
[ 700.797568] uvcvideo: Using default frame interval 40000.0 us (25.0 fps).
[ 700.811099] uvcvideo: Trying format 0x47504a4d (MJPG): 320x240.
[ 700.811134] uvcvideo: Using default frame interval 40000.0 us (25.0 fps).
[ 700.823784] uvcvideo: Setting frame interval to 1/25 (400000).
[ 700.836394] uvcvideo: Control 0x00980927 not found.
[ 700.837570] uvcvideo: Control 0x00980927 not found.
[ 700.841902] uvcvideo: Device requested 3060 B/frame bandwidth.
[ 700.841919] uvcvideo: Selecting alternate setting 11 (3060 B/frame bandwidth).
[ 700.844448] uvcvideo: Allocated 5 URB buffers of 32x3060 bytes each.
[ 700.846293] dwc2 3f980000.usb: DWC OTG HCD URB Enqueue failed adding QTD. Error status -28
[ 700.854749] uvcvideo: Failed to submit URB 0 (-28).
[ 700.861789] uvcvideo: USB isochronous frame lost (-63). The reason the kernel reports failure is because it thinks there isn’t enough bandwidth on the USB bus to support both of these camera streams at once. It has good reason to take this viewpoint though in practice there is enough bandwidth for both. So what’s gone wrong?
Let’s start with figuring out how much bandwidth is available, let’s assume USB2.0 Hi-Speed – even though its maximum bandwidth is 480 Mbps, only 196 Mbps of this is available for isochronous transfers per device. This is the type of transfer that’s typically used by devices that stream time-sensitive information as the USB protocol provides a guaranteed allocation of bandwidth with bounded latency. The USB standard imposes that Hi-Speed isochronous transfers take no more than 80% of the available bandwidth – thus leaving 384 Mbps for our cameras to share. (The purpose of this is to save bandwidth for other types of transfers on the bus such as control and bulk transfers – however as the limit is enforced in software you can override it, e.g. for the standard EHCI driver you can use the uframe_periodic_max sysfs attribute – though you’ll be in violation of the USB standard and who knows if the underlying hardware will have issues with this.)
If we take a look at our trace output above, we can see that each camera requires ‘3060 B/frame’ of bandwidth. The USB protocol divides time on the bus into fixed-length periods called frames – for USB2.0 each micro-frame is 125us long. Therefore 3060 bytes per frame is 195 Mbps (3060 x 1000000/125 x 8). So if one camera requires 195Mbps and the maximum available for all devices is 384 Mbps then clearly there won’t be any room for a second device.
The problem is that our stream is an MJPEG stream with resolution 320×240 at a framerate of 25 fps – even without the lossy compression that JPEGs bring, at 3 bytes per pixel our stream should require no more than 320x240x3x25 or 5760000 bytes per second or 46 Mbps. This is much less than the 196 Mbps requested and so with this resolution/framerate we could expect up to 8 streams at once. We need to understand why the device is requesting this much bandwidth.
When the kernel attempts to stream video from a UVC device, the kernel will select a USB interface descriptor from the device that represents the required video characteristics such as frame-rate, frame-size, compression, etc (you can view these via lsusb). The kernel then tells the device of its selection and reads it back – it’s at this point where we obtain a value from the device known as ‘dwMaxPayloadTransferSize’ – this represents the required bandwidth as calculated by the device. It seems that this calculation may not always be correct, for example if we attempt to stream video at every available frame-rate we can see that for this device we always get the same bandwidth value of 3060 bytes per frame (pretty much the maximum allowed).
$ dmesg | egrep "Trying|bandwidth|fps"
[ 3685.308943] uvcvideo: Trying format 0x47504a4d (MJPG): 320x240.
[ 3685.308969] uvcvideo: Using default frame interval 40000.0 us (25.0 fps)..
[ 3685.352224] uvcvideo: Device requested 3060 B/frame bandwidth.
[ 3685.352239] uvcvideo: Selecting alternate setting 11 (3060 B/frame bandwidth).
[ 3722.800562] uvcvideo: Trying format 0x47504a4d (MJPG): 800x600.
[ 3722.806611] uvcvideo: Using default frame interval 40000.0 us (25.0 fps).
[ 3722.862864] uvcvideo: Device requested 3060 B/frame bandwidth.
[ 3722.868823] uvcvideo: Selecting alternate setting 11 (3060 B/frame bandwidth).
[ 3733.936648] uvcvideo: Trying format 0x47504a4d (MJPG): 1280x720.
[ 3733.942785] uvcvideo: Using default frame interval 40000.0 us (25.0 fps).
[ 3734.002870] uvcvideo: Device requested 3060 B/frame bandwidth.
[ 3734.008831] uvcvideo: Selecting alternate setting 11 (3060 B/frame bandwidth).
[ 3743.881198] uvcvideo: Trying format 0x47504a4d (MJPG): 1600x900.
[ 3743.887335] uvcvideo: Using default frame interval 66666.6 us (15.0 fps).
[ 3743.952295] uvcvideo: Device requested 3060 B/frame bandwidth.
[ 3743.958258] uvcvideo: Selecting alternate setting 11 (3060 B/frame bandwidth).
[ 3750.525393] uvcvideo: Trying format 0x47504a4d (MJPG): 1600x1200.
[ 3750.531619] uvcvideo: Using default frame interval 100000.0 us (10.0 fps).
[ 3750.601195] uvcvideo: Device requested 3060 B/frame bandwidth.
[ 3750.607161] uvcvideo: Selecting alternate setting 11 (3060 B/frame bandwidth).
[ 3760.016892] uvcvideo: Trying format 0x47504a4d (MJPG): 640x480.
[ 3760.022945] uvcvideo: Using default frame interval 40000.0 us (25.0 fps).
[ 3760.077727] uvcvideo: Device requested 3060 B/frame bandwidth.
[ 3760.083679] uvcvideo: Selecting alternate setting 11 (3060 B/frame bandwidth).USB isochronous devices typically present multiple ‘alternate’ interface configurations – effectively allowing you to select between a fixed selection of streaming bandwidths that the device will adhere to. Thus if the kernel has determined that 3060 bytes per frame are needed, then it’ll select an alternate interface that has at least this much bandwidth.
So the root cause of this issue is that some UVC devices indicate they require more bandwidth than they need. In our case every possible configuration results in the same bandwidth requirement, in our customers case the calculated bandwidth changed but was often much higher than expected.
However its not uncommon for devices to be a little broken and so the Linux kernel provides some mitigation (‘quirks’) from these issues within the UVC video module. One quirk is UVC_QUIRK_FIX_BANDWIDTH – this makes the assumption that the device calculated bandwidth is wrong and instead the kernel calculates its own value based off the frame size, frame rate and bits per pixel. Unfortunately this quirk is increasingly unhelpful as this only applies to uncompressed video formats – these days most cameras stream compressed video formats.
It’s not practicable for the kernel to estimate the bandwidth requirements of compressed video formats – there are many formats available and for each one the kernel would need to understand and have the required information for all the factors that effect the compressed video size. Therefore we can’t expect the kernel to solve this problem for us – instead any solution we create will likely require knowledge from the user.
We’ve learnt that if we are confident about the actual bandwidth requirements of our device then we can ignore the device calculated value and select a different alternate configuration that uses less bandwidth. Therefore the simple and pragmatic solution we delivered to our customer allows the user, via a sysfs control, to set a cap on the bandwidth that any single UVC isochronous device can use. This works well for our customer as they use identical cameras, have complete control over how the cameras are used and can perform adequate testing to ensure that performance is met.
By default the bandwidth cap is set to unlimited (-1). It can be changed on the kernel command line by adding:
uvcvideo.bandwidth_cap=Xor via sysfs, e.g:
$ echo X > /sys/module/uvcvideo/parameters/bandwidth_capWhere X is the bytes per frame, e.g. 2400 is 153.6 Mbps ((2400*8000*8)/1000000). Of course, this is the maximum bandwidth that you anticipate that the device will use – the kernel will then take this value and compare it with the alternate descriptors in the device to select a configuration that can deliver at least this amount of bandwidth. You can see the available alternate descriptors by looking at the output from lsusb.
As this is a very common problem that users face we felt it would be helpful to upstream this such that future releases of the kernel would have this capability. However, our solution requires that end-users can determine a suitable value for bandwidth which may not be trivial to get right. As a result our solution is considered a hack and so it hasn’t gathered any traction on the mailing lists. You can find our patch here.