Low-Latency Everywhere: How to Implement LL-HLS Across Platforms

With the publication of the iOS 14 family last September, Apple has officially released LL-HLS support across its ecosystem. The Apple device family, which spans iPhones, iPads and Apple TVs has a significant market share. With about 25% of mobile devices worldwide running iOS, and this number reaching even more than 54% in North America according to Statcounter, there is no way around the Apple ecosystem.

The question at this point is not “Should I implement LL-HLS to satisfy my low latency needs", but  “Can I leverage LL-HLS across other platforms as well”. Let’s dig into that!

How big is this Apple ecosystem really?

As mentioned, there are quite a lot of iOS devices around. With LL-HLS only being available on the iOS/iPadOS/tvOS 14 families, one could wonder when the critical mass of devices is large enough. In contrast to the Android family, Apple’s iOS family has a good reputation when it comes to updating the OS version on their devices. Devices often require updates, already pre-downloading the update and nagging to users who are trying to postpone. When we look at iOS 13 for example, about 92% of iOS devices sold over the last year were running iOS 13 at the end of June, which is less than one year after introduction. As a result, it is quite realistic that within a year, we are seeing similar numbers for iOS 14. In fact, according to MixPanel, you better have LL-HLS available today: more than 79% of all iPhones are already running iOS 14. Keeping in mind the release was only a few months ago, this is not bad at all.

Should we leverage LL-HLS on other devices?

To date, most people tend to leverage LL-DASH in order to reach the other 88% of the market. While Apple tends to discourage quite actively the use of MPEG-DASH on its own devices (and making it technically very difficult to do so), using LL-HLS is a must for most publishers. As a result, switching to an LL-DASH only solution is often not feasible. Compatibility between LL-HLS and LL-DASH can be achieved, as we showed during our webinar with Ateme and Akamai, but if possible, why not switch to a single solution entirely?

Thanks to LL-HLS’ backwards compatibility towards “legacy HLS” players, even players which haven’t been optimised for LL-HLS playback can play, albeit at higher latency, the same streams. This is very valuable as it allows you to reuse the same stream for both higher latency and low latency playback, regardless of the platform and availability of a low latency player. There will be no need to set up a separate stream to deliver to devices not compatible with LL-HLS.

This leaves us with the next question to answer, which is about the relevance of LL-HLS beyond the Apple ecosystem. For most of you, the list of relevant devices will be highly specific for your target market. We are seeing some big trends however. It makes sense to talk about the other part of the mobile market, with Android mobile devices having a huge footprint, even bigger than iOS’. There is also the smart TV market, and markets such as the desktop browser market (where Apple’s Safari only holds about 8%) can be quite relevant, and Apple’s Apple TV device for the big screen only covers about 8.7% of the connected TV devices market according to Conviva.

When digging into the consumption graphs, Conviva’s reports leave little to the imagination: when we are looking at the global viewing time of content across iPhones, iPads, Apple TV and the Safari browser, only about 12% of all content is being consumed on these devices, leaving about 88% of viewing time up for grabs on devices not supporting LL-HLS to date. We see three relevant families of devices here:

1. HTML5 powered devices, spanning browsers as well as smart TVs, set-top boxes and others,
2. devices running Android, Android TV and Fire OS, and
3. Roku’s devices (and smart TVs running the same system).

Bringing LL-HLS to the web and HTML5 powered connected devices

When we are talking about HTML5 based devices, we are actually talking on a very large range of devices. Based on the latest usage statistics, about 38% of all viewing takes place on devices which are running HTML5 powered apps (not including mobile platforms). This is a big number. The form factors of HTML5 based devices also differ significantly. Where of course you have your classical desktop and mobile web browser usage, often forgotten segments are the smart TV ecosystem with platforms such as Samsung’s Tizen, LG’s webOS, Panasonic, Hisense VIDAA U and many more and the gaming console ecosystem where both PlayStation and Xbox are HTML5 powered platforms. Other notable ways to get on the first screen are HTML5 based STBs, Chromecast.

With the HTML5 ecosystem, rich APIs become available. The most notable APIs are Media Source Extensions (MSE) and Encrypted Media Extensions (EME). These APIs allow you to respectively access the hardware decoder and the in platform DRM system. Two crucial components to develop video applications for premium content.

Video player developers make use of the HTML5 APIs in order to implement their own streaming protocol handling. THEOplayer for example was the first HTML5 based video player to provide HLS support. When Apple announced LL-HLS, we implemented LL-HLS in our Web SDK as well, bringing the power of LL-HLS to the HTML5 based ecosystem.

It doesn’t stop there though. Other player vendors are also working towards support for LL-HLS. While this is still a moving target, none of the open source players seem to have implemented LL-HLS to date. The hls.js team is working towards a first version (hls.js 1.0.0), but this is at time of writing incomplete and still in alpha-phase. If all goes well, we should see a release in 2021 of their first version. Shakaplayer missed their roadmap which had support planned for their Q3 release, but to date this seems to be running hopelessly late, with a release for both LL-DASH and LL-HLS support still being under development. For those players however, you can perfectly run LL-HLS streams at a higher latency until they add support thanks to LL-HLS’ backwards compatibility. 

All in all, the HTML5 ecosystem seems well equipped to support LL-HLS, and already does so today when using THEOplayer.

Tackling LL-HLS on Android and Fire TV

The Android device family is, similar to the HTML5 family, quite extensive and spans almost every form factor. It spans mobile devices running on Android, smart TVs running Android TV (Sony, Philips, …) and connected devices running the compatible FireOS (FireTV), or Android TV itself (various stick-like devices and STBs). Adding up all of the market share of these devices, you reach a solid 25%.

Often the reflex on the Android ecosystem is to use ExoPlayer. To date however, ExoPlayer didn’t complete support for LL-HLS. As ExoPlayer is used by almost every commercial video player vendor as well, most of them will need to wait until support is added there. In the meantime, you can use the feed at a higher latency, or use LL-DASH to get your low latency feed to Android.

At THEOplayer we maintain our own pipeline for Android and have no dependency on ExoPlayer to handle playback on the Android device family. Support for LL-HLS was added to THEOplayer early Q3, allowing you to run low latency streams with HLS (LL-DASH support was added already in 2019) across the ecosystem.

As a conclusion, seeing mass adoption of LL-HLS on Android will probably wait until the ExoPlayer team manages to merge in support (but it’s open source, so development at the “speed of open source” is to be expected). In the meantime, THEOplayer customers can already use LL-HLS on Android as full support is available there.

The pains of bringing LL-HLS to the Roku ecosystem

One thing to be said about the Roku ecosystem though, it’s big. Very big. Worldwide, about half of all connected TV devices are Roku devices. Keeping in mind they don’t play in some parts of the world, the market share is massive depending on the region. When comparing to others, the Roku ecosystem has the same size the Android ecosystem has (so if you’re not targeting it yet… why?).

So what about support for LL-HLS on this last remaining platform? The Roku ecosystem is a bit of an odd duck when compared to the others. There is no way to develop your own video processing pipeline on the device (apparently unless you buy a massive amount of devices, and even then it’s less than straightforward). As a result, you are forced to use the Roku native player.

To date, the Roku pipeline doesn’t support LL-HLS at low latency just yet. While no official public statements have been made on support, we can deduct a few things. 

• One of those is timing of releases. Roku usually publishes updates to their devices in April and September. 
• A second thing is the speed of updates spreading across the devices. As devices are automatically being updated, it is safe to say that when support is added, it will be available to your viewers closely after that (without any action on their side).
• As a third item, we can see Roku usually adopts new capabilities quite quickly (at least for a vendor which is in essence a hardware vendor). Initial support for LL-DASH came about one year after first vendors started providing support. I would take the bet Roku will come with LL-HLS support about one year after support for most other vendors is there.

This leaves us to the guessing game of when the support will actually be there. A reasonable guess would be to assume most vendors will announce LL-HLS support at NAB, which coincides with the April release of Roku. Personally, I think it would be early to hope for an April release of LL-HLS (but please surprise me!). More reasonable would be to assume it coincides with the IBC release in September, or (if we are unlucky) in the April 2022 release. While this might be bad news compared to the other platforms, you can still deliver the same LL-HLS feed to the platform, or you can look at one of the compatibility profiles for LL-HLS and LL-DASH to find a go-between.

Quick note on the (im)maturity of LL-HLS

So does this mean we can start using LL-HLS everywhere? Often questions rise such as: “What with DRM?” and “How about subtitles and alternative audio tracks?”. Well, these are good questions.

While using HLS everywhere is a decent option, LL-HLS is still relatively new and vendor support is not that broad yet. It is however to be expected that it is only a matter of time before you can really use LL-HLS at full feature everywhere. 

While to date DRM support is still a feature not supported by most packagers. When we however combine the capabilities of LL-HLS with the industry trend of CBCS being used as an encryption schema across DRM systems, it becomes interesting. Vendors are working fast towards support of DRM in combination with LL-HLS. While nothing much needs to change on the packager side, it is important to ensure license servers can deliver DRM licenses fast: when a player is playing in low latency mode, buffers are very small and every delay in obtaining a valid license can incur a stall. If you add up that all viewers will request a new license at the same time when keys rotate, it’s important to test and scale for these scenarios.

From the subtitle and alternative audio track side, there is some good news. Most vendors are already working towards support for this. The trickiest bit is likely getting your captions in the packager: speech-to-text systems can require some time to come up with a first draft, and ideally you need to clean this out still. This likely takes more time than the target latencies you can achieve with LL-HLS, which is in the lower single digit seconds ballpark. It is of course easier if you know what will be said up front or if you are broadcasting a pre-recorded piece of content.

Key takeaways

LL-HLS is showing a lot of potential to bring low latency streaming in a single format across all platforms. While the entire ecosystem is still moving and vendors are working towards support, there is a very high potential for LL-HLS to become a leading protocol bringing low latency in the 2-8s range to all platforms. If you already want to start working with LL-HLS today, just reach out to our team: we have LL-HLS support across HTML5, Android and iOS ecosystems today already.

Questions about our LL-HLS? Contact our THEO experts.