That’s how Black Mirror Bandersnatch works on iOS and why it doesn’t work on Apple TV

The networks are revolutionized with the new Black Mirror movie/episode. Following in the wake of previous experiences such as an episode of The Cat in Boots on Netflix itself or the adaptation of Minecraft’s Story Mode. The new episode inspired by the 80s and the world of video games has a peculiarity that has brought to mind many of us those wonderful books of «Choose Your Own Adventure». Books that weren’t read linearly because they forced you to make decisions that had to do with the book’s own history. If you go north go to page 62, if you prefer to talk to the elf go to page 51. This is how these books were structured and in this new installment of Black Mirror we have an interesting evolution for the online audiovisual format.

However, the launch has not been uncontroversial. We’re not going to go into the one who rates the content of the movie itself as bad or good. But technically we have found the fact that two devices of the most used to watch Netflix today (especially the first one) do not support this feature and will not be able to enjoy the interactivity of the episode: Google Chromecast and the Apple TV.

-Netflix reported that technical impossibility prevented the interactive Black Mirror Bandersnatch experience on Google Chromecast devices and Apple TV-

To understand this technical difficulty we will explain what the system used by Netflix consists of and then we will see why both devices do not have support for this experience.

The fountain, seamless branching

The technique Netflix uses for this interactive movie is not new. To understand it we have to go back to 1997, the year the world met the Digital Versatile Disc or DVD. The beginning of the digital revolution in the entertainment industry. The DVD, among its functions, had a rather interesting one: seamless branching or «fluid ramification». This system basically allows you to create playlists of different chapters with the aim of not repeating the same movie several times and being able to create several montages of it just by including new pieces or scenes. And the seamless indicates that the transition is clean and the viewer will not notice that there has been a jump between chapters of your DVD. They were also known as angles, and were very useful for displaying content in different languages.

Let’s look at a concrete example to better understand it: Star Wars. When we put a film of the saga, the first thing we see is always the beginning with the story in our language (if we put it in Spanish). But if we put the film in English you will see the letters in English. Is the film recorded twice? No. The film is a composition of chapters structured in playlists. Thus, when we play the audio in Spanish it takes a track that has the beginning crawler in that language and when it ends, it goes on to play the chapter immediately after those letters that is already common to the rest of languages. The cut is frame-to-frame and instant jump. This way we don’t have to put the whole movie if we want multiple versions. Only chapters.

-Seamless branching was used, for example, for movies that had the extended director version and the normal version on the same disc. There was not the film twice, but from the chapters and an orderly list of them, one or the other version could be seen and, most importantly, in a fluid reproduction of the

This technique can also be used to jump between chapters smoothly and instantly to create interactive experiences. A format that technically already had the LaserDisc digital discs that used classics like the one from 1983 called Dragon’s Lair. A real interactive cartoon film, where we had to make the right decision or movement at the right time. Directed by Master Don Bluth, by the way, create from «Fievel and the New World» (An American Tail), «In Search of the Enchanted Valley» (The Land Before Time) or Anastasia.

Well, this same technique that Blu-ray also supports, is the one used by Netflix. The film is divided into chapters, in a rather complex organization chart (you can see the complete diagram in Reddit, but watch with the spoilers, clicking here), and when we have to make a decision it is always at the end of a chapter. When a few seconds remain we are asked for a decision and shown a scene selection for a few seconds (the decision).

If we don’t choose, the system will choose one for us, but if we’ve chosen one when that chapter has just played, it dynamically hooks the next one in seamless branching, so we feel that the movie hasn’t stopped and that it’s all part of a same flow. Definitely quite ingenious and with an interesting result. However, for now decisions are always binary: either one thing or another.

The technical impediment

The Netflix team faced a pretty serious problem when it wanted to implement this feature on iOS a few months ago when the episode of The Cat in Boots was released that allowed it to be tested: the official Apple API for playing video (AVFoundation) does not support seamless branching. Without going into much technical detail, what Apple does is download the entire playlist to the device and ask for the pieces that make up it. Streaming, which uses the m3u8 format, is a succession of chunks or chunks that make up the playback of the content. Some chunks that have nothing to do with chapters. That’s another metadata that goes aside to classify, if the content supports it.

But as we say, Apple’s API lowers the list before playback and goes to order each piece continuously. We may think that we watch a single video, but in reality a movie can have thousands of pieces in a row one after the other. The problem begins when we need that reproduction to be dynamic. Allow us to choose one or the other chapter and move to that point in the list without stopping in playback. And Apple developers know that well: when a feature isn’t officially supported, it’s a pain to get what you want (if you make it).

In the case of Netflix, it has not made public how they have finally achieved it on iOS, but they have had to trick the player to get this functionality. As Jordanna Kwok, development engineer at Netflix, mentions, getting smooth playback on iOS has been nothing trivial. My theory, but it’s just an assumption, is that they have superimposed different players and activated one or the other depending on the option.

When a video finishes playing, it skips a system event that reports it and allows us to do something when this happens. I would have done it with a player already loaded with the two possible options that the viewer can choose, hidden so that they are not seen. A programmer can know the exact position of the content playing, so when there was a minute or two left would create two new playback views, with the content preloaded and with enough buffer to give it PLAY and make it look fluid. When it’s time for the decision, I’ll show the buttons on the player. If the viewer taps on one (or randomly chooses) when the end event arrived, I hide the player that just died and show the new one giving PLAY automatically. And maybe they had to put in a gap for a second or so. That is: I throw the playback hides a second before the one that is seen and then I remove and put views.

Yes, it’s a bit, but Netflix itself has rated the implementation on iOS as non-trivial as we’ve already seen. So I reckon something like that, adjusted to the millimeter with a lot of trial and error, is what they’ll have done in the end. This on iOS is operational, but not on Apple TV. On Apple TV the player is more limited and does not allow that game because it is not a view that we can control, but a process to which we launch a list and nothing else. It is thus made to integrate with the TVMLKit library, made in javascript. That way, a javascript code can launch a player.

The Netflix hybrid player

Is it technically impossible to make this interactive feature work on tvOS? Yes and no. If Netflix had created a native part of playback over UIKIt, we could have that control over the player and do the iOS trick. But in the tests I have done, however, the necessary fluidity is not achieved. There is always a pause to fill the buffer with the flower rotating. The same trick that works on iOS on Apple TV there’s no way it works.

This assumes that functionality would really be possible, but we wouldn’t have the smooth experience. Being Netflix I’m sure they’ve reached a point similar to mine and have seen that there’s no way to get it right. However, that doesn’t mean that Apple will be interested in including this feature in the future. Not just for your own streaming service, but to improve experiences on your systems and make life easier for developers. I’m sure that among the new features of iOS 13 there will be the possibility of this seamless branching natively, both on iOS and tvOS.

The case of Google Chromecast is even more complex and difficult to solve. Functionally, when a Chromecast supports a service, what the player does is send the particular playlist of the service to the registered device. Basically, there is no connection between our mobile and Chromecast, except sending the request.

When it is through a registered service, it is said that service (such as Netflix) sends the information directly to the Chromecast and the Chromecast will play to the end, with the only option of minimal playback control (volume, pause and little else). Therefore, there we also can’t play with interactive playlists and, much less, with custom buttons that allow for specific functionality. It’s a clean player.

From my point of view and analyzing Google Cast technology, it would only be possible if Google pulls out a new device that would allow us to choose the decision from the mobile at a given moment and change seamlessly between different playlists. But loading the buffer of a playlist and being ready to play is something that is not trivial and on such a device it would be complex to do so without the viewer noticing a cut.

The key: fluency

As we’ve already mentioned, Chromecast and Apple TV may do this functionality, but what’s not possible (yet) is fluid: the viewer will see that there’s a continuity and not a pause. If it doesn’t meet that premise, Netflix will obviously rule it out.

I hope that it has become clear and similarly, we hope that there will be more experiences of this kind in the future and that Apple will put the batteries (I am sure that with the media level that this episode has taken, it will) and for iOS 13 offers us this functionality in a way na Tiva. It would allow very interesting experiences to many levels of new content.