After years of speculation about an Apple AR headset in the making, Apple unveiled Apple Vision Pro during a keynote presentation at the 2023 Worldwide Developers Conference in June. The vision? A revolutionary mixed reality headset capable of blending digital content with the physical world.
And whilst headsets are nothing new in the world of tech, the Vision Pro appears to promise something more. In fact, Apple doesn’t call its new device a headset at all, instead referring to it as a “spatial computer”.
But what does Apple’s much anticipated leap into augmented reality (AR) mean for the streaming industry?
The Apple AR headset runs on VisionOS, the first ever spatial operating system. Users operate the headset using natural controls such as voice commands, eye, and hand movements. The Vision Pro also features dual-driver audio pods with audio raytracing technology that accounts for the acoustic properties of the surrounding environment.
Ultimately these features allow users to feel like they’re physically present in whichever digital content environment they’re experiencing – whether that be a film or TV series, sporting event, live gaming experience, exercise class, or within a business setting.
Unlike VR headsets such as the MetaQuest2, which generate completely virtual worlds, AR applications and devices project images over a user’s physical surroundings. “With Vision Pro, you’re no longer limited by a display. Your surroundings become an infinite canvas”, explains Apple’s Chief Executive, Tim Cook.
With a 3D camera that takes spatial photos combined with conventional VR features, apps running on the device appear as a floating screen in front of the user. And considering that Apple has already compared its device to an all-in-one surround sound system, games console, and powerful laptop, the possibilities for the streaming industry are many.
New innovations in AR and VR are always met with excitement from the streaming world. These technologies are designed to heighten audio-visual experiences, after all. In September last year, Disney+ released its first AR-enabled short film. And AR-enabled live streaming platforms have been steadily emerging over the past few years. But Apple being Apple, the new Vision Pro has been met with another level of anticipation entirely.
So, what could the Vision Pro enable streaming platforms to achieve?
Streaming giants like Disney+ have already leapt at the opportunity to incorporate Apple’s AR headset into their product offerings. In fact, the CEO of Walt Disney Company, Bob Iger joined Apple’s Tim Cook as part of the launch of the new Vision Pro to demonstrate how the headset will be used to bring its movies, TV shows and even a virtual Walt Disney World to life “in ways that were previously impossible”.
“We believe that Apple Vision pro is a revolutionary platform that can make our vision a reality” comments Iger. It will help Disney to create “deeply personal experiences that bring our fans closer to the characters they love”, he continues.
In a recent report ahead of Apple’s launch, Bloomberg’s Mark Gurman speculated that “one selling point for the headset will be viewing sports in an immersive way”. Gurman furthered, “the company already offers games from Major League Soccer (MLS) and Major League Baseball (MLB) on Apple TV Plus, but it’s looking to make that a richer experience”.
These speculations were well founded. As well as securing deals with MLS and MLB, Apple already bought VR sport and concert events company NextVR back in 2020. And Apple TV Plus broadcasts of MLB games have been experimenting with new camera angles for some time now.
Both Disney’s Bob Iger and NBA Commissioner, Adam Silver, spoke to the Vision Pro’s potential impact on the sports viewing experience at the Sun Valley conference in Idaho this year. So, if industry optimism is anything to go by, Apple might soon make AR sports broadcasts a reality that sees widespread adoption.
VR has inspired mixed reactions from the gaming community in the past, what with its not insignificant financial barriers to entry and some users reportedly experiencing feelings of motion sickness when using VR. But despite the mixed reception from gamers, some are positive that Apple Vision Pro is the next big thing. So much so that one fan took it upon himself to create a prototype of the iconic RPG game, Skyrim, for the new headset.
Powered by Unity and using self-made 3D scans of a medieval Italian town, Frosali’s creation gives us a glimpse of what a VR version of the game could look like on Apple’s Vision Pro headset– seamlessly merging the creator’s living room with the Skyrim world. It’s early days but the potential for both single and multiplayer games, as well as video game live streaming is there for the taking.
Beyond the world of entertainment, augmented reality holds huge potential for enterprise organizations. Everything from remote video assistance to online training. In fact, Apple has documented a clear case for AR in the workplace, writing that “AR improves and even redefines how employees and customers complete tasks that are challenging or expensive -or even physically impossible - with greater efficiency and enhanced productivity”.
The Apple AR headset will also enable FaceTime users to create hyper realistic avatars. Apple’s headset has been trained using a diverse group of thousands of individuals. Users can scan their faces using the headset which uses an advanced encoder-decoder neural network to create a digital persona to use during video calls.
From OTT entertainment to live streaming, the possibilities that AR affords the industry are great. But to capitalize on these possibilities streaming platforms need to be AR-ready.
Factors to consider include:
A study published by the Institute of Electrical and Electronics Engineers emphasizes the importance of optimized transmission performance in the presence of AR and VR. This is because AR and VR applications are “bandwidth hungry, time-sensitive, and exhibit abrupt variations in bitrate”. As a result, issues like limited bandwidth aggregation efficiency are even more critical.
Streaming platforms will need to integrate AR into their existing applications. Some organizations, like the NBA, are already striding ahead. The NBA has become one of the most technologically progressive sporting institutions in the world. After relaunching its global app to take advantage of new tech trends the NBA implemented a series of planned updates to its official application and streaming service. Now fans will be able to add a three-dimensional avatar of themselves into live games.
OTT streaming platforms will need to either repurpose existing video content for AR or create AR-specific content from scratch. As part of its “requirements for great AR”, Apple advises the use of advanced cameras, motion sensors, powerful processors with machine-learning, immersive displays, and AR supported software development toolkits.
A method that is reportedly being adopted on Apple TV sets already, with some claiming to have seen 180-degree camera rigs in operation. Something that would suggest a move towards filming content designed for use with the Vision Pro.
For platforms looking to mobilize AR, choosing the right streaming infrastructure will become even more critical. AR applications are resource intensive. So, any streaming platform using VR, AR, 360-degree, or volumetric display needs video streaming servers that can handle these data-intensive workloads.
Typically, this requires dedicated servers with powerful central processing unit (CPU) capabilities. And since AR applications are so computationally intensive, they may also require dedicated GPU servers (servers with one or more graphics processing units designed to handle complex mathematical and graphical computations).
Then there are also network requirements to consider. High bandwidth applications like these typically require a network throughput of at least 25 Megabits per Second (Mbps) and sub-50 millisecond latency.
Partnering with an infrastructure provider with the architecture and expertise to host AR applications is paramount. Often, shared resource environments like the hyperscale cloud are not best suited to these applications. That’s because server resources are split between virtual machines which can impair the end user experience. By comparison, dedicated streaming servers are single tenanted meaning all server resources are made available to a single client.
The excitement that continues to surround this product launch is undeniable. With streaming giants the likes of Disney+ already embracing the technology, it’s possible that we’ll start to see more AR content made available to the public.
At present, the biggest barrier remains cost. Priced at $3500, the Apple AR headset is still a way off becoming accessible to the general population. In fact, Netflix won’t be making a native version of its app for the Vision Pro at all. A decision that Bloomberg’s Gurman boils down to Netflix’s skepticism over the price point (and uptake as a result). Vision Pro apps are likely to be significantly more expensive than iOS counterparts. “[I] wouldn’t be surprised if $20 is the new $1 for most,” Gurman speculates.
Now Apple is planning to launch a more affordable version of the Vision Pro to come out in early 2025. But with a speculated price tag of between $1500 to $2500, the cost will still be prohibitive to the average consumer.
The Vision Pro is an exciting innovation, but it seems unlikely that it will catapult AR into our living rooms in the immediate future. What it might do, however, is catalyze the rate at which streaming services start planning for a future where AR makes it to the mainstream.
