Overnight, Mark Zuckerberg revealed Meta’s latest progress with Virtual Reality headsets. The company has a wall of headset prototypes, many of which offer significant advances over the options available today, the challenge however is to integrate all of those benefits into a single, elegant, lightweight and affordable headset for the Metaverse.
We know Zuck is in love with VR, so much so that he changed the name of his company and is investing billions to make the industry mainstream. While Oculus Rift enabled many enthusiasts to understand the immersion possible with VR, it had plenty of downsides. To begin, the visual quality was fairly average and the tethered connection restricted movement.
The Oculus Quest provided the freedom to explore and hand-tracking, but still lacked the visual fidelity necessary to really trick your brain into thinking what you are experiencing is real.
To live up to Zuckerberg’s vision of the metaverse that he shared last Fall, experiences need to be almost indistinguishable from reality.
Adding to the issue of low resolution (relatively to the real world and what’s offered by laptops, TVs and phones etc, the lenses also distort what you see cannot be used for extended periods of time.
To get transform the experience of VR, Meta says they need to build a completely new type of VR display system — a lightweight display that is so advanced it can deliver what
our eyes need to function naturally so they perceive we are looking at the real world in VR.
This is ultimately known as the “visual Turing Test” and passing it is considered the holy grail of display research.
To pass the visual Turing test, the Display Systems Research (DSR) team at Reality Labs Research (RL Research) is building a new tech stack that includes:
- “Varifocal” technology that ensures the focus is correct and enables clear and comfortable vision within arm’s length for extended periods of time
- Distortion correction to help address optical aberrations, like warping and colour fringes, introduced by viewing optics
- Resolution that approaches or exceeds 20/20 or 6/6 human vision
- High dynamic range (HDR) technology that expands the range of colour, brightness, and contrast in VR
DSR experienced its first major breakthrough with varifocal technology in 2017 with a research prototype called Half Dome Zero. They used this prototype to run a first-of-its-kind user study, which validated that varifocal would be mission-critical to delivering more visual comfort in future VR.
Here we are 5 years on from that and we still don’t have consumer VR headsets that offer it, but Zuck was fast to show off prototypes that addressed this, and the other challenges above.
Over the last 7 years, the Display Systems Research team at Reality Labs has built over two dozen fully functional AR and VR research headsets. Some of the highlights include Butterscotch, Starburst and Holocake 2.
This prototype was created to focus on image quality, something Zuck calls retinal resolution. While he doesn’t provide a specific pixel count, he does suggest it would need to be around 60 pixels per degree. If we consider that the Oculus Quest offers 97 degrees horizontally and offers 1832×1920 resolution (per eye), that means we’re currently getting just 18.8 pixels per degree.
This means the butterscotch prototype would need to more than triple the current Oculus Question2 resolution, all while maintaining a similar form factor, battery life and thermal envelope.
Another key attribute of how we perceive what’s real and what’s not is lighting. The key metric to achieving better HDR is nits, or how much light an object emits. Research has shown the ideal number for peak brightness is 10,000 nits, and the TV industry has made progress in introducing HDR displays that come close to this.
In VR displays like those found in the Oculus Quest 2, you’ll find the Nits count at just 100, requiring a 100x improvement. As insane as that sounds, they actually did it. The Starburst prototype can reach 20,000 nits, being one of the brightest HDR displays ever built.
Holocake 2 is the thinnest and lightest VR headset they have ever made and looks much more like a commercial product. The electronics have clearly been miniaturised to the point where you may look at this and question if it’s actually a working prototype.. it is, but it is tethered.
Holocake 2 is a fully functional, PC-tethered headset capable of running any existing PC VR title.
To achieve the ultra-compact form factor, the Holocake 2 team needed to significantly shrink the size of the optics while making the most efficient use of space.
The solution was two fold: first, use polarization-based optical folding (or pancake optics) to reduce the space between the display panel and the lens; secondly, reduce the thickness of the lens itself by replacing a conventional curved lens with a thin, flat holographic lens.
The creation of the holographic lens was a novel approach to reducing form factor that represented a notable step forward for VR display systems. This is our first attempt at a fully functional headset that leverages holographic optics, and we believe that further miniaturization of the headset is possible.
These individual prototypes are one thing, but almost useless on their own, only when combined like Captain Planet, can the power be truly realised.
Mirror Lake is a concept design with a ski goggle-like form factor that will integrate nearly all of the advanced visual technologies DSR has been incubating over the past seven years, including varifocal and eye-tracking, into a compact, lightweight, power-efficient form factor. It shows what a complete, next-gen display system could look like.
Meta sees Mirror Lake as one of several potential pathways to that could help them pass the visual Turing test.
It’s really hard to estimate from this, just how far Meta is from the whole grail VR headset that lights the world on fire. Over the next decade, we can certainly expect many more prototypes, but regarding devices, you can actually buy, expect Quest 3,4,5,6 to keep the money rolling in to fund all of this.
Technology cost declines and scale, along with battery breakthroughs will also be required to provide this utopian VR experience we will apparently want to live, eat and breathe once it arrives. Iterating the technology is one thing, having it be affordable is quite another, so a realistic timeframe I’d estimate would be around 2030, around 7.5 years from now.