Image Credit: Will Greenwald
Though not new, the Apple Vision Pro is the pinnacle of eye and hand-tracking technology Apple has ever implemented. Every consumer headset display should take a page from Apple’s playbook regarding augmented and virtual reality user interface design.
Built for mixed reality, the new vision combines iOS, iPadOS, and macOS elements, and it powers the Vision Pro. The principal means of interaction with the system is via the tracking of eye and hand movements, which is accomplished using the numerous internal and external cameras in the headset. Though it’s compatible with various input devices, including gamepads, mouse, and keyboards, the Vision Pro is best used with the eyes and hands.
Focusing on anything in the interface will cause it to light up somewhat, whether it’s a menu item, an icon, or something else entirely. As with a mouse or touch screen, pressing what you want to click on is as simple as tapping.
Choosing each element with my eyes, then pinching to relocate it on the board.
Image credit:Will Greenwald
But interacting with a working computer requires more than just clicking a button. That is why it is possible to pinch something by joining your fingers and then moving your hand in a way that mimics physically moving the object you squeeze. Find an empty area on the page and scroll up and down by swiping and flicking your wrist. In visionOS, you can see a little bar at the bottom of nearly all windows; pinch and drag to reposition it. In the same way, you can resize the window by dragging its corner.
You can manage the Vision Pro with just two basic gestures—focusing your eyes to pick objects around you—and, if you get the hang of it, they can even be faster than a mouse or touch screen. Tap your fingers together if you want to make a motion that can’t be noticed when your hands are behind your head.
It’s reasonable to claim that this represents a long-overdue, revolutionary change in how mixed-reality interfaces are designed. You’ll need a separate controller to control most augmented reality and virtual reality goggles. By establishing consistent and ubiquitous eye and hand control, the Vision Pro eliminates the need for any such device.
On previous occasions, I have been astounded by the utilization of eye-tracking technology in headset displays. Meta Quest Pro does an excellent job incorporating it into many functions and programs, although it’s only sometimes used consistently. With the PlayStation VR2, eye-tracking is even less of a control scheme. It’s only compatible with games that actively seek to integrate it—using eye tracking for foveation is the primary use case for both. It improves performance by directing more resources towards drawing the object directly before you and less towards what’s in your peripheral vision.
Foveation is also used by the Vision Pro, which, according to my preliminary testing, is a fantastic performance feature. However, it has resulted in some peculiar focus issues with the screenshots I’ve been capturing for our article.
Their respective physical controllers nearly entirely control the headsets from Meta and Sony. They are indispensable for almost any task you can imagine. In the rare instances where you can use your gaze alone to make a selection, you still have to wait for a confirmation bar to fill up before you can be sure it’s the one you want to activate.
The Quest Pro and the Quest 3 (the latter of which lacks eye tracking) also offer hand-tracking capabilities. However, they could be more accurate. To sustain monitoring, you’ll need to become acclimated to rather particular motions and gestures and keep your hands in a sweet spot in front of the headset’s cameras. It’s an excellent add-on for those headphones but can’t replace the controllers.
In particular, VisionOS’s hand tracking is the most sophisticated and dependable I’ve encountered; it integrates and enhances both eye and hand tracking as core features of the OS. I mean it literally when I say you can tap your fingers together in practically any position. After you’ve set up the Vision Pro by scanning your hands front and back with the headset, it can follow your hands anywhere in a nearly 180-degree dome.
Even while I reclined and tapped my fingers on my lap, the Vision Pro picked up on the motions. Both tapping and dragging with my hands over and on each side of my head have translated. Even though it’s not as effective when your hands are hidden from the numerous cameras on the headset—for example, when they’re behind you or on a blanket—I’ve found that tapping or dragging works in far more relaxed and natural situations than I had anticipated. It’s a vast improvement over the clumsy and inaccurate hand tracking of the Meta Quest goggles.
Even when your hands are hidden, tapping may still occur. Although Apple has remained mum on the subject, I wouldn’t be surprised to see visionOS and watchOS upgrades in the future that allow the Apple Watch Series 9 and later models to connect with a visionOS device to enable the double-tap gesture only through motion detection. I could see such a connection functioning.
However, it’s just hypothetical.
No exaggeration when I say the Vision Pro’s user interface is the greatest I’ve ever seen; I’ve used every primary consumer VR headset and several AR devices. Similar to the dramatic improvement in usability with iOS, this will allow smartphones and tablets to appeal to a much broader audience than just techies and experts. The technology and components that make up the headset aren’t novel, but they’ve been fine-tuned to the point where nearly anyone could use it and enjoy it.
This one can be the Meta Quest 2. The apple vision pro may not be available yet. But the Meta Quest 2 is! Buy it here on Amazon!🚀 
As an affiliate I get a small fee from Amazon.
