Hand-Tracking Technology to be used in controlling Future MR Headset User Interfaces is Closer to Market than you Think

On Wednesday May 11, Patently Apple posted a report titled “A new Google patent reveals upcoming AR Glasses that will work to sync device accessories to capture In-Air Gestures to control UI +. ” Our report noted that next-generation Mixed Reality headsets and AR smartglasses manufactured by Apple, Google, Facebook and many other gamers have a common feature of development related to hand tracking and interfaces. which can be used to sync in-air hand or finger gestures to control UIs, move virtual objects and more.

Currently, Patently Apple covers 59 Apple patents covering in-air actions. One of its most recent Apple patents was posted last month titled “A new Apple Patent that deeply describes the use of In-Air Gesturing to control a new class of Mixed Reality Headset Interface. ” Technology is closer to reality than most think.

Last week I discovered two Facebook patents showing some work done by their engineers, but most of all, I stumbled upon a developer video showing a Facebook SDK with hand tracking. The snippet below can help you imagine, beyond patent graphics, how far hand-tracking will go for future MR Headsets for controlling the next-gen user interface and beyond.

Notice in the lower right corner of the video you can see the developer moving his hands and in the larger display area you can see his hand-tracked virtual hands interacting with different parts of a potential Headset User Interface.

If you want to see more, check out Dilmer Valecillos ’full video here.

While Apple has never released video prototypes of what they’re working on, you can be sure that Apple’s engineers are more than willing to file patents and have working models to track hands on development as well. . It will be very interesting to see this happen in the future.

One of two new Facebook patents published by the U.S. Patent Office is titled “Hand Presence over Keyboard Inclusiveness” dated April 21, 2022. Here, Facebook engineers showcase their invention of hand-tracking specific for to use a virtual keyboard is unbelievable. accuracy and dexterity.

Facebook patent FIGS. 5A and 5B below illustrate examples of user #210 hands displayed above the translated virtual keyboard #420 in a virtual reality environment #500. In patent FIG. 5B the display says: “Hello, I’m typing in VR !.

(Click image to enlarge)

(Click image to enlarge)

See Facebook patent application 20220121343 for details.

In Facebook’s second patent application filed in Europe last week titled “Artificial Reality System with sliding menu” it included an AR Headset menu UI that can manipulate hand-selected movements on selection in menu items.

Facebook’s patent abstract states: “An artificial reality system is described as interpreting, presenting, and controlling user interface elements within an artificial reality environment, and performing actions in response to an or more detected user gestures.The artificial reality system may include a menu that can be activated and interacted with with one hand.In response to the detection of an activation gesture in a menu created with one hand, the artificial reality system can cause a menu to be rendered.A menu sliding gesture (e.g., horizontal motion) of the hand can be used to create a slidably engageable user interface ( UI) element to move along the horizontal dimension of the menu while the horizontal positioning of the UI menu is kept constant.motion) can cause the menu to be repositioned.communicate with the artificial cyst but in reality. “

On Facebook patent FIG. 2 below we see how a camera (#210) mounted on an HMD is used to track the hand movements of users used to sync the AR user interface to the internal display of the HMD; Fig. 6 is a flowchart illustrating the operations of an example method for setting up and interacting with a UI menu.

On Facebook’s patent FIGS. 7G and 7E below we see some examples of a user manipulating a menu by selecting “Settings” and/or a “Social” media button using user finger gestures captured by HMD’s camera system .

You can review European patent filing EP3980870 here for more details.

One of the key fights of the next generation of Head Mounted Devices is to provide users with a sophisticated menu to control content and games using eye and hand tracking. Apple, Facebook, Sony and others are deeply involved in the development of the systems and the Facebooks SDK video presented above really helped us see how cool it will be in the future. Will Facebook and others be able to beat Apple’s legendary UI development? Only time will tell.