MetaReality Labs is making big strides in avatar rendering with the latest advances in combining machine learning (ML) with sensor data from Quest VR headsets to show your entire body, including arms, legs, torso and head. The result is a very realistic and accurate representation of the poses and movements of a person wearing the Quest 2 headset.
This is quite surprising as only the position and orientation of the Quest VR headset and its two controllers were used to estimate leg speed and position. There is no tracking band fitted on the legs and no external cameras have been used for this remarkable system. meta research scientist Alexander Winkler Shared several videos on Twitter on arXiv along with a link to the scientific paper and a YouTube video with more details.
In a scientific sense, this is known as motion tracking from sparse sensors and ML is particularly adept at extracting meaningful information from very little, if there are substantial dependencies between the known and the unknown. Since we swing our arms for balance when walking and running, arm movement is a good indicator of what the legs are typically doing. Combined with head tilt and direction, the ML system can very accurately predict most human motions.
More traditional approaches to tracking limbs rely on additional hardware, such as reflective markers placed on your legs and torso that are recognized by external cameras, or at various locations on your feet to transmit position and motion data. Wearable wireless beacon bands.
While these methods work, they are usually sold as accessory items that cost more and are not well supported in most apps and games. If Meta Quest 2 can achieve such great results with a headset, developers will be more likely to build body tracking into their games and apps.
Despite these significant advances, Meta Reality Labs acknowledges that more work is needed. If you move fast enough, the ML model fails to recognize your currency correctly. It is difficult for a computer to predict abnormal stance and if the virtual environment has constraints that do not exist in reality, the movement will not match. However, the overall effect appears to be great, and being able to see the full body instead of the floating toros while chatting with friends in VR would be a nice upgrade.
Hopefully this technology will be ready for launch soon. With Meta’s Quest Pro headset expected to be announced in a matter of weeks, the timing seems perfect for a full-body avatar. Meta Quest Pro can track eye movements and facial expressions, giving your friends, family and coworkers a better sense of presence in the near future.