Doing office work in VR isn’t easy. It isn’t just the lack of software plugins for VR and most productivity apps, since VR headsets can already transform into wearable monitors. The keyboard part is what gets really frustrating. Facebook’s working on a way to bypass physical keyboards altogether with hand tracking.
Facebook Reality Labs’ latest research project, being presented at the Symposium on User Interface and Software Technology, demonstrates hand-typing on a desk at up to 73 words per minute. The project isn’t ready for Oculus Quest VR headsets yet, which already have hand tracking but lack the support for typing input. The research project looks to be using extra tracking sensors right now, along with specialized gloves studded with motion-capture markers. But the idea is to eventually evolve the Oculus VR hand tracking to enable typing, just like it does hand tracking now.
Facebook’s motion model sees the keystrokes as they’re happening in 3D space.
“To support touch typing without a physical keyboard — and without the benefit of haptic feedback from individual keys — the team had to make sense of erratic typing patterns,” Facebook’s post says of the research project. Facebook’s typing AI is adapted from speech recognition, but searching for particular keystroke motions amid the general finger movement. It works predictively to anticipate letters as they’re being typed. The research so far is producing typing at “73 words per minute with a 2.4% uncorrected error rate using their hands, a flat surface, and nothing else,” Facebook’s post claims.
Facebook’s already supporting certain VR-optimized physical keyboards with Oculus Quest VR by the end of the year, but keyboard-free typing could be a way for VR and eventually smart glasses to work without any accessories at all.
