FaceWidgets blends on-face switches with the VR world

When using a virtual reality (VR) system, you may need to flip a switch, touch a button, etc., which can be represented by a carefully coordinated series of pixels in front of your eyes. As a physical alternative — or augmentation — researchers at the National Chiao Tung University in Hsinchu, Taiwan have developed a system of interchangeable physical control panels, called FaceWidgets, that reside on the backside of head-mounted unit itself.

When a wearer places their palm near their face (and headset), this is sensed and an on-screen canvas appears depending on the application. They can then manipulate these widgets both physically and in the virtual world to control the experience. 

Physical interactions are detected with the help of an Arduino Mega and the facial control pad even extends and retracts for optimal usage via a motor shield and stepper motors.

We present FaceWidgets, a device integrated with the backside of a head-mounted display (HMD) that enables tangible interactions using physical controls. To allow for near range-to-eye interactions, our first study suggested displaying the virtual widgets at 20 cm from the eye positions, which is 9 cm from the HMD backside. We propose two novel interactions, widget canvas and palm-facing gesture, that can help users avoid double vision and allow them to access the interface as needed. Our second study showed that displaying a hand reference improved performance of face widgets interactions. We developed two applications of FaceWidgets, a fixed-layout 360 video player and a contextual input for smart home control. Finally, we compared four hand visualizations against the two applications in an exploratory study. Participants considered the transparent hand as the most suitable and responded positively to our system.