Zhu Wang

Purpose: To investigate how adults with unilateral vestibular hypofunction and healthy controls incorporate visual and auditory cues for postural control in an abstract visual environment.
Conclusions: Patients with vestibular hypofunction used sounds to reduce sway in a static abstract environment and were somewhat destabilized by it in a dynamic environment. This suggests that sounds, when played from headphones, may function as an auditory anchor under certain level of challenge and specific tasks regardless of whether it’s stationary or moving. Our results support that increased sway in middle frequencies reflects vestibular dysfunction.

We introduce RoboTerrain, a cost-efficient and expandable shape-changing system which utilizes a self-lifting structure composed of modular robotics that actuate support beams. RoboTerrain generates dynamic surface displays and enclosures by modulating a grid of robotic units with linear movements, each with two actuators and four beams connecting to adjacent units. The modular design allows the structure to scale to different grid sizes and to be arranged in flexible layouts. The self-lifting nature of RoboTerrain makes it possible to utilize the space on both sides of the surface. The design of a sparse grid structure makes the system more efficient in simulating large-scale structures such as smart architecture, and the spaces between the beams enable objects to pass through the actuated surface for novel interactions. In this paper, we demonstrate a few prototypes with different layouts and validate the proof of concept.

We present our VR-based terrain generation and authoring system, which utilizes hand tracking and a generative model to allow users to quickly prototype natural landscapes, such as mountains, mesas, canyons and volcanoes. Via positional hand tracking and hand gesture detection, users can use their hands to draw mid-air strokes to indicate desired shapes for the landscapes. A Conditional Generative Adversarial Network trained by using real-world terrains and their height maps then helps to generate a realistic landscape which combines features of training data and the mid-air strokes. In addition, users can use their hands to further manipulate their mid-air strokes to edit the landscapes.

Our collaborative XR system integrates physical and virtual design spaces, using video passthrough to superimpose visual modifications on physical objects. With features such as multimodal inputs, real-time physical object tracking, and object-based 3D annotation, it speeds up design iterations for digital prototyping involving with physical objects.

The system enables users to intuitively control a multi-robot system to manipulate objects on tabletop surfaces through various modalities like gestures, GUI, tangible manipulation, and speech. The multi-robot system then autonomously anc collectively executes the instructions using a generalizable control algorithm. This approach emphasizes object-level interaction, allowing users to work at a higher level without managing individual robot movements.

This paper presents the ongoing development of a spatial audio system for co-located, multi-participant, extended reality (CMXR) experiences. By integrating spatial audio and informative auditory displays, the system can enhance the sense of immersion and presence among participants and facilitate collaboration.

This study is to determine the extent to which sensory integration strategies via head sway, derived from a Head-Mounted Display (HMD), change in people with vestibular disorders following vestibular rehabilitation.

We present a system that utilizes hand-held devices for non-VR instructors, enabling them to explore VR content and interact with students who are fully immersed in VR. The instructor can observe the VR environment or switch between different students’ first-person views by using commonly available hand-held devices, such as smartphones and tablets. The instructor can also use hand-held devices to interact with the VR world itself. The students can see the real-time video stream of the physical environment as well as a video stream of the instructor.

Our approach allows for multi-modal conditional feature-driven retrieval through a 3D asset database, by utilizing a combination of input latent embeddings. We explore the effects of different combinations of feature embeddings across different input types and weighting methods.

This pilot study aimed to identify postural strategies in response to sensory perturbations (visual, auditory, somatosensory) in adults with and without sensory loss.

Our project combines immersive VR, multitouch AR, real-time volumetric capture, motion capture, robotically-actuated tangible interfaces at multiple scales, and live coding, in service of a human-centric way of collaborating.

The purpose of this study was to test the feasibility of a novel VR application (app) developed for a Head Mounted Display (HMD) to target dizziness, imbalance and sensory integration in a functional context for patients with vestibular disorders.

We present a low-cost novel VR gait assessment system that simulates virtual obstacles, visual, auditory, and cognitive loads while using motion tracking to assess participants’ walking performance. The system utilizes in-situ spatial visualization for trial playback and instantaneous outcome measures which enable experimenters and participants to observe and interpret their performance.

we present a novel walking balance assessment system with eye tracking to investigate the role of eye movement in walking balance and spatial data visualization to better interpret and understand the experimental data. The spatial visualization includes instantaneous in-situ VR replay for the gaze, head, and feet; and data plots for the outcome measures. The system fills a need to provide eye tracking and intuitive feedback in VR to experimenters, clinicians, and participants in real-time.

We developed a novel assessment platform with untethered virtual reality, 3D sounds, and pressure sensing floor mat to help assess the walking balance and negotiation of obstacles given diverse sensory load and/or cognitive load.

We investigated concurrent validity of head tracking of two Head Mounted Displays (HMDs), Oculus Rift and HTC Vive, vs. a gold-standard motion capture system (Qualisys). Our results generally support the concurrent validity of Oculus Rift and HTC Vive head tracking during static and dynamic standing tasks in healthy young adults. Specific task- and direction-dependent differences should be considered when planning measurement studies using these novel tools.

We evaluated the feasibility of a novel paradigm for evaluation of dynamic balance within complex visual environments in people with PPPD. Results indicated that performance of the FSST-VR is feasible and did not aggravate symptoms for people with PPPD.

We developed a VR HMD application that allows patients to practice contextual sensory integration (C.S.I) while sitting, standing, turning or stepping within diverse scenes. This application can become an integral part of vestibular rehabilitation. In this pilot study, usability and preliminary outcomes were tested in a mixed-methods descriptive study. Recommendation for future research and clinical implementation are discussed.

We developed a novel virtual reality [VR] platform with 3-dimensional sounds to help improve sensory integration and visuomotor processing for postural control and fall prevention in individuals with balance problems related to sensory deficits, such as vestibular dysfunction (disease of the inner ear). The 3D game-like scenes make participants feel immersed and gradually exposes them to situations that may induce dizziness, anxiety or imbalance in their daily-living.