publications | Zhu Wang

Award † Equal contribution ‡ Equal advising

2025

Celestial Blossom: A Collaborative, Meditative VR Journey of Fractal Art and Music

Pincun Liu, Keru Wang, Yi Wu, Xiaoan Liu, Yushen Hu, Zhu Wang, Agnieszka Roginska, and Ken Perlin

In SIGGRAPH Asia 2025 XR, 2025

Abs DOI PDF Video

Celestial Blossom is a collaborative, meditative VR experience where participants co-create generative fractal art through hand-drawn sketches, transforming movements into evolving audiovisual symphonies. Set in a celestial void, strokes morph into glowing fractal particles, from microscopic grains to cosmic stars, each shaping sonic textures and evolving into an immersive soundscape. Multiplayer interaction allows users to weave their creations into a shared immersive composition. At the end, participants receive a printed rendering of their artwork, which can be scanned to revisit the fractals and melodies they created in VR.
Audio-Influenced Pseudo-Haptics: A Review of Effects, Applications, and Research Directions

Keru Wang, Yi Wu, Pincun Liu, Zhu Wang, Agnieszka Roginska, Qi Sun, and Ken Perlin

In Proceedings of the 30th International Conference on Auditory Display (ICAD), 2025

Abs PDF

Haptic feedback plays an important role in interaction, but traditional haptic devices are often difficult to access due to their complexity and high cost. A cost-effective approach is to use auditory displays to evoke pseudo-haptic sensations by leveraging the brains multisensory processing. Auditory cues can enhance or modify existing haptic feedback, refine low-resolution haptic devices, or generate convincing haptic illusions without physical contact. Despite their potential, audio-influenced pseudo-haptics remain underexplored. In this survey, we provide a comprehensive overview of existing research on audio-influenced pseudo-haptics, summarizing the mapping between different auditory cues and the pseudo-haptics that they elicit, identifying key challenges, and discussing future opportunities. We aim to provide a road map in audio-influenced pseudo-haptics research and inspire further exploration in this field, fostering innovation and guiding the development of more immersive, functional, and accessible interactive systems.
An Overview of the 1st International Workshop on Spatial Memory in XR: The Future of Memory Capture and Replay Through XR and AI (XRMemory)

Dooyoung Kim, Kangsoo Kim, Cláudio T Silva, Qi Sun, Dishita Turakhia, Zhu Wang, Keru Wang, Ken Perlin, Steven Feiner, and Woontack Woo

In 2025 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2025

Abs DOI PDF

The 1st International Workshop on Spatial Memory in eXtended Reality (XRMemory) aims to explore and advance technologies that enable immersive, memory-rich experiences, allowing users to revisit moments as if they were truly back in those spaces. While photos and videos have traditionally served this purpose, advancements in Augmented Reality (AR), Virtual Reality (VR), and spatial computing now enable far more realistic and immersive memory recall. This workshop investigates how AI and XR technologies can be harnessed to record, generate, and recall user experiences in transformative ways. By enabling individuals to relive their own memories or immerse themselves in others’ experiences as if physically present, it seeks to open new avenues for shared connections that transcend the limitations of time and space.
Frequency analyses of postural sway demonstrate the use of sounds for balance given vestibular loss

Anat V. Lubetzky, Maura Cosetti, Daphna Harel, Katherine Scigliano, Marlee Sherrod, Zhu Wang, Agnieszka Roginska, and Jennifer Kelly

Gait & Posture, 2025

Abs DOI PDF

Purpose To investigate how adults with unilateral vestibular hypofunction and healthy controls incorporate visual and auditory cues for postural control in an abstract visual environment. Methods Participants stood on foam wearing the HTC Vive, observing an immersive 3-wall display of ‘stars’ that were either static or dynamic (moving front to back at 32 mm, 0.2 Hz) with no sound, static white noise, or moving white noise played via headphones. Each 60-second condition repeated twice. We recorded the center-of-pressure variance, and its power spectral density [PSD, cm2] components in low [0, 0.25 Hz], mid [0.25, 0.5 Hz] and high [0.5, 1 Hz] frequencies in the anterior-posterior direction. We used linear mixed-effects models to compares healthy controls (n = 41, mean age 52 years, range 22–78) to participants with unilateral peripheral vestibular hypofunction (n = 28, 61.5, 27–82), adjusting for age. Results Variance and low PSD: we observed a significant vestibular by visual load interaction in the presence of sounds, such that the vestibular group had significantly higher sway than controls only on dynamic visuals in the presence of sounds. Mid PSD: the vestibular group had significantly higher sway than controls regardless of condition. High PSD: the vestibular group had significantly higher sway than controls, except for the presence of sounds on static visuals. Conclusions Patients with vestibular hypofunction used sounds to reduce sway in a static abstract environment and were somewhat destabilized by it in a dynamic environment. This suggests that sounds, when played from headphones, may function as an auditory anchor under certain level of challenge and specific tasks regardless of whether it’s stationary or moving. Our results support that increased sway in middle frequencies reflects vestibular dysfunction.
Robotecture: A Modular Shape-changing Interface Using Actuated Support Beams

Yuhan Wang, Keru Wang, Zhu Wang^‡, and Ken Perlin^‡

In Proceedings of the Nineteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI), 2025

Abs DOI PDF Video

Shape-changing User Interfaces (SCUIs) can dynamically adjust their shape or layout in response to user interactions or environments. It is challenging to design expandable, affordable, and effective SCUIs with optimal space utilization for novel interactions. To tackle this challenge we introduce Robotecture, a cost-efficient and expandable shape-changing system which utilizes a self-lifting structure composed of modular robotics that actuate support beams. Robotecture generates dynamic surface displays and enclosures by modulating a grid of robotic units with linear movements, each with two actuators and four beams connecting to adjacent units. The modular design allows the structure to scale to different grid sizes and to be arranged in flexible layouts. The self-lifting nature of Robotecture makes it possible to utilize the space on both sides of the surface. The design of a sparse grid structure makes the system more efficient in simulating large-scale structures such as smart architecture, and the spaces between the beams enable objects to pass through the actuated surface for novel interactions. In this paper, we demonstrate a few prototypes with different layouts and validate the proof of concept. Additionally, we showcase various scenarios where Robotecture can enhance tangible interactions and interactive experiences for versatile and eco-friendly applications across different scales, such as tabletop tangible displays, room-scale furniture simulation, smart architecture, etc.

2024

A Spatial Audio System for Co-Located Multi-Participant Extended Reality Experiences

Yi Wu, Agnieszka Roginska, Keru Wang, Zhu Wang, and Ken Perlin

In Proceedings of the 29th International Conference on Auditory Display (ICAD), 2024

Abs PDF

This paper presents the ongoing development of a spatial audio system for co-located, multi-participant, extended reality (CMXR) experiences. By integrating spatial audio and informative auditory displays, the system can enhance the sense of immersion and presence among participants and facilitate collaboration. We navigate this development by addressing the challenges encountered during the implementation of spatial audio in server-client frameworks and CM-XR environments, including accurate audio localization, synchronization across multiple users, and designing immersive and informative sounds. The paper covers the architecture of the system, along with technical features, sound design choices, and the use of audio for providing information to participants about the environment. This exploration offers insights into spatial audio integration and suggests directions for future research in the design and implementation of spatial auditory display solutions for CM-XR settings.
Generative Terrain Fast Prototyping in Virtual Reality with Freehand Interface

Yushen Hu, Keru Wang, Zhu Wang, and Ken Perlin

In SIGGRAPH Asia 2024 XR, 2024

Abs DOI PDF Video

Terrain generation and authoring in Virtual Reality (VR) offers unique benefits for terrain authoring including stereo display, immersive and intuitive design experience, and natural input modalities. We present this VR-based terrain fast prototyping system to integrate natural input modalities, preserve artistic controls and lower the effort of landscape prototyping. The system utilizes freehand interfaces and a generative model to help users quickly prototype different types of natural landscapes, such as mountains, mesas, canyons, and volcanoes. With the freehand interfaces, users can use their hands to draw mid-air strokes as the 3D contours of the desired landscapes. Then, a Conditional Generative Adversarial Network (CGAN) generates realistic landscapes based on the 3D contour. The freehand interface detects users’ gestures to control landscape editing. By incorporating CGAN as the terrain synthesizer, our system allows users to immersively, rapidly, and easily prototype terrains using intuitive interactive hand controls.
Generative Terrain Authoring with Mid-air Hand Sketching in Virtual Reality

Yushen Hu, Keru Wang, Yuli Shao, Jan Plass, Zhu Wang^‡, and Ken Perlin^‡

In Proceedings of the 30th ACM Symposium on Virtual Reality Software and Technology (VRST), 2024

Abs DOI PDF Video Best Paper Award

Terrain generation and authoring in Virtual Reality (VR) offers unique benefits, including 360-degree views, improved spatial perception, immersive and intuitive design experience and natural input modalities. Yet even in VR it can be challenging to integrate natural input modalities, preserve artistic controls and lower the effort of landscape prototyping. To tackle these challenges, we present our VR-based terrain generation and authoring system, which utilizes hand tracking and a generative model to allow users to quickly prototype natural landscapes, such as mountains, mesas, canyons and volcanoes. Via positional hand tracking and hand gesture detection, users can use their hands to draw mid-air strokes to indicate desired shapes for the landscapes. A Conditional Generative Adversarial Network trained by using real-world terrains and their height maps then helps to generate a realistic landscape which combines features of training data and the mid-air strokes. In addition, users can use their hands to further manipulate their mid-air strokes to edit the landscapes. In this paper, we explore this design space and present various scenarios of terrain generation. Additionally, we evaluate our system across a diverse user base that varies in VR experience and professional background. The study results indicate that our system is feasible, user-friendly and capable of fast prototyping.
A Collaborative Multimodal XR Physical Design Environment

Keru Wang, Pincun Liu, Yushen Hu, Xiaoan Liu, Zhu Wang, and Ken Perlin

In SIGGRAPH Asia 2024 XR, 2024

Abs DOI PDF Video

Traditional design processes for physical prototypes can be time-consuming and costly due to iterations of prototyping, testing, and refinement. Extended Reality (XR) technology with video passthrough offers unique benefits for alleviating these issues by providing instant visual feedback. We have developed an XR system with multimodal input capability that provides annotations and enables interactive visual modifications by superimposing and aligning visual counterparts to physical objects. This system can help designers to quickly experiment with and visualize a wide range of design options, keep track of design iterations, and explore innovative solutions without the constraints of physical prototyping. As a result, it can significantly speed up the iterative design process, while requiring fewer physical modifications in each iteration.
“Push-That-There”: Tabletop Multi-robot Object Manipulation via Multimodal’Object-level Instruction’

Keru Wang, Zhu Wang, Ken Nakagaki, and Ken Perlin

In Proceedings of the 2024 ACM Designing Interactive Systems Conference (DIS), 2024

Abs DOI PDF Video

We present "Push-That-There", an interaction method and system enabling multimodel object-level user interaction with multi-robot system to autonomously and collectively manipulate objects on tabletop surfaces, inspired by "Put-That-There". Rather than requiring users to instruct individual robots, users directly specify how they want the objects to be moved, and the system responds by autonomously moving objects via our generalizable multi-robot control algorithm. The system is combined with various user instruction modalities, including gestures, GUI, tangible manipulation, and speech, allowing users to intuitively create object-level instruction. We outline a design space, highlight interaction design opportunities facilitated by "Push-That-There", and provide an evaluation to assess our system’s technical capabilities. While other recent HCI research has studied interaction using multi-robot system (e.g. Swarm UIs), our contribution is in the design and technical implementation of intuitive object-level interaction for multi-robot system that allows users to work at a high level, rather than needing to focus on the movements of individual robots.

2023

Asymmetrical VR for Education

Keru Wang, Zhu Wang, and Ken Perlin

In ACM SIGGRAPH 2023 Immersive Pavilion, 2023

Abs DOI PDF Video

In collocated VR classes, instructors need to guide their students, while also remaining aware of the physical environment in order to ensure students’ safety. It is hard to do both simultaneously.We present a system that utilizes hand-held devices for non-VR instructors, enabling them to explore VR content and interact with students who are fully immersed in VR. The instructor can observe the VR environment or switch between different students’ first-person views by using commonly available hand-held devices, such as smartphones and tablets. The instructor can also use hand-held devices to interact with the VR world itself. The students can see the real-time video stream of the physical environment as well as a video stream of the instructor. The system enables seamless communication and collaboration, thereby helping to create a better and richer educational experience for VR classes.
Decrease in head sway as a measure of sensory integration following vestibular rehabilitation: A randomized controlled trial

Anat V. Lubetzky, Daphna Harel, Santosh Krishnamoorthy, Gene Fu, Brittani Morris, Andrew Medlin, Zhu Wang, Ken Perlin, Agnieszka Roginska, Maura Cosetti, and Jennifer Kelly

Journal of Vestibular Research, 2023

Abs DOI PDF

Objective: The purpose of this study was to determine the extent to which sensory integration strategies via head sway, derived from a Head-Mounted Display (HMD), change in people with vestibular disorders following vestibular rehabilitation.
Design: Randomized Controlled Trial
Setting: Vestibular Rehabilitation Clinic
Participants: Thirty participants with vestibular dysfunction and 21 age-matched controls.
Main Outcome Measures: Participants experienced two levels of visual surround (static or moving ‘stars’, front to back at 0.2 Hz, 32 mm) and white noise (none or rhythmic) while their head sway was recorded via the HTC Vive. We quantified head sway via Directional Path (DP) and Root Mean Square Velocity (RMSV) in 5 directions: anterior-posterior, medio-lateral, pitch, yaw, and roll and Power Spectral Density in low (PSD 1), medium (PSD 2) and high (PSD 3) frequencies in the anterior-posterior direction.
Interventions: Participants performed the assessment prior to being randomized into 8-weeks of contextual sensory integration training in virtual reality or traditional vestibular rehabilitation and once again following completion of the intervention. Controls performed the assessment once. Twelve participants dropped out, half due to covid lock-down. We applied an intention to treat analysis.
Results: We observed significant increases in AP DP, RMSV and all PSDs with change in visual level. Both intervention groups significantly decreased medio-lateral, pitch and roll DP and RMSV and anterior-posterior PSD 2 with no group differences. Vestibular participants were significantly higher than controls on all outcomes pre rehabilitation. Post rehabilitation they were only significantly higher on PSD 2. Sound was not a significant predictor of head sway in this protocol.
Conclusions: Head sway decreased following vestibular rehabilitation regardless of visual load or type of intervention applied. This change was measured via head kinematics derived from a portable HMD which can serve as a sensitive in-clinic assessment for tracking improvement over time

2022

Zero-Shot Multi-Modal Artist-Controlled Retrieval and Exploration of 3D Object Sets

Kristofer Schlachter^†, Benjamin Ahlbrand^†, Zhu Wang, Ken Perlin, and Valerio Ortenzi

In SIGGRAPH Asia 2022 Technical Communications, 2022

Abs DOI PDF Video

When creating 3D content, highly specialized skills are generally needed to design and generate models of objects and other assets by hand. We address this problem through high-quality 3D asset retrieval from multi-modal inputs, including 2D sketches, images and text. We use CLIP as it provides a bridge to higher-level latent features. We use these features to perform a multi-modality fusion to address the lack of artistic control that affects common data-driven approaches. Our approach allows for multi-modal conditional feature-driven retrieval through a 3D asset database, by utilizing a combination of input latent embeddings. We explore the effects of different combinations of feature embeddings across different input types and weighting methods.
Insight into postural control in unilateral sensorineural hearing loss and vestibular hypofunction

Anat V Lubetzky, Jennifer L Kelly, Daphna Harel, Agnieszka Roginska, Bryan D Hujsak, Zhu Wang, Ken Perlin, and Maura Cosetti

PloS One, 2022

Abs DOI PDF

This pilot study aimed to identify postural strategies in response to sensory perturbations (visual, auditory, somatosensory) in adults with and without sensory loss. We tested people with unilateral peripheral vestibular hypofunction (N = 12, mean age 62 range 23–78), or with Unilateral Sensorineural Hearing Loss (USNHL, N = 9, 48, 22–82), or healthy controls (N = 21, 52, 28–80). Postural sway and head kinematics parameters (Directional Path in the anterior-posterior and medio-lateral directions (sway & head); pitch, yaw and roll (head) were analyzed in response to 2 levels of auditory (none, rhythmic sounds via headphones), visual (static, dynamic) and somatosensory cues (floor, foam) within a simulated, virtual 3-wall display of stars. We found no differences with the rhythmic auditory cues. The effect of foam was magnified in the vestibular group compared with controls for anterior-posterior and medio-lateral postural sway, and all head direction except for medio-lateral. The vestibular group had significantly larger anterior-posterior and medio-lateral postural sway and head movement on the static scene compared with controls. Differences in pitch, yaw and roll emerged between vestibular and controls only with sensory perturbations. The USNHL group did not increase their postural sway and head movement with the increased visual load as much as controls did, particularly when standing on the foam. They did not increase their medio-lateral sway with the foam as much as controls did. These findings suggest that individuals with USNHL employ a compensatory strategy of conscious control of balance, the functional implications of which need to be tested in future research.
Mixed Reality Collaboration for Complementary Working Styles

Keru Wang, Zhu Wang, Karl Rosenberg, Zhenyi He, Dong Woo Yoo, Un Joo Christopher, and Ken Perlin

In ACM SIGGRAPH 2022 Immersive Pavilion, 2022

Abs DOI PDF Video

Our project combines immersive VR, multitouch AR, real-time volumetric capture, motion capture, robotically-actuated tangible interfaces at multiple scales, and live coding, in service of a human-centric way of collaborating. Participants bring their unique talents and preferences to collaboratively tackle complex problems in a shared mixed reality world.
Contextual sensory integration training via head mounted display for individuals with vestibular disorders: a feasibility study

Anat V. Lubetzky, Jennifer Kelly, Zhu Wang, Marta Gospodarek, Gene Fu, John Sutera, and Bryan D. Hujsak

Disability and Rehabilitation: Assistive Technology, 2022

Abs DOI PDF

Purpose:Virtual reality (VR) interventions can simulate real-world sensory environments. The purpose of this study was to test the feasibility of a novel VR application (app) developed for a Head Mounted Display (HMD) to target dizziness, imbalance and sensory integration in a functional context for patients with vestibular disorders. Here we describe the design of the app as well as self-reported and functional outcomes in vestibular patients before and after participating in vestibular rehabilitation using the app.
Material and methods: Our app includes a virtual street, airport, subway or a park. The clinician controls the visual and auditory load including several levels of direction, amount and speed of virtual pedestrians. Clinicians enrolled 28 patients with central (mild-traumatic brain injury [mTBI] or vestibular migraine) and peripheral vestibular disorders. We recorded the Simulator Sickness Questionnaire, Visual Vertigo Analogue Scale (VVAS), Dizziness Handicap Inventory (DHI), Activities-Specific Balance Confidence Scale (ABC), 8-foot up and go (8FUG) and Four-Step Square Test (FSST) before and after the intervention
Results:Within the 15 patients who completed the study, 12 with peripheral hypofunction showed significant improvements on the VVAS (p = 0.02), DHI (p = 0.008) and ABC (p = 0.02) and a small significant improvement on the FSST (p = 0.015). Within-session changes in symptoms were minimal. Two patients with mTBI showed important improvements, but one patient with vestibular migraine, did not.
Conclusion: HMD training within increasingly complex immersive environments appears to be a promising adjunct modality for vestibular rehabilitation. Future controlled studies are needed to establish effectiveness.

2021

VRGaitAnalytics: Visualizing Dual Task Cost for VR Gait Assessment

Zhu Wang, Liraz Arie, Anat V. Lubetzky, and Ken Perlin

In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology (VRST), 2021

Abs DOI PDF Video

Among its many promising applications, Virtual Reality (VR) can simulate diverse real-life scenarios and therefore help experimenters assess individuals’ gait performance (i.e., walking) under controlled functional contexts. VR-based gait assessment may provide low-risk, reproducible and controlled virtual environments, enabling experimenters to investigate underlying causes for imbalance by manipulating experimental conditions such as multi-sensory loads, mental processing loads (cognitive load), and/or motor tasks. We present a low-cost novel VR gait assessment system that simulates virtual obstacles, visual, auditory, and cognitive loads while using motion tracking to assess participants’ walking performance. The system utilizes in-situ spatial visualization for trial playback and instantaneous outcome measures which enable experimenters and participants to observe and interpret their performance. The trial playback can visualize any moment in the trial with embodied graphic segments including the head, waist, and feet. It can also replay two trials at the same time frame for trial-to-trial comparison, which helps visualize the impact of different experimental conditions. The outcome measures, i.e., the metrics related to walking performance, are calculated in real-time and displayed as data graphs in VR. The system can help experimenters get specific gait information on balance performance beyond a typical clinical gait test, making it clinically relevant and potentially applicable to gait rehabilitation. We conducted a feasibility study with physical therapy students, research graduate students, and licensed physical therapists. They evaluated the system and provided feedback on the outcome measures, the spatial visualizations, and the potential use of the system in the clinic. The study results indicate that the system was feasible for gait assessment, and the immediate spatial visualization features were seen as clinically relevant and useful. Limitations and considerations for future work are discussed.
Walking Balance Assessment with Eye-tracking and Spatial Data Visualization

Zhu Wang, Anat Lubetzky, and Ken Perlin

In ACM SIGGRAPH 2021 Immersive Pavilion, 2021

Abs DOI PDF Video

Virtual Reality (VR) based assessment systems can simulate diverse real-life scenarios and help clinicians assess participants’ performance under controlled functional contexts. Our previous work demonstrated an assessment paradigm to provide multi-sensory stimuli and cognitive load, and quantify walking balance with obstacle negotiation by motion capture and pressure sensing. However, we need to fill two gaps to make it more clinically relevant: 1. it required offline complex data processing with external statistical analysis software; 2. it utilized motion tracking but overlooked eye movement. Therefore, we present a novel walking balance assessment system with eye tracking to investigate the role of eye movement in walking balance and spatial data visualization to better interpret and understand the experimental data. The spatial visualization includes instantaneous in-situ VR replay for the gaze, head, and feet; and data plots for the outcome measures. The system fills a need to provide eye tracking and intuitive feedback in VR to experimenters, clinicians, and participants in real-time.

2020

A Virtual Obstacle Course within Diverse Sensory Environments

Zhu Wang, Anat Lubetzky, Charles Hendee, Marta Gospodarek, and Ken Perlin

In ACM SIGGRAPH 2020 Immersive Pavilion, 2020

Abs DOI PDF Video

We developed a novel assessment platform with untethered virtual reality, 3D sounds, and pressure sensing floor mat to help assess the walking balance and negotiation of obstacles given diverse sensory load and/or cognitive load. The platform provides a city-like scene with anticipated/unanticipated virtual obstacles. Participants negotiate the obstacles with perturbations of: auditory load by spatial audio, cognitive load by a memory task, and visual flow by generated by avatars movements at various amounts and speeds. A VR system tracks the position and orientation of the participant’s head and feet. A pressure-sensing walkway senses foot pressure and visualizes it in a heatmap. The system helps to assess walking balance via pressure dynamics per foot, success rate of crossing obstacles, available response time as well as head kinematics in response to obstacles and multitasking. Based on the assessment, specific balance training and fall prevention program can be prescribed.

2019

A Virtual Reality Four-Square Step Test for Quantifying Dynamic Balance Performance in People with Persistent Postural Perceptual Dizziness

Moshe M.H. Aharoni, Anat V. Lubetzky, Zhu Wang, Maya Goldman, and Tal Krasovsky

In 2019 International Conference on Virtual Rehabilitation (ICVR), 2019

Abs DOI PDF

Persistent-postural perceptual dizziness (PPPD) is a recently-defined diagnosis of chronic vestibular symptoms, which is exacerbated by exposure to moving objects and self-motion but is typically undetectable by clinical tests. The current work evaluates the feasibility of a novel paradigm for evaluation of dynamic balance within complex visual environments in people with PPPD – the Virtual Reality Four Step Square Test (FSST-VR). The FSSTVR measures spatiotemporal head kinematics while subjects perform the FSST pattern of 8 steps in a predefined sequence in a virtual environment of varying levels of visual complexity. Eight healthy individuals and 3 people diagnosed with PPPD were asked to perform the FSST-VR while spatiotemporal head kinematics and heartrate were measured. Additionally, participants reported their anxiety levels and cybersickness. Results indicated that performance of the FSST-VR is feasible and did not aggravate symptoms for people with PPPD. Descriptive statistics further may suggest that people with PPPD move less smoothly and perform smaller steps in anteroposterior direction, corresponding with the visual stimuli flow in the virtual environment. Data collection is ongoing and may provide further evidence as to dynamic balance in people with PPPD within complex virtual environments that mimic visual load in daily living.
Head Mounted Display Application for Contextual Sensory Integration Training: Design, Implementation, Challenges and Patient Outcomes

Anat V. Lubetzky, Jennifer Kelly, Zhu Wang, Makan TaghaviDilamani, Marta Gospodarek, Gene Fu, Erin Kuchlewski, and Bryan Hujsak

In 2019 International Conference on Virtual Rehabilitation (ICVR), 2019

Abs DOI PDF

Patients with vestibular disorders display difficulty with multisensory integration and complain of dizziness and imbalance in busy and complex visual environments. Patients who experience symptoms or a fall within a certain environment are likely to develop fear of that situation. It has been suggested that multisensory integration should be addressed in conditions as close as possible to real-life situations, but these are often challenging to replicate in the clinical setting. Virtual reality (VR) can provide a non-threatening method for patients to practice multisensory integration in a functional context. Advances in technology make Head Mounted Displays (HMDs) accessible and affordable in the clinic. We developed a VR HMD application that allows patients to practice contextual sensory integration (C.S.I) while sitting, standing, turning or stepping within diverse scenes. This application can become an integral part of vestibular rehabilitation. For successful implementation, usability is critical. In this pilot, usability and preliminary outcomes were tested in a mixed-methods descriptive study. Six physical therapists in a Vestibular Rehabilitation Clinic treated 12 patients with peripheral or central vestibular disorders. Therapists viewed the system as a bridge for a functional carry over from the clinic to the outside world. While they reported challenges in operating the technology within the clinical time constraints, they liked the ability to gradually introduce a challenging sensory stimulus and bring reality to the clinic. Several patients dropped out prior to completing training. Nine out of 12 patients who completed training thus far improved their disability score, 9 improved their visual vertigo and 10 improved their balance confidence following training with the app. Recommendation for future research and clinical implementation are discussed.
Head mounted displays for capturing head kinematics in postural tasks

Anat V. Lubetzky, Zhu Wang, and Tal Krasovsky

Journal of Biomechanics, 2019

Abs DOI PDF

Tracking head motion in a simple, portable and accurate manner during performance of postural tasks in a virtual reality environment could have important implications for investigating normal and pathological head kinematics. We investigated concurrent validity of head tracking of two Head Mounted Displays (HMDs), Oculus Rift and HTC Vive, vs. a gold-standard motion capture system (Qualisys). Head kinematics of N = 20 healthy young adults was quantified during static and dynamic postural tasks. While wearing the Oculus Rift or HTC Vive, participants observed moving stars (static tasks) or a flying ball (dynamic task). Head kinematics were recorded simultaneously by the Rift or Vive and Qualisys camera system. We calculated head directional path, acceleration in 6 directions and volume of translation movement. Intra-Class Correlations (ICC) and 95% Limits of agreement were calculated. Most ICC values were around 0.9 with several at 0.99 indicating excellent agreement between the HMDs and Qualisys. Weaker agreement was observed for vertical displacement during a static task and moderate agreement was observed pitch and yaw displacement during a dynamic task. A negative bias of a small magnitude (indicating more movement in VR) was observed for most variables in static tasks, while a positive bias was observed for most variables in the dynamic task (indicating less movement in VR). Our results generally support the concurrent validity of Oculus Rift and HTC Vive head tracking during static and dynamic standing tasks in healthy young adults. Specific task- and direction-dependent differences should be considered when planning measurement studies using these novel tools.
Virtual Environments for Rehabilitation of Postural Control Dysfunction

Zhu Wang, Anat V. Lubetzky, Marta Gospodarek, Makan TaghaviDilamani, and Ken Perlin

arXiv, 2019

Abs DOI

We developed a novel virtual reality [VR] platform with 3- dimensional sounds to help improve sensory integration and visuomotor processing for postural control and fall prevention in individuals with balance problems related to sensory deficits, such as vestibular dysfunction (disease of the inner ear). The system has scenes that simulate scenario-based environments. We can adjust the intensity of the visual and audio stimuli in the virtual scenes by controlling the user interface (UI) settings. A VR headset (HTC Vive or Oculus Rift) delivers stereo display while providing real-time position and orientation of the participants’ head. The 3D game-like scenes make participants feel immersed and gradually exposes them to situations that may induce dizziness, anxiety or imbalance in their daily-living.