CraGL: Computational Reality, Creativity and Graphics Lab

ShapeSonic: Sonifying Fingertip Interactions for Non-Visual Virtual Shape Perception

Jialin Huang, Rana Hanocka, Alexa Siu, Yotam Gingold
ACM SIGGRAPH Asia 2023 (Conference Papers)

Paper

Supplemental Material

A hand moves from the left to touch a pyramid with four fingers. As the hand moves closer, it glows more strongly. Each fingertip in contact with the pyramid has a musical note labeled with the letters A, G, E, and D, respectively. A hand also moves from the right to touch a cat's ear. As the hand moves closer, it glows more strongly. The fingertip in contact has a musical note labeled with the letter E. Users’ hands are guided to shape surfaces with sonified distance. Fingertips contacting the shape play notes in a pentatonic scale. Sharp edges and corners are sonified with distinct sounds. These shapes were used in our landmark localization experiment. Users explored the shapes with ShapeSonic, and then were asked to localize features.

Abstract

For sighted users, computer graphics and virtual reality allow them to model and perceive imaginary objects and worlds. However, these approaches are inaccessible to blind and visually impaired (BVI) users, since they primarily rely on visual feedback. To this end, we introduce ShapeSonic, a system designed to convey vivid 3D shape perception using purely audio feedback or sonification. ShapeSonic tracks users' fingertips in 3D and provides real-time sound feedback (sonification). The shape's geometry and sharp features (edges and corners) are expressed as sounds whose volumes modulate according to fingertip distance. ShapeSonic is based on a mass-produced, commodity hardware platform (Oculus Quest). In a study with 15 sighted and 6 BVI users, we demonstrate the value of ShapeSonic in shape landmark localization and recognition. ShapeSonic users were able to quickly and relatively accurately “touch” points on virtual 3D shapes in the air.

Video (3:11 minutes)

Download as:

Fast Forward (40 seconds)

Download as:

Presentation (9 minutes)

Download as:

Demos

APK files for the Oculus Quest (285 MB each):

BibTeX

@article{Huang:2023:SSS,
 author    = {Huang, Jialin and Hanocka, Rana and Siu, Alexa and Gingold Yotam},
 title     = {Shape{S}onic: Sonifying Fingertip Interactions for Non-Visual Virtual Shape Perception},
 year      = {2023},
 booktitle = {SIGGRAPH Asia 2023 Conference Proceedings},
 series    = {SA '23},
 keywords  = {shape, perception, 3D, virtual reality, sonification, non-visual interfaces}
}