Spatial audio feature discovery with convolutional neural networks

The advent of mixed reality consumer products brings about a pressing need to develop and improve spatial sound rendering techniques for a broad user base. Despite a large body of prior work, the precise nature and importance of various sound localization cues and how they should be personalized for an individual user to improve localization performance is still an open research problem. Here we propose training a convolutional neural network (CNN) to classify the elevation angle of spatially rendered sounds and employing Layerwise Relevance Propagation (LRP) on the trained CNN model. LRP provides saliency maps that can be used to identify spectral features used by the network for classification. These maps, in addition to the convolution filters learned by the CNN, are discussed in the context of listening tests reported in the literature. The proposed approach could potentially provide an avenue for future studies on modeling and personalization of head-related transfer functions (HRTFs).

Link (opens in new tab) to presentation slides

Ipsilateral and contralateral HRTF magnitude responses (top) at 30 degrees lateral angle for subject 154 from the CIPIC dataset, and the corresponding saliency maps (bottom) for CNN model HP produced via Layer-wise Relevance Propagation (LRP).