Re: [AUDITORY] Visual references in sound localisation

[Brian FG Katz <katz@xxxxxxxxxxx>]

Regarding audio-visual integration, I would be wary of the measure of acceptance/annoyance in film production as a negation of our perception of audio-visual spatial coherence. If you have the opportunity to compare accurate audio-visual spatial coherence of the actor's voice, or musical instrument, etc. the difference is flagrant. As a measure of the "acceptable" spatial disparity, I invite you to look at a study of ours a few years ago comparing stereo and WFS reproduction and audience position:
	Subjective evaluation of the audiovisual spatial congruence in the case of stereoscopic-3D video and Wave Field Synthesis; doi:10.1016/j.ijhcs.2013.09.004

For localization tests, personally, I would avoid visual cues/anchors, unless you present a visual reporting grid that is *finer* than perceptual blur. 

Regarding audio influence visual tasks, and vice versa, I point you to a study we carried out regarding distance perception:
	Audio, visual, and audio-visual egocentric distance perception by moving subjects in virtual environments
doi:10.1145/2355598.2355602
A second study examined conditions in which comparable saliency degradations were applied to both audio and visual stimuli, allowing for an investigation of the audio/visual decision weighting in a search task. 
	Cueing multimedia search with audio-visual blur; doi:10.1145/2465780.2465781

Best regards,
--
Brian FG Katz, Ph.D, HDR
Research Director, CNRS
Groupe Lutheries - Acoustique - Musique 
Sorbonne Université, CNRS, UMR 7190, Institut Jean Le Rond ∂'Alembert
bureau 510, 5ème, allé des tours 55-65
4, Place Jussieu 
75005 Paris 

From: AUDITORY - Research in Auditory Perception [mailto:AUDITORY@xxxxxxxxxxxxxxx] On Behalf Of Les Bernstein
Sent: Sunday, February 25, 2018 6:52 PM
To: AUDITORY@xxxxxxxxxxxxxxx
Subject: Re: [AUDITORY] Visual references in sound localisation

I believe the question was about what would occur under the circumstances described and the nature of visual influences.  It's also important to recognize that the visual modality is not dominant, per se.  Rather, it depends upon the reliability (indexed by variance) of the transduced information within each modality.  One can manipulate those variances and humans will generally weight the information within each modality in a near-optimal fashion (inverse to the variance).  Think Alais and Burr.
On 2/25/2018 12:50 AM, Kent Walker wrote:
When doing localization tests best practice is to use visually-opaque acoustically-transparent curtains. However, it's also best practice to provided respondents with visual references which they can use to respond.  

Depending on the perceptual task, providing a reference stimulus with known location (visual & acoustic) can be extremely useful. 

In audio engineering, things get more interesting when visual and auditory cues are in different spatial locations. 

For example, in film sound mixing dialogue is pretty much always mixed to the centre channel only, even when actors are visible at the left and right of the projected image. There are technical limitations that prevent using phantom sources to match the sound to the  viewed location of the actors. The visual-auditory mismatch is generally not annoying or troublesome and we perceive the dialogue as eminating from the visual location on the screen - not the physical location of the loudspeaker. In large theatres the physical mismatch between the stimuli can be quite large, routinely 30 feet.

This is because in multimodal perception vision generally dominates (think McGurk effect).

On Feb 24, 2018 22:11, "Engel Alonso-Martinez, Isaac" mailto:isaac.engel@xxxxxxxxxxxxxx wrote:
Dear all,

I am interested in the impact of audible visual references in sound localisation tasks.

For instance, let's say that you are presented two different continuous sounds (e.g., speech) coming from sources A and B, which are in different locations. While source A is clearly visible to you, B is invisible and you are asked to estimate its location. Will source A act as a spatial reference, helping you in doing a more accurate estimation, or will it be distracting and make the task more difficult?

If anyone can point to some literature on this, it would be greatly appreciated.

Kind regards,
Isaac Engel


-- 
Leslie R. Bernstein, Ph.D. | Professor
Depts. of Neuroscience and Surgery (Otolaryngology)| UConn School of Medicine 
263 Farmington Avenue, Farmington, CT 06030-3401
Office: 860.679.4622 | Fax: 860.679.2495