Re: audio-visual correlation

[TODD@xxxxxxxxxxxxxxxxx]

Dear List

I have also been reading with great interest the recent contributions on
A-V interactions. My own research has been concerned with the
relationship between auditory rhythm and visual motion and over the last
ten years I have accumulated a body of quotes on this matter, from which
selected items are listed below.

"I am speaking of things moved in the way that the voice is moved in
speaking and singing, and the body in making a gesture and dancing".
Aristoxenus of Tarentum c. 320 BC

 "The material of music is sound and bodily movement". Aristdes
Quintilianus c. 300 AD

"Rhythm in speech, rhythm in sound and rhythm in motion , were in the
beginning, parts of the same thing". Herbert Spencer, 1870.

"...hence it becomes possible for motion in music to imitate the
peculiar characteristics of motive forces in space, that is, to form an
image of the various impulases and forces which lie at the root of
motion. And on this, as I believe, essentially depends the power of
music to picture emotion". Helmholtz, 1885.

"The dynamic form of a phrase is a form of a movement.. In reciting
verse, or in singing, a phrase becomes a single act of expiration;
indeed, just this movement of breathing is probably the origin of
musical phrasing." R.H. Stetson, 1905

"The origin of music lies in inner motion...a pure sensation of
motion...which may be counted among the vestibular sensations...Musical
motion can be likened to an invisible, imaginary dance which is free
from all phusical constraints." Alexander Truslitt, 1938.

Paul Fraisse (1982) also gives an account of the etymolology of the word
rhythm. "Rhythm comes from the Greek words 'rithmos' [sorry can't get
Greek characters] and 'reo' (to flow).... 'Rhythmos' appears as one of
the key words in Ionian philosophy, generally meaning 'form', but an
improvised, momentary, and modifiable form. 'Rhythmos' literally
signifies a 'particular way of flowing'..."

>From a computational view this essential unity of rhythm and motion can
be realised if one considers the spatio-temporal power spectrum of
visual and auditory images. E.g.

Heeger, D. (1987) A model for the extraction of image flow. J. Opt. Soc.
Am. 4(8), 1454-1470.

Todd, N.P.McAngus (1998) A model of auditory image flow. Proc. 16th ICA.

In terms on neurophysiology, as has been mentioned by Robert Zatorre,
there exist various multi-modal areas, including in the cortex. V5, the
visual motion area of the cortex, lives in the temporal lobe. It makes
sense for the brain to evolve a common strategy for coding motion across
all senses and indeed recent physiology (e.g. Kowalski et al, 1996. J.
Neurophys. 76(5), 3504-3523.) is providing strong evidence for this. It
should not suprise us that audio-visual interactions occur if there is
high cross-correlation in the temporal power spectrum. On this basis I
have developed some software for real-time visualisation of rhythm. I
gave a little demo. at the ICMPC96 at Montreal. Goes down a treat with
Manchester ravers!

cheers

Neil Todd

Dept. of Psychol.
Univ. of Manc.
Manc.
UK