Consciousness, Literature and the Arts
Archive
Volume 2 Number 2, July 2001
_______________________________________________________________
Synaesthesia
- A Cognitive Model of Cross Modal Association
Composition
Unit
The
Sydney Conservatorium of Music.
The
University of Sydney
Sydney
NSW 2000 Australia
http://www.vislab.usyd.edu.au/user/alyons
Abstract
The cognitive characteristics of the rare perceptual condition known as synesthesia provides a clinical insight into the relationship between the various human sensory modeliaties and in particular for the relationship between audition and vision. Following a discussion of the nature of synesthetic perception, this nature is discussed within the context of the relationship between auditory and visual arts.
1
Introduction
Traditionally,
the arts have been separated into disciplines delimited by medium and other
criterion. Painting and music for example are delimited by, amongst other
things, the different senses by which they are perceived - we hear music and see
painting. One of the great dreams of the romantic tradition has been that works
particular to each artistic discipline might be meaningfully represented in
another artistic discipline. One of the great challenges to the
inter-disciplinary translation of artworks has been the development of a system
of mapping perceptual attributes between each of the five human senses. Whilst
mapping betwen sculpture and painting may be achieved in a very literal way,
mapping between music and painting has always presented itself as more of a
challenge. A study of human perception, especially as it pertains to the
relationship between audition and vision, can prove very useful toward this end.
2
Clinical Synaesthesia
2.1
Human Sensory Systems
"Human
sensory systems mediate four attributes of a stimulus that can be correlated
quantitatively with a sensation: Modality, intensity, duration and
location." [1] The attributes of intensity, duration
and location apply to all five sensory modalities: vision, hearing, touch, taste
and smell. Each of these sensory modalities has sub modalities, which in the
case of vision include color whilst in hearing they include pitch. Our
perception of light arrives to the brain via a series of Photo-receptive rods
and cones in the eye. Audition on the other hand uses information gathered by
mechano-receptive hair cells in the ear that measure vibrations in air pressure.
The nature of the differences between the five modalities is suggested by the
disparate nature of these sensory receptors.
Whilst
both photo-receptors and mechano-receptors measure intensity, location and
duration, they both also measure a property of frequency. Pythagorus, Sir Isaac
Newton, and numerous other physicists, have hypothesised about the existence of
a physical relationship between the frequencies of light and sound responsible
for the sub-modalities of colour and pitch. However, explanations of the
relationships that exist between sensory sub-modalities, have been made in more
recent times by Psychologists and Neurophisicists.
2.2
Definition of Synesthesia
"Synesthesia
is a word derived from the Greek words 'syn' meaning together and 'aisthesis'
meaning perception. It is used to describe the involuntary physical experience
of a cross-modal association. That is, the stimulation of one sensory modality
reliably causes a perception in one or more different senses.’ [2]
For example, a synesthete will see coloured shapes projected into their field of
vision as a result of auditory stimulation.
2.3
Does Synesthesia Exist?
Synesthesia
has in the past been considered a less than scientific area of research by some,
due to its reliance on subjective sources for any observations of its nature.
Scientists have recently been convinced of the existence of synesthesia and cite
evidence in support:
·
The impressive test-retest reliability in the consistency of colours
triggered by different words (in the case of "coloured hearing").
·
The similarity of reports from different cultures and different times
across the century.
·
The consistency of sex ratio (it is overwhelmingly a female condition).
·
The familial pattern to the condition.
·
The neuroimaging data (using PET) showing different cortical blood flow
patterns in women with synaesthesia in comparison to women without the
condition.
2.4
Clinical Diagnosis of Synesthesia.
Psychologists
and more recently Neuropsychologists have documented the nature of synesthetic
experience in a useful manner for over a hundred years. Varying criteria has
been applied to the diagnosis of synesthesia although in general psychologists
have always differentiated clinical synesthesia from metaphor, literary tropes,
sound symbolism, and deliberate artistic contrivances that sometimes employ the
term "synesthesia" to describe their multi-sensory joinings. Dr.
Richard Cytowic has proposed five criteria for the diagnosis of a type of
clinical synesthesia called ideopathic or developmental synesthesia as opposed
to acquired forms of clinical synesthesia such as drug induced synesthesia,
epileptic synesthesia, and synesthesia due to acquired brain lesions:
·
Synesthesia is involuntary but elicited.
·
Synesthesia is projected. If visual, a photism will appear outside the
body in the region close to the face.
·
Synesthetic percepts are durable and discrete. The associations for an
individual Synesthete are stable over their lifetime. If a sound is blue, it
will always be blue.
·
Synesthetic experience is memorable. Many synesthetes exhibit hypermnesis.
·
Synesthesia is emotional in nature. A synesthetic experience is
accompanied by a sense of noetic certitude.
2.5
Non Uniformity in Synesthetic Perception.
In
addition to these characteristics it should be added that there is no uniformity
amongst the experience of synesthetes. Each individual experiencing synesthesia
experiences it in a unique form. "In fact, this rather glaring problem -
that two individuals with the same sensory pairings do not report identical, or
even similar, synesthetic responses - has sometimes been taken as 'proof' that
synesthesia is not 'real.'”[3] Yet it remains that
certain patterns have remained constant in the statistical information derived
from scientific observation of synesthetic perception. Some of these patterns,
such as the correpondance between pitch and visual brightness, have been
documented repeatedly since they were first described in the experiments of
Bleuler and Lehmann in 1881.
Lawrence
E. Marks describes the situation thus: "One should not come away with the
impression that all our knowledge about our sensory and perceptual experiences
can be captured in a set of independent - or even interrelated - verbal
categories; nor that sensory/perceptual experiences themselves reduce in any
simple manner to a list of attributes... Still, the study of synesthetic
metaphor may serve as a useful model system. By being amenable to psychophysical
analysis, synesthetic metaphors not only permit ready quantification, but enable
us to assess development trends in the ways that at least certain aspects of
such metaphors are interpreted... A psychophysics of synesthetic metaphor as
described here may eventually reveal much about perception and language; but to
appreciate the depth and extent of human metaphorical capacity will demand a
psychological analysis that is as yet hardly dreamt in our philosophy."
2.6
Explanations of Synesthetic Perception.
Over
the past 200 years a number of hypotheses have been put forward to explain the
cause of synesthesia. Current theories however in some way recognise the
findings of recent neurological studies that suggest the possibility that the
executive areas of the human brain, primarily in the frontal lobes, manifest a
high degree of sensory integration. The Cross-Modal Transfer (CMT) hypothesis is
now a widely accepted explanation for the occurrence of synesthesia although it
was radical when it was first proposed. The CMT hypothesis supports the view
that detection of intersensory equivalence is present from birth, and that
perceptual development is characterized by gradual differentiation.
2.7
The Neonatal Synaesthesia hypothesis
The
Neonatal Synaesthesia hypothesis builds on the CMT evidence, but suggests that
early in infancy, probably up to about 4 months of age, all human babies
experience sensory input in an undifferentiated way. Sounds trigger auditory,
visual and tactile experiences all at once. Following this early initial phase
of normal synaesthesia, the different sensory modalities become increasingly
modular. Adult synaesthesia, has been suggested to be as a result of a breakdown
in the process of modularization, such that during infancy the modularization
process was not completed. This of course implies that if not now, then at some
time in the past, we have all experienced synesthetic perception.
3
Synesthesia and Art
3.1
Photisms in Coloured-Hearing Synesthesia.
In
coloured hearing synesthesia, a photism, usually coloured in some way, appears
in the field of vision of a synesthete as a response to some form of aural
stimuli. Synesthetic photisms usually vary in shape and color according to the
nature of the stimuli that triggered them. "Synesthetes never see complex
dream-like scenes or have otherwise elaborate percepts. They perceive blobs,
lines, spirals, lattices, and other geometric shapes." [4]
Dr Richard Cytowic notes that the generic and restricted nature of synesthetic
percepts bear a considerable likeness to a series of forms first developed by
Heinrich Kluver in the 1920's known as Kluvers "form constants". [5]
These generic shapes are common to synesthesia, hallucinations and are
frequently seen in primitive art."
Variations in
photism color, brightness, symmetry, and shapes have been recorded to vary as a
result of variation in musical stimuli. Tempo for instance effects the shape of
a photism; the faster the music, the sharper and more angular the photism. That
pitch has a direct effect on the size of a photism has also been recorded. It
has been observed universally that photism size increases as auditory pitch
decreases. In this way high pitched sounds produce small photisms and low
pitched sounds produce synesthetic percepts that are large in size. Loudness
also has an effect on the size of the photism perceived by a synesthete.
Lawrence
E. Marks shares his understanding of Synesthetic response to music :"Just
as the important dimensions of the auditory stimulus that are responsible for
musical synesthesiae can be quite complex, so too can be the synesthetic
responses themselves." "Visual sensations aroused by music need not be
limited or confined to simple spots of color. Often the entire visual field
fills with colors that change over time with the music; some subjects report
several colors simultaneously, each color reflecting a particular aspect of the
music." [6]
3.2
Musical Perception in
Chromaesthesia.
It
is of interest to open this section by quoting the concerns of one group of
psychologists who conducted numerous investigations into synesthesia during the
first half of this century. Published in The Journal Of General Psychology in
1942, they write, "Although it is generally agreed that relationships
between visual and auditory experiences exist commonly in our language forms,
nevertheless, we have no quantitative measure of just how common a given
relationship between sound and sight actually is in the population. Such a
measure would be useful to determine what per cent of an audience could be
expected to grasp an artists purpose if , for example, he represented the
harmony of his musical composition by background and the melody by figures in
his color-music production. Also it would facilitate the process of
conventionalising associations between music and vision if one could determine
quantitatively which of several acceptable ways of representing a melody, for
instance, is already predominately in use-i.e., is considered appropriate by
most people."
At
least 23 psychological publications between 1862 and 1974 concern themselves
with correlations between sound composition and colour as a result of research
directed at synesthesia. Many more have concerned themselves with studies of
synesthesia triggered by speech stimulus. Research into these areas of
synesthesia has furnished artists with some information by which to start
developing the formulations suggested in the quote from 1942 above. In his
article, On Coloured-Hearing Synesthesia
originally published in the Psychological bulletin, Lawrence E. Marks compiled
all the information extant on such matters into a series of tables. In these
tables it can be seen that there is a very strong correlations between auditory
pitch and visual luminance. Dark hues are associated with low tones and light
colours are associated with high tones.
3.3
Vowel Colour
The
study of chromesthetic phenomena often concerns itself with associations
triggered by speech rather than music. This is perhaps due to the fact that
speech is pathologically superior in its ability to evoke a synesthetic
response. The component of speech that bears the greatest influence on the
nature of the induced response is the sound of vowels. Both areas have
tremendous significance in mapping out perceptual parallels between the
modalities of hearing and vision. Firstly, when it comes to reports on musical
synesthesia, we find that the important principles of visual-auditory
association that manifest themselves in color music are basically the same
principles that manifest themselves in coloured vowels - that is, the relations
of visual brightness and size to auditory pitch and loudness. Secondly, in an
article published in 1968, Wayne Slawson showed that artificial two formant
sounds are readily interpretable as vowels and as musical notes and that the
vowel quality and musical timbre depend in similar ways on the structure of the
sound (formant frequency and spectrum envelope.)
3.4
Slawsons' Sound
Colour
Slawson
went on to elaborate his comparison of vowel sounds to the field of musical
timbre in his Sound Color of 1985. In
this book, Slawson uses the four characteristics set out by Chomsky and Halle in
1968 in The Sound Patterns of English
and adapts them so as to form a basis for organising musical timbre derived from
varied sources. Besides being the only text to date to take up Arnold
Schoenbergs' 1911 request for a treatise on the subject of KlangenFarbe, Sound
Color forms an essential bridge between the colors commonly associated with
vowel sounds and the formant composition of synthesised sounds with which
Slawson is largely concerned. Slawson indicates that sound colour is primarily a
function of the frequencies of the first two resonances. Slawson uses the three
categories by which vowel features are organised - compactness, acuteness and
laxness, changing compactness to openness, and adding a fourth category -
smallness; which has no corresponding vowel feature.
4
Conclusion
The
nature of Synesthetic perception does not on its own provide artists with a
template for mapping between visual and auditory arts. Whilst it describes the
constant nature of the relationship between brightness and pitch, brightness and
volume, photism shape and aural texture - it does not map hue to timbre or pitch
in any way not related to a particular musical context. Also, because
synesthetic photisms are two dimensional in nature, research into synesthesia
can shed limited light on relationships between music, and objects with three
dimensions.
[1]
John H. Martin, “Coding and Processing of Sensory Information” in Principles
of Neural Science, ed. Eric R Kandel, James H. Schwartz and Thomas M. Jessel.
(London: Prentice Hall, 1991), 329.
[2]
Richard E. Cytowic, "Synesthesia, phenomenology and neuropsychology: a
review of current knowledge," Psyche 2.10 (1995), 1.
[3]
ibid.
[4]
Richard E. Cytowic, Synesthesia: a union of the senses, (New York: Springer
Verlag, 1989), 138.
[5]
Kluver, H. Mescal and Mechanisms of Hallucinations, Chicago: University of
Chicago Press, 1966.
[6]
Lawrence E. Marks, "On Coloured-Hearing Synesthesia", " in Simon
Baron-Cohen and John Harrison, eds. Synesthesia: Classic and Contemporary
Readings, (Oxford: Blackwells, 1996), 70.
6
Bibliography
Baron-Cohen,
Simon. "Is There a Normal Phase of Synaesthesia in Development?" PSYCHE 2(27), June 1996.
Baron-Cohen,
S., and J. Harrison. eds. Synesthesia:
Classic and Contemporary Readings. Oxford: Blackwells, 1996.
Bleuler,
E. and Lehmann. K. Zwangsmassige
Lichtempfindungen durch Schall und Verwandte Erscheinungen. Leipzig: Fues's
Verlag, 1881.
Chomsky,
N., and M. Halle. The Sound Pattern of
English. New York: Harper and Row, 1968.
Cytowic,
R.E. "Synesthesia, phenomenology and neuropsychology: a review of current
knowledge." in Synesthesia: Classic
and Contemporary Readings, eds., S. Baron-Cohen, and J. Harrison. Oxford:
Blackwells, 1996.
Cytowic,
R.E. Synesthesia: a union of the senses.
New York: Springer Verlag, 1989.
Frith,
Christopher D., and Eraldo Paulesu, "Physiological basis of Synesthesia."
in Synesthesia: Classic and Contemporary
Readings, eds., S. Baron-Cohen, and J. Harrison. Oxford: Blackwells, 1996.
Jacobs,
L., A. Karpik., D. Bozian, and S. Gothgen. "Auditory-visual synaesthesia:
sound induced photisms." Archives of
Neurology 38 (1981) : 211-16.
Karwoski,
T. F., H. S. Odbert and Charles E. Osgood. "Studies in Synesthetic Thinking
II: The Role of Forms in Visual Responses to Music." Journal
of General Psychology 26 (1942).
Kluver,
H. Mescal and Mechanisms of Hallucinations. Chicago: University of
Chicago Press, 1966.
Marks,
L. E. The Unity of the Senses:
Interrelations among the Modalities. New York: Academic Press, 1978.
Marks,
Lawrence E. " Categories of Perceptual Experience: A Psychophysicist
Peruses Synesthetic Metaphors." in Modern
issues in perception. edited by Hans-Georg Geissler. Amsterdam: Elsevier
Science Publishers B.V., 1983. 351-352.
Marks,
Lawrence E. "On Coloured-Hearing Synesthesia" in Synesthesia:
Classic and Contemporary Readings, edited by S. Baron-Cohen, and J.
Harrison. Oxford: Blackwells, 1996.
Martin,
John H. "Coding and Processing of Sensory Information" in Principles
of Neural Science, edited by Eric R Kandel, James H. Schwartz and Thomas M.
Jessel. London: Prentice Hall, 1991.
Myers,
C. S. "Two cases of Synesthesia." British
Journal of Psychology 7 (1915) : 112-17.
Newton,
Isaac. Optiks. New York: Dover
Publications, 1952. (1730)
Peacock,
K. "Synesthetic perception: Alexander Scriabin's color hearing." Music Perception 2 (1985) : 498.
Sabaneev,
L. "The relation between Sound and Colour." Music
and Letters 10 (1926): 266.
Slawson,
Wayne. "Vowel quality and musical timbre as functions of spectrum envelope
and fundamental frequency." Journal
of Acoustical Society of America 43 (1968) : 87-101.
Slawson,
Wayne. Sound Color. University of
California Press, 1985.
Stein,
Barry M. The Merging of the Senses.
Cambridge, Mass.: MIT Press, 1993
a