Dear Hugo and all,
as a cognitive neuroscientist I’d like to add: don’t forget the brain! Jan made some very valid points about how much
we can infer from vocoders from a technical point of view. Complicating things further is the fact that no brain will interpret the sound in the same fashion. How much hearing experience you have had before you get a CI is crucial for what you can extract
from the CI signal. A person whose brain experienced decades of hearing and only a relatively short period of deafness before getting the implant will extract much more from the CI signal than someone whose brain has never learned to decode (audio) speech
and gets an implant late in life like your sister. Speech discrimination may come almost effortlessly for some in the first case, while it is out of reach for almost everybody in the latter case. The CI might still be useful because it informs you about environmental
sounds (your child is crying in the next room, someone is addressing you from behind etc) but understanding speech without lip-reading is no hope one should pose on cochlear implantation if born deaf and not getting the implant early in life.
With music, it will be similar in certain aspects. The interviews of CI users Kathy and Angela made for the CI hackathon
that Alan send around (https://cihackathon.com/docs/CI_interviews) describe very nicely, I think, how they are able to fill in missing information for songs they know from before their hearing loss (and
which they are able to enjoy) and how this does not work for new pieces of music for which they do not have a “pre-CI” memory. On the other hand, music enjoyment has so much to do with your own expectations that in one of our studies we found that those who
have never experienced music bore they got the CI actually tend to enjoy it much more than those who can compare it to “how it used to sound” before their hearing loss and who are disappointed by how different the music sounds with the CI ( Hahne et al., 2020,
doi: 10.1055/s-0040-1711102).
This is just to give you an idea of how diverse the experience of one and the same CI output may be depending on your
individual history and how it has shaped your brain. Of course, there are more factors that shape what you hear with the CI (many related to the individual brain, others linked to the technology itself), but one’s hearing history is a very fundamental one.
I for one would be very interested to hear your CI art project, maybe you could point me/us towards it when the time comes?
That would be great!
All the best
Niki
***********************************
Dr. rer. nat. Niki K. Vavatzanidis (she/her)
Saxonian Cochlear Implant Center Dresden
University Hospital Dresden
Fetscherstr. 74
01307 Dresden
Germany
niki.vavatzanidis(at)ukdd.de
https://www.uniklinikum-dresden.de/scic/research
Von: Jan Schnupp <jan.schnupp@xxxxxxxxxxxxxx>
Gesendet: Dienstag, 9. März 2021 11:22
Betreff: Re: How is the signal of a cochlear implant? [Sound art Project in honor to my deaf sister]
Dear Hugo,
one thing you must appreciate is that, although there are a number of vocoders out there to simulate cochlear implants, aone Alan recommended is perfectly fine, it is nevertheless important to appreciate it
is fundamentally impossible to give a true, veridical impression of the sensation cochlear implants create through acoustic stimulation of the normal cochlea. The main reason for this is that the mechanics of the cochlea links temporal stimulation patterns
to places of stimulation, and CIs don't do anything like that. Many established CI designs do not pay much attention to the precise temporal patterning of stimuus pulses, so CI users lose important cues for the pitch of complex sounds, for binaural scene analysis
and for spatial fearing. What exactly that means cannot be simulated with sound, although "vocoding techniques" give an impression. You may have seen the demo here which I like to use of a Beethoven sonata: http://auditoryneuroscience.com/prosthetics/music
If you listen to the original it is very clearly two instruments playing two distinct melodies. The vocoded version sounds much more like a single stream and the melody is much harder to appreciate, but the rhythm is unimpaired.
That demo I made with a bit of simple Matlab code, a bank of bandpass followed by envelope extraction, and then I use the envelope to modulate narrow band noise. Happy to share the code but it is pretty trivial.
Good luck with your public engagement artwork, and all the best to your sister.
Jan
---------------------------------------
Prof Jan Schnupp
City University of Hong Kong
Dept. of Neuroscience
On Tue, 9 Mar 2021 at 13:15, Alan Kan <alan.kan@xxxxxxxxx> wrote:
Hi Hugo,
Check out https://cihackathon.com/docs/presentations. It’s a hackathon that just finished but they provide Python code for a vocoder that follows the Advanced Bionics cochlear implant signal processing. All you would need to do is just run your sound files through it.
Cheers
Alan
---
Alan Kan, PhD
Research FellowSchool of Engineering | Macquarie University
Level 1, 50 Waterloo Road, Macquarie Park, NSW 2113, AustraliaAustralian Hearing Hub
Level 1, 16 University Avenue, Macquarie University, NSW 2109, Australia
T: +61 (2) 9850 2247
E: alan.kan@xxxxxxxxx
W: https://researchers.mq.edu.au/en/persons/alan-kan | mq.edu.auL: www.linkedin.com/in/alan-kan
CRICOS Provider 00002J. ABN: 90 952 801 237.
This message is intended for the addressee named and may
contain confidential information. If you are not the intended
recipient, please delete the message and notify the sender.
Views expressed in this message are those of the individual
sender and are not necessarily the views of Macquarie
University and its controlled entities.
From: AUDITORY - Research in Auditory Perception <AUDITORY@xxxxxxxxxxxxxxx> On Behalf Of Hugo Solís
Sent: Sunday, 7 March 2021 11:55 PM
To: AUDITORY@xxxxxxxxxxxxxxx
Subject: [AUDITORY] How is the signal of a cochlear implant? [Sound art Project in honor to my deaf sister]
Hi everybody,
My names is Hugo and I am a sound artist with some background in
computer science. I have a sister that was born fully deaf and she got a
cochlear implant when she was 40 years old. She is now 48. The cochlear
changed too little in my sister's live and she doesn't describe music as
a pleasant experience.
I want to create a piece of art where hearing people could hear the real
signal that the cochlear implant sends to the brain. I know that the
signal is processed and that pulses are generated on each one of the
electrodes. However I do not nothing about the details of the
transformation.
I am capable of write code in Python (ussing the Essentia Library
(https://essentia.upf.edu/) in order the emulate the transformation to a
signal but I don´t know what is the typical process. I could also write
the code in SuperCollider (https://supercollider.github.io/) but
although it has tons of unit generators it does not have as many
extractors of audio descriptors and common phsyacoustic process as Python.
I am not an audiologist and I have a lack of the signal processing
transformation that happens in a cochlear implant. I do know a lot about
digital signal processing though.
So I need some basics:
1. Code or libraries in any programming code but ideally in Python that
does the emulation. I could write the process but I imagine that many
people has already done this and that there is opensource code already
written.
2. Basic reference about the process that happens in the cochlear device
that could help me to either write the code or tuning the opensource
code in order to make my piece. The work will be shown in a exhibition
and I am running out of time. So any help would be more than appreciate it.
I will be forever thankfull with your support.
Warm regards
Hugo Solís
