| Github | https://github.com/mdhk |
| https://twitter.com/mariannedhk | |
| Website | https://mdhk.net |
Want to learn how to analyze the inner workings of speech processing models? 🔍
Check out the programme for our tutorial, taking place at this year's Interspeech conference in Rotterdam: https://interpretingdl.github.io/speech-interpretability-tutorial/
The schedule features presentations and interactive sessions with a great team of co-organizers: Charlotte Pouw, Gaofei Shen, Martijn Bentum, Tom Lentz, @hmohebbi, @wzuidema, @gchrupala (and me!). We look forward to seeing you there 😃
📏 We also compare three analysis methods for decoding phoneme preference from model internals, and find interesting differences between them!
➡️ Read more in the paper: https://arxiv.org/abs/2407.03005
What do deep neural speech models know about phonology? Existing work has examined the encoding of individual linguistic units such as phonemes in these models. Here we investigate interactions between units. Inspired by classic experiments on human speech perception, we study how Wav2Vec2 resolves phonotactic constraints. We synthesize sounds on an acoustic continuum between /l/ and /r/ and embed them in controlled contexts where only /l/, only /r/, or neither occur in English. Like humans, Wav2Vec2 models show a bias towards the phonotactically admissable category in processing such ambiguous sounds. Using simple measures to analyze model internals on the level of individual stimuli, we find that this bias emerges in early layers of the model's Transformer module. This effect is amplified by ASR finetuning but also present in fully self-supervised models. Our approach demonstrates how controlled stimulus designs can help localize specific linguistic knowledge in neural speech models.
One case of such contextual biasing effects comes from phonotactic constraints.
For example in English: TL << TR, SL >> SR
This has been demonstrated in human listeners a while ago! (https://doi.org/10.3758/BF03203046)
✨ Do current neural speech models show human-like linguistic biases in speech perception?
We took inspiration from classic phonetic categorization experiments to explore whether & where sensitivity to phonotactic context emerges in Wav2Vec2 models 🔍
(w/ @wzuidema )
📑
https://arxiv.org/abs/2407.03005
⬇️
What do deep neural speech models know about phonology? Existing work has examined the encoding of individual linguistic units such as phonemes in these models. Here we investigate interactions between units. Inspired by classic experiments on human speech perception, we study how Wav2Vec2 resolves phonotactic constraints. We synthesize sounds on an acoustic continuum between /l/ and /r/ and embed them in controlled contexts where only /l/, only /r/, or neither occur in English. Like humans, Wav2Vec2 models show a bias towards the phonotactically admissable category in processing such ambiguous sounds. Using simple measures to analyze model internals on the level of individual stimuli, we find that this bias emerges in early layers of the model's Transformer module. This effect is amplified by ASR finetuning but also present in fully self-supervised models. Our approach demonstrates how controlled stimulus designs can help localize specific linguistic knowledge in neural speech models.
Call for Abstracts The Young Female* Researchers in Speech Workshop (YFRSW) is a workshop for female* Bachelor’s and Master’s students currently working in speech science and technology. The workshop aims to promote interest in research in our field among women* who have not yet committed to