These results prove that face-to-face communication is shaped by deeply ingrained prior expectations about how signals should be structured and interpreted. They also raise fascinating questions about the evolutionary, developmental, and cultural origins of these priors.
We found that people tend to combine vocal and bodily signals more strongly when they convey the same communicative intent. Thus, the brain uses prior expectations to bind multisensory signals that carry converging communicative meaning.
We tackled this issue by framing face-to-face communication as a Bayesian Causal Inference problem. Combining psychophysics with computational modelling, we formally tested whether prior expectations arbitrate between the integration and segregation of vocal and bodily signals.
Face-to-face communication is complex: what we say is coupled with bodily signals, which may or may not work in concert to convey meaning. How does the brain determine which multisensory signals belong together and which, instead, must be kept apart?
🚨PREPRINT🚨 Prior expectations guide multisensory integration during face-to-face communication.
🔗
https://biorxiv.org/content/10.1101/2025.02.19.638980v1
Prior expectations guide multisensory integration during face-to-face communication
Face-to-face communication relies on the seamless integration of multisensory signals, including voice, gaze, and head movements, to convey meaning effectively. This poses a fundamental computational challenge: optimally binding signals sharing the same communicative intention (e.g. looking at the addressee while speaking) and segregating unrelated signals (e.g. looking away while coughing), all within the rapid turn-taking dynamics of conversation. Critically, the computational mechanisms underlying this extraordinary feat remain largely unknown. Here, we cast face-to-face communication as a Bayesian Causal Inference problem to formally test whether prior expectations arbitrate between the integration and segregation of vocal and bodily signals. Moreover, we asked whether there is a stronger prior tendency to integrate audiovisual signals that show the same communicative intention, thus carrying a crossmodal pragmatic correspondence. In a spatial localization task, participants watched audiovisual clips of a speaker where the audio (voice) and the video (bodily cues) were sampled either from congruent positions or at increasing spatial disparities. Crucially, we manipulated the pragmatic correspondence of the signals: in a communicative condition, the speaker addressed the participant with their head, gaze and speech; in a non-communicative condition, the speaker kept the head down and produced a meaningless vocalization. We measured audiovisual integration through the ventriloquist effect, which quantifies how much the perceived audio position is misplaced towards the video position. Bayesian Causal Inference outperformed competing models in explaining participants' behaviour, demonstrating that prior expectations guide multisensory integration during face-to-face communication. Remarkably, participants showed a stronger prior tendency to integrate vocal and bodily information when signals conveyed congruent communicative intent, suggesting that pragmatic correspondences enhance multisensory integration. Collectively, our findings provide novel and compelling evidence that face-to-face communication is shaped by deeply ingrained expectations about how multisensory signals should be structured and interpreted.
### Competing Interest Statement
The authors have declared no competing interest.
We are excited about these results supporting a functional similarity between statistical and reinforcement learning. Yet, many questions remain: What types of statistical regularities are extracted, and which metrics govern their extraction? More to come!
Interestingly, we found no forward blocking (details in the paper), but we did see backward blocking. This suggests the presence of retrospective revaluation in statistical learning.
We wanted to see if the same applies to statistical learning, the automatic and incidental extraction of state-state associations without explicit rewards.
But there’s more, as shown by backward blocking: new info can update our understanding of earlier cues, suggesting a retrospective revaluation process in reinforcement learning. Computationally, see e.g. Kalman filter. Beautiful walkthrough here:
https://doi.org/10.1371/journal.pcbi.1004567
A Unifying Probabilistic View of Associative Learning
Author Summary How do we learn about associations between events? The seminal Rescorla-Wagner model provided a simple yet powerful foundation for understanding associative learning. However, much subsequent research has uncovered fundamental limitations of the Rescorla-Wagner model. One response to these limitations has been to rethink associative learning from a normative statistical perspective: How would an ideal agent learn about associations? First, an agent should track its uncertainty using Bayesian principles. Second, an agent should learn about long-term (not just immediate) reward, using reinforcement learning principles. This article brings together these principles into a single framework and shows how they synergistically account for a number of complex learning phenomena.
This is captured by forward (Kamin) blocking: we don’t learn new info about a cause if we already have a strong explanation for an outcome. Once prediction errors are gone, learning stops. Computationally, see Rescorla-Wagner model.
