Behavioral signals in the audio and visual modalities available in speech, spoken language and body language offer a window into decoding not just what one is doing but how one is thinking and feeling. At the simplest level, this could entail determining who is talking to whom about what and how using automated audio and video analysis of verbal and nonverbal behavior. Computational modeling can also target more complex, higher level constructs, like the expression and processing of emotions. Behavioral signals combined with physiological signals such as heart rate, respiration and skin conductance offer further possibilities for understanding the dynamic cognitive and affective states in context. Machine intelligence could also help detect, analyze and model deviation from what is deemed typical. This talk will focus on multimodal bio-behavioral sensing, signal processing and machine learning approaches to computationally understand aspects of human affective expressions and experiences. It will draw upon specific case studies to illustrate the multimodal nature of the problem in the context of both vocal encoding of emotions in speech and song, as well as processing of these cues by humans.

A Successive Difference Feature for Detecting Emotional Valence from Speech

Visualizing Carnatic music as projectile motion in a uniform gravitational field

You are sitting on a commuter train. How many passengers are wearing headphones? What are they listening to? What else are they doing? Most importantly, amid the cornucopia of distractions, what exactly are they hearing? Much research in music cognition pits ‘musicians’, variously defined, against non-musicians. Recently, especially since the appearance of reliable measurement instruments for musicality in the general population (e.g., MÃŒllensiefen et al., 2014), there has been growing interest in the space in between. Moreover, the ubiquity of smartphones has greatly enhanced the ability of techniques like gamification or Sloboda’s ‘experience sampling’ to reach this general population outside of a psychology lab. Music information retrieval (MIR) – and signal processing research more generally – can provide the last ingredients to understand what is happening between our commuters’ earbuds: everyday features for studying everyday listening. Since Aucouturier and Bigand’s 2012 manifesto on the poor interpretability of traditional DSP measures, clever dimensionality reduction paired with feature sets like those from the FANTASTIC (MÃŒllensiefen and Frieler, 2006) or CATCHY (Van Balen et al., 2015) toolboxes have sought a middle ground. This talk will present several uses of everyday features from the CATCHY toolbox for studying everyday listening, most notably a discussion of the Hooked on Music series of experiments (Burgoyne et al., 2013) and a recent user study of thumbnailing at a national music service. In conclusion, it will outline some areas where MIR expertise can go further than just recommendation to learn about and engage with listeners during their daily musical activities.