The Audio-Visual Processing of Horror

May 4, 2020   /  

Student: Timothy Perales
Major: Cognitive Behavioral Neuroscience
Advisors: John Neuhoff, Evan Wilhelms

Timothy PeralesThe effects of fearful visual stimuli type and linearity or nonlinearity of audio stimuli was examined using a reaction time test. Participants were randomly assigned to conditions that contained either horror-film-type images or snake images as well as either a linear or nonlinear audio stimulus, and were tasked with determining whether there was a fearful (horror or snake) image present among multiple five image arrays. Descriptively, participants in non-horror visual conditions had faster average reaction times and participants in nonlinear audio conditions also had faster reaction times, however these differences were not found to be significant. Preliminary results suggest that ecological validity and arousal factors may contribute to differences in reaction time, but more research is necessary to confirm.

Loader Loading...
EAD Logo Taking too long?

Reload Reload document
| Open Open in new tab

Download [172.48 KB]

Timothy will be online to field comments on May 8:
2-4pm EDT (PST 11am-1pm, Africa/Europe: evening)

18 thoughts on “The Audio-Visual Processing of Horror”

  1. What an interesting research question! What do you think the practical implications might be? What suggestions might you make to a sound engineer for a horror film?

    1. Applications for horror sound design basically boil down to adding more non-linearity! The more of those sounds you’re introducing regardless of where they come from, the more intense the response will be. At least, that would be my guess. I very recently saw a very interesting and intense horror film called Climax (2018) by Gaspar Noé, and that film demonstrated that point about how effective adding more non-linearity can be. There is literally near constant screaming at a few points in the film that put your guard up and don’t let you drop it for several minutes at a time, which is a stressful and frightening experience and for lack of a better term – it’s like a psychological endurance run. A lot of times in horror films, you get a quick scary scene that will typically end in a kill, and then the film will typically go back to not being stressful until the villain strikes again. Climax is unrelenting in its visual and narrative horror, but also in its acoustic properties as well. The constant screaming in combination with an almost non-stop electronic dance soundtrack not only test the limits of how long we can stress our amygdala and stress systems, but (at least to me) very clearly show how long-term exposure to non-linear sounds can affect your psychological state.

  2. Hi Tim, thanks for sharing your research! I know you state that your findings are not significant in the way you might have expected, but I was hoping you could say more about the interaction effect. The snake findings make sense to me – if the sound is nonlinear, a participant might be a little more in tune with a bad thing happening, and therefore the snake is less of a pop-out effect. Whereas, when the sound is linear, the snake seems more like a pop-out effect. If you think otherwise, I’d love to hear it — but my actual question is, why do you think it’s reversing for the horror images?

    1. Hi Professor Stavnezer! That’s an interesting point about non-linearity potentially setting up a desensitization response. Going into the project, I had initially thought that it would have been the opposite in the sense that the non-linearity would cause sensitization and therefore faster reaction times, but the data showed somewhat the opposite. I think that finding could be due to attentional processing – maybe the non-linearity takes up more of the brain’s processing capacity and causes a slower reaction time as a result, whereas linearity is less novel and may be easier to not attend to. Maybe the same framework could be used to explain that pattern reversal for horror images. Linear sound could allow for more processing, which for the horror images could mean more visual searching and analysis for when something is/isn’t horror. Anecdotally, I had one participant comment that she spent longer than intended on one array because she couldn’t determine if one of the nature images was a nature image (it was a large mushroom growing in a pot) or a severed head (her words, not mine.) Maybe freeing up brain RAM for analyzing those types of details for horror means spending more time on that cognitive process, and conversely having less computing power as a result of non-linearity could cause more “gut” reactions than analytical ones.

  3. Hi Timothy,
    My question dovetails with Dr. Stavnezer’s question. I was wondering about the unpredicted reversed (though not significant) effect of liner and nonlinear sounds on the horror movie image identification. Do you think it’s possible that the horror SOUNDS are so strongly ASSOCIATED with the horror IMAGES that the sounds in that condition are actually PRIMING identification of the visual stimulus? That wouldn’t happen for the snake stimulus, since horror sounds are not as often encountered with seeing a snake.
    Your presentation was interesting and clearly presented. I really liked your red-ink annotations — very helpful.
    You did make me reflect on how much in in what variety nonlinear sounds are used in movies to show suspense and tension and danger and distress. That happens in classical music, too — and of course in some computer games….
    What do you think about nonlinearity in rock and indie music these days???
    Well, thanks for a thought-provoking afternoon. Time to say…
    PS — Please stay in touch!!! Send music once in a while. Thanks

    1. Hi Professor Thompson!
      I think that explanation also makes perfect sense. Priming and congruency are definitely factors in identification and discrimination tasks like this, and your point about horror films being congruent with non-linear audio could explain the reaction time difference there. If the audio and visual stimuli are being processed as related and matching, it would follow that the reaction time for those cases would be faster. That congruence would also rely on the association between horror and non-linearity you mentioned, which makes sense given that horror films are culturally and aesthetically paired with non-linearity (both from screams and instruments.) Like you said, that would also explain why the inverse pattern happened for snakes – because we don’t associate snakes with horror sounds usually.

      I’m glad the annotations helped! I know these symposium presentations are professional, but I had hoped that adding a personal and informal touch to the presentation would help everyone’s understanding (and it sounds like it paid off!)

      As for music, ironically I really haven’t liked anything super non-linear that I’ve stumbled on or been recommended. Noise and ambient as genres will often jumble sounds together to create often chaotic and non-linear sounds, but I find the lack of traditional song structure or even melody leaves me wanting and unsatisfied. A friend recommended this song (, and I told him I could see its artistic merit but it definitely wasn’t for me. Rock and indie music (in my opinion) traditionally call for more structure, but if you’re looking for non-linearities then that song and ambient as a larger genre is the place to go.

      As long as there’s interesting music being made, I’ll be more than happy to send some!

      Thank you as always, and may the force be with you!

  4. Wow, thanks for the detailed answer. I’ll have to check out that movie you mentioned – if I can handle all the non-linear sound.

    1. Of course! A lot of what makes Climax (2018) of merit is also what makes it borderline unwatchable, including the sounds! Practical gore effects (a personal hallmark of good horror), mind-bending camerawork, and a nonstop narrative all make it an amazing piece of cinematic art – but also make it a very challenging watch. Good luck diving in with a new non-linear angle to appreciate it from!

    1. Thank you! Horror Nights will be fun – maybe we can appreciate the screaming there in a different lens this year 🙂

        1. Actually more so the amygdala and adrenals, but basal ganglia could definitely come into play if we need to start running from all the chainsaw-wielding maniacs!

  5. Hey Tim, great job! I love the comparison of a woman screaming to the flute…very cool! If you could, would you conduct this experiment again with a larger amount of participants?

  6. Hi Timothy,

    What a pitch perfect explanation — just beautiful. And what a wonderful way to create scholarship of maximal personal interest! I won’t be surprised if someday you apply everything you have learned about the psychoacoustics of horror to scaring the daylights out of us on film…

    I’m an evolutionarily-minded clinical research psychologist, and a big admirer of Prof. Neuhoff’s work. I rely on evolutionarily-informed models in my work on theory and treatment of anxiety and depression.

    One question has to do with the choice of nonlinear auditory stimulus: Birds might well constitute a ‘mildly prepared’ stimulus for mammals. It’s not intuitive to think that they would, because you have to reach quite far back phylogenetically to identify a context in which they represented a threat to our much tinier selves, or reach way back ontogenetically, to toddlerhood… but it’s ‘back there’. In fact, bird phobias are quite common, among the top five or ten animal phobias (and animals are a Very Big Deal phobia-wise); they’re often severe when they occur, and, like other prepared phobias, they tend to subside as we outgrow the biggest raptors. All in all, it’s probably not an accident that Hitchcock’s flock in The Birds works so especially well as a horror stimulus. Does this matter? Would you expect the same pattern of results whether or not there was some ‘prepared fear variance’ in responses to the nonlinear auditory stimulus? Forgive me if you addressed this in your talk and I blew by it.

    Another question is just great curiosity on my part about whether, as a scientist and aficionado of horror, you’ve thought about our society’s extreme resistance to wearing face masks during the pandemic. I wonder whether the study of horror, panic, and phobic stimuli has anything to offer: What is it about masks that is so especially frightening, and does that aversion/fear contribute to our resistance to adopting them now? (It’s not self-evident to me that a mask ‘should’, evolutionarily speaking, make an aggressor more frightening than, say, a steely-eyed or glowering unmasked face… and I’m fascinated now to notice that I’ve never even considered the mask riddle until your talk! )

    Best of everything to you, Timothy, and thank you for so much good, scary food for thought!


  7. Great job, Timothy! I had an awesome time working with you in Neuroscience in Society this semester. I wish you well and congratulations!

Comments are closed.