Opera 'Sensorium Ex' Uses AI to Give Non-Verbal Individuals a Voice

"Sensorium Ex": Art, Advocacy, and AI

• 🎭 The opera "Sensorium Ex" explores the intersection of art, disability, and technology, aiming to give voice and opportunities to individuals with disabilities.
• 💡 The work challenges notions of who has a voice and who gets heard, creating more welcoming spaces for diverse forms of expression.

Grounding the Narrative in Lived Experience

• ✍️ Librettist Brenda Shaughnessy grounded the opera in her personal experience as the mother of a nonverbal, nonambulatory son.
• 🌟 The goal was to ensure her son, and others like him, could be part of the narrative, becoming the hero of a story and being heard.
• 🎤 Kitsune, the disabled son in the opera, has a voice that is central to the score, music, technology, and plot.

AI-Powered Voice Creation

• 🤖 The opera tackles the challenge of writing for individuals thought to have no voice by using AI to amplify existing nonverbal expressions.
• 🗣️ Actor Jakob Jordan, who is nonverbal, uses AI to translate his thoughts and expressions into a synthesized voice, overcoming the limitations of his body's response to his brain's signals.
• 💡 NYU's Ability Project developed "Sensorium A.I.," an open-source technology that infers an individual's physical body from recordings of their sounds (whistles, singing, speech-like utterances) to create a unique voice.
• 🎨 This technology allows for expressive control of pitch and pace, enabling individuals like Jakob to communicate purposefully with a voice that sounds like them.

Impact and Future of the Technology

• 🚀 The project demonstrates the potential benefits of AI, using it to create a narrative about its potential evils while simultaneously leveraging its power for good.
• 🌍 The open-source nature of the technology aims to put it in the hands of everyone who needs it, fostering further innovation in speech research.
• ✨ Composer Paola Prestini hopes "Sensorium Ex" will inspire more productions, offering a blueprint for the future and representing the complexity of human life on stage, serving as a pathway to a better way.
• ❤️ For individuals like Jakob Jordan, performing on stage with his AI-generated voice was described as the most fulfilling experience of his life.