AI-Driven Immersive Experience
As humans, our relationship to animals is - to put it mildly - complicated. Yet the ability to empathize with others is exactly what makes us human. To demonstrate this, we teamed up with PETA, the world's foremost animal rights organization, to create a groundbreaking experience that aims to help us empathize with animal rights through an immersive dialogue. By provoking deep emotional reflection, this highly personalized and ever-changing AI-driven experience is designed to meaningfully educate individuals who may be unaware of the impact their daily actions have on animal welfare.
The situation: virtual advocacy
Raising awareness about animal rights and the conditions animals endure in various industries presents significant challenges. Traditional advocacy methods often fall short in creating deep emotional connections or lasting impacts. Identifying this gap, we aimed to develop a novel approach that not only educates but also emotionally engages individuals in understanding and supporting animal rights.
Conversational AI breaking down barriers
We aimed to create a one-of-a-kind simulation in VR that fosters personal connection, and encourages deep reflection. By venturing into the realm of AI-driven non-player characters (NPCs) to enhance personalization, the need to balance a desired narrative flow with the potential unpredictability of responses from a large language model (LLM) was a critical to our development process requiring extensive testing and fine-tuning.
The approach: Innovation first
As an innovation agency, we recognize the transformative potential of immersive technologies for businesses and brands. Early discussions with PETA revealed a significant opportunity to push the envelope by leveraging ChatGPT-4o technology inside of Unity. "When They Came for Us" is our first commercial project to fully integrate an AI 'agent' showcasing the vast possibilities of conversational AI through voice control interactions.
A new benchmark in Virtual Reality
With previous experience developing a dialogue-based VR activation for PETA (2018), which utilized a live mocap actor to converse with guests through a virtual rabbit, our team has conceptualized and designed the next iteration that employs the latest Artificial Intelligence (AI) technology to create a more personalized, immersive and thought-provoking experience.
The setting: thrilling sci-fi simulation
We envisioned a confrontational scenario where users are abducted by a superior alien species and need to debate their basic human rights, mirroring the power dynamics animals face in human society. Under the surface is a predefined points scale to which ChatGPT ‘awards’ the users statements to determine their outcome: will they be free to leave or deactivated?
AI model training through ChatGPT-4o
To put the user dialogue in the literal spotlight, players navigate the experience hands-free as they must use the microphone input to reason with their abductor. While environmental descriptions and details about the alien’s appearance are drawn from a manually created vector database, the rest of the context is generated by ChatGPT-4o. The AI is guided by our prompt engineer’s briefing which incorporates PETA's values and key themes in order to generate intelligent and unique responses that stay as much on theme as possible. For optimization, we used 250 million input and output tokens to ensure precise and consistent responses.
Global accessibility through multilingual support
To meet our technical and performance requirements, we tested Llama language models as well as various versions of ChatGPT, starting with GPT-3.5. Experimenting with our own AI model proved too labor-intensive, especially considering the PeTA-specific data. Ultimately, Llama models were inferior to the OpenAI models, leading us to choose ChatGPT-4o as it became available in the final stages of development. For global accessibility, we incorporated multilingual support in German and English using ElevenLabs services.
Real-time 3D workflow with OpenXR
Developed in Unity and integrated via the OpenXR API, we achieved a high-quality and seamless integration with the Meta Quest 2, 3, and Pro VR headsets. This approach provided more flexibility and allowed easier porting to other systems while maintaining close support for the Meta Quest platform using the Meta plugin. These tools provided the necessary framework to bring our vision to life, delivering an engaging and immersive user experience.
Challenging perspectives for a wider audience
This unassuming and intense seven-minute experience is made accessible through publishing on the Meta Store for those with VR devices to discover at home. The very careful presentation of a thriller game is designed to ‘catch’ unwilling but curious participants and provide an educational experience for the greater good of animals.
The voice of the voiceless
Our collaboration with PETA showcases the potential of immersive technologies in advocacy. "When They Came for Us" is a pioneering example of how emerging AI technologies can be leveraged in VR to create impactful and empathetic experiences that drive social change.
