NVIDIA Omniverse Audio2Face

Omniverse Audio2Face

Omniverse Audio2Face is an AI-powered application that generates expressive facial animation from just an audio source.

SYSTEM REQUIREMENTS

Sneak Peek: Audio2Face Early Access Preview

Audio2Face simplifies animation of a 3D character to match any voice-over track, whether you’re animating characters for a game, film, real-time digital assistants, or just for fun. You can use the app for interactive real-time applications or as a traditional facial animation authoring tool. Run the results live or bake them out, it’s up to you.

Omniverse Audio2Face App Early Access is coming soon.

Audio2Face simplifies animation of a 3D character to match any voice-over track.

How It Works

Omniverse Audio2Face App is based on an original NVIDIA Research paper. Audio2Face is preloaded with “Digital Mark”— a 3D character model that can be animated with your audio track, so getting started is simple. Just select your audio and upload it into the app.  The technology feeds the audio input into a pre-trained Deep Neural Network and the output of the network drives the facial animation of your character in real-time.  Users have the option to edit various post-processing parameters to edit the performance of the character. The output of the network then drives the 3D vertices of your character mesh to create the facial animation. The results you see on this page are mostly raw outputs from Audio2Face with little to no post-processing parameters edited.

(Note:  Direct blendshape support will be released at a later date)

Omniverse Audio2Face App

See Audio2Face in Action

Languages

Languages

Audio2Face will be able to process any language easily. We're adding more and more languages periodically. Check out these tests in English, French, Italian and Russian.

Misty the Animated Chat Bot

Misty the Animated Chat Bot

Presented at GTC Spring 2020, Misty is an interactive weather bot that is driven by Audio2Face in run-time. We demonstrated retargeting from a realistic human mesh to a stylized character mesh to be used as an interactive service agent.

Omniverse Machinima

Omniverse Machinima

Unveiled during the GeForce 30 series launch, Audio2Face is seen in the Omniverse Machinima demo. Facial animation is notoriously complex and cost-prohibitive. Audio2Face automates detailed facial animation to democratize the 3D content creation process.   

Upcoming Features - Character Transfer

Human Faces

Human Faces

Audio2Face can use any dialogue-driven audio track and eventually have the ability to retarget to any 3D human face,  whether realistic or stylized. Watch this test as we retarget from Digital Mark to Rain.

Experimental Characters

Creatures and Aliens

This feature allows you to retarget and drive facial animation of fantastical creatures and aliens.  Here we have Digital Mark driving the performance of the Alien.

Get a Deeper Look at Audio2Face

Watch this technical overview presented at GTC 2020.

WATCH ON DEMAND
System Requirements
Element Minimum Specifications
OS Supported Windows 10 64-bit, 1909 and up
CPU Intel I7, AMD Ryzen
CPU Cores 4 or higher
RAM 16 GB or higher
Storage 500 GB or higher
GPU Any RTX GPU
VRAM 6 GB or higher
Min. Video Driver Version 455.28 (Linux), 456.71 (Windows)
Note: Omniverse is built to run on any RTX-powered machine. For ideal performance, we recommend using GeForce RTX 2080, Quadro RTX 5000, or higher. For latest drivers, visit here.

Support

Join the Omniverse Forums
See the support documentation and guides
Watch Omniverse video tutorials and trainings

Let’s Invent
The Future Together

Discover how next-generation collaboration and simulation are bringing the future closer for creators, innovators, and businesses across industries.

Register for NVIDIA GTC 2021 for free sessions >

Enter the Omniverse

Deep dive into the platform through a series of GTC webinars hosted by Omniverse experts in various industries.

LEARN MORE

More Omniverse Apps

Omniverse Apps are purpose-built to enhance specific workflows. See them in action.

nvidia-omniverse-view-64

View

Collaborative design, immersive visualization, and photorealistic rendering for architecture, engineering, and construction.

LEARN MORE
nvidia-omniverse-create-64

Create

Accelerate advanced world building and interactively assemble, simulate, and render scenes in Pixar USD in real-time.

LEARN MORE
nvidia-omniverse-kaolin-icon-64

Kaolin

Simplify and accelerate 3D deep learning research using NVIDIA’s Kaolin PyTorch library with this powerful visualization tool.

LEARN MORE
nvidia-omniverse-machinima-64

Machinima

Remix, recreate, and redefine animated video game storytelling with an AI-powered toolkit for gamers.

LEARN MORE
nvidia-omniverse-isaac-sim-icon-64

Isaac Sim

Import, build, and test robots in a photorealistic and high-fidelity physics 3D environment.

LEARN MORE

PLEASE REGISTER TO DOWNLOAD

Products
Resources
Community
Company Info