January 1, 2018 1:29 pm
(CNBC) Facebook artificial intelligence researchers have come up with a novel way to make cartoonish avatars look like they’re really playing musical instruments. The work could lead to fascinating augmented reality or virtual reality experiences in the future.
Facebook research scientists Eli Shlizerman and Ira Kemelmacher-Shlizerman and collaborators Lucio Dery and Hayden Schoen talk trained AI systems using YouTube videos of piano and violin recitals. They then used the trained systems to make avatars move their hands and fingers with fake instruments based solely on audio recordings, with the help of Apple’s ARKit AR software for developers.