We propose a simple algorithm for automatic transfer of facial expressions, from videos to a 3D character, as well as between distinct 3D characters through their rendered animations. Our method begins by learning a common, semantically-consistent latent representation for the different input image domains using an unsupervised image-to-image translation model. It subsequently learns, in a supervised manner, a linear mapping from the character images' encoded representation to the animation coefficients. At inference time, given the source domain (i.e., actor footage), it regresses the corresponding animation coefficients for the target character. Expressions are automatically remapped between the source and target identities despite differences in physiognomy. We show how our technique can be used in the context of markerless motion capture with controlled lighting conditions, for one actor and for multiple actors. Additionally, we show how it can be used to automatically transfer facial animation between distinct characters without consistent mesh parameterization and without engineered geometric priors. We compare our method with standard approaches used in production and with recent state-of-the-art models on single camera face tracking.
We present a unified pipeline for high-resolution facial capture that replaces the initial traditional seated capture with a head-mounted camera setup. At its core, our approach relies on improving roughly personalized blendshapes by fitting handle vertices, in a Laplacian framework, to depth and image data. Thus, refining the geometry. This pipeline has been used in production to generate high quality animation to train our proprietary marker-based solution, leading to large time and cost savings.
The last few years have seen a decisive move of the movie making industry towards rendering using physically based methods, mostly implemented in terms of path tracing. While path tracing reached most VFX houses and animation studios at a time when a physically based approach to rendering and especially material modelling was already firmly established, the new tools brought with them a whole new balance, and many new workflows have evolved to find a new equilibrium. Letting go of instincts based on hard-learned lessons from a previous time has been challenging for some, and many different takes on a practical deployment of the new technologies have emerged. While the language and toolkit available to the technical directors keep closing the gap between lighting in the real world and the light transport simulations ran in software, an understanding of the limitations of the simulation models and a good intuition of the trade-offs and approximations at play are of fundamental importance to make efficient use of the available resources. In this course, the novel workflows emerged during the transitions at a number of large facilities are presented to a wide audience including technical directors, artists, and researchers. This is the second part of a two part course. While the first part focuses on background and implementation, the second one focuses on material acquisition and modeling, GPU rendering, and pipeline evolution.
We have built a real-time (60 fps) photo-realistic facial motion capture system which uses a single camera, proprietary deep learning software, and Unreal Engine 4 to create photo-real digital humans and creatures. Our system uses thousands of frames of realistic captured 3D facial performance of an actor (generated from automated offline systems) instead of a traditional FACS-based facial rig to produce an accurate model of how an actor's face moves. This 3D data is used to create a real-time machine learning model which uses a single image to accurately describe the exact facial pose in under 17 milliseconds. The motion of the face is highly realistic and includes region based blood flow, wrinkle activation, and pore structure changes, driven by geometry deformations in real-time. The facial performance of the actor can be transferred to a character with extremely high fidelity, and switching the machine learning models is instantaneous. We consider this a significant advancement over other real-time avatar projects in development. Building on top of our real-time facial animation technology, we seek to make interaction with our avatars more immersive and emotive. We built an AR system for the actor who is driving the human / character to see and interact with people in VR or others viewing in AR. With this technique, the character you are interacting with in VR can make correct eye contact, walk around you, and interact as if you were together all while still achieving the highest quality capture. This process allows for a much more tangible VR / AR experience than any other system. Another goal of ours is to achieve photo-real avatar telepresence with minimal latency. We have been able to successfully live-drive our digital humans from our office in Los Angeles to our office in Vancouver.
Objective: To describe the implementation and patient experience with outpatient subspecialty teleneurology visits at an academic institution over four years. Background: Teleneurology has been well described for acute stroke but its outpatient use has been limited. In particular, there are few studies assessing personal use video-conferencing platforms in neurology. At home virtual visits have the potential to improve patient access to neurological subspecialty care which otherwise often requires travel. Travel may be difficult for patients due to expense and disability. There is a need for improved access to quality neurological care. Design/Methods: All adult patients completing virtual neurologic follow-up visits from a single tertiary care institution over a four year period were included. Virtual visits were conducted by personal smartphone or computer via video-conferencing with a provider. Patients were asked to rate their overall experience with the visit and provider (5 point scale). Travel distance from the institution was calculated using patient’s home address. Results: 3958 patients completed 5646 virtual visits during the four year period. Virtual visits were completed in all outpatient subspecialities with breakdown as follows: Headache 21.4%, Epilepsy 19.1%, Spine 15.7%, Movement 10.8%, Psychiatry 6.4%, Autonomic 5.7%, Cerebrovascular 4.8%, Cognitive 3.6%, and the remainder being 270 miles). Across the 5646 visits, 1,418,688 miles of travel was prevented. Number of virtual visits increased from 30 visits year 1 to 4514 year 4. 25.1% of patients completed multiple virtual visits. On average, patients rated their overall experience with the virtual visit 4.7/5 and rated their provider 4.9/5. Conclusions: This study demonstrates the feasibility of implementing subspecialty teleneurology services and high patient ratings for these services. Future efforts should focus on demonstrating value for patients and the health care system. Disclosure: Dr. Ross has nothing to disclose. Dr. Bena has nothing to disclose. Dr. Bermel has received personal compensation for consulting, serving on a scientific advisory board, speaking, or other activities with Biogen, Genzyme, Genentech, Novartis, Roche. Dr. Bermel has received research support from Biogen, Genentech, Roche. Dr. McCarter has nothing to disclose. Dr. Ahmed has nothing to disclose. Dr. Goforth has nothing to disclose. Dr. Cherian has nothing to disclose. Dr. Kriegler has received personal compensation for consulting, serving on a scientific advisory board, speaking, or other activities with DynaMed Plus. Dr. Kriegler has received research support from Allergan and Amgen. Dr. Estemalik has nothing to disclose. Dr. Stanton has nothing to disclose. Dr. Rasmussen has nothing to disclose. Dr. Fernandez has received personal compensation for consulting, serving on a scientific advisory board, speaking, or other activities with Prime Education Inc. International Parkinson and Movement Disorders Society, Carling Communications, Medscape, Vindico, AbbVie, Biogen, Blackthorn, Inventiv, Kyowa Hakko Kirin, Medscape, Voyager, Sunovion, and Pfizer Pharmaceuticals. Dr. Fernandez has received royalty, license fees, or contractual rights payments from Demos Publishing, Cambridge University Press. Dr. Fernandez has received research support from Abbvie, Biotie/Acorda Therapeutics, Michael J. Fox Foundation, Movement Disorders Society, NIH/NINDS, Parkinson Study Group, Rhythm, Sunovion. Dr. Najm has received personal compensation for consulting, serving on a scientific advisory board, speaking, or other activities with Eisai. Dr. McGinley has received personal compensation for consulting, serving on a scientific advisory board, speaking, or other activities with Sanofi Genzyme and Genentech.
