Improving Video Voice Dubbing Through Deep Learning

improving-video-voice-dubbing-through-deep-learning

Posted by Paul McCartney, Software Engineer, Vivek Kwatra, Research Scientist, Yu Zhang, Research Scientist, Brian Colonna, Software Engineer, and Mor Miller, Software Engineer

People increasingly look to video as their preferred way to be better informed, to explore their interests, and to be entertained. And yet a video’s spoken language is often a barrier to understanding. For example, a high percentage of YouTube videos are in English but less than 20% of the world’s population speaks English as their first or second language. Voice dubbing is increasingly being used to transform video into other languages, by translating and replacing a video’s original spoken dialogue. This is effective in eliminating the language barrier and is also a better accessibility option with regard to both literacy and sightedness in comparison to subtitles.

In today’s post, we share our research for increasing voice dubbing quality using deep learning, providing a viewing experience closer to that of a video produced directly for the target language. Specifically, we describe our work with technologies for cross-lingual voice transfer and lip reanimation, which keeps the voice similar to the original speaker and adjusts the speaker’s lip movements in the video to better match the audio generated in the target language. Both capabilities were developed using TensorFlow, which provides a scalable platform for multimodal machine learning. We share videos produced using our research prototype, which are demonstrably less distracting and – hopefully – more enjoyable for viewers.

Cross-Lingual Voice Transfer

Voice casting is the process of finding a suitable voice to represent each person on screen. Maintaining the audience’s suspension of disbelief by having believable voices for speakers is important in producing a quality dub that supports rather than distracts from the video. We achieve this through cross-lingual voice transfer, where we create synthetic voices in the target language that sound like the original speaker voices. For example, the video below uses an English dubbed voice that was created from the speaker’s original Spanish voice.

Original “Coding TensorFlow” video clip in Spanish.

The “Coding TensorFlow” video clip dubbed from Spanish to English, using cross-lingual voice transfer and lip reanimation.

Inspired by few-shot learning, we first pre-trained a multilingual TTS model based on our cross-language voice transfer approach. This approach uses an attention-based sequence-to-sequence model to generate a series of log-mel spectrogram frames from a multilingual input text sequence with a variational autoencoder-style residual encoder. Subsequently, we fine-tune the model parameters by retraining the decoder and attention modules with a fixed mixing ratio of the adaptation data and original multilingual data as illustrated in Figure 1.

fine tuning voice imitation architecture
Figure 1: Voice transfer architecture

Note that voice transfer and lip reanimation is only done when the content owner and speakers give consent for these techniques on their content.

Lip Reanimation

With conventionally dubbed videos, you hear the translated / dubbed voices while seeing the original speakers speaking the original dialogue in the source language. The lip movements that you see in the video generally do not match the newly dubbed words that you hear, making the combined audio/video look unnatural. This can distract viewers from engaging fully with the content. In fact, people often even intentionally look away from the speaker’s mouth while watching dubbed videos as a means to avoid seeing this discrepancy.

To help with audience engagement, producers of higher quality dubbed videos may put more effort into carefully tailoring the dialogue and voice performance to partially match the new speech with the existing lip motion in video. But this is extremely time consuming and expensive, making it cost prohibitive for many content producers. Furthermore, it requires changes that may slightly degrade the voice performance and translation accuracy.

To provide the same lip synchronization benefit, but without these problems, we developed a lip reanimation architecture for correcting the video to match the dubbed voice. That is, we adjust speaker lip movements in the video to make the lips move in alignment with the new dubbed dialogue. This makes it appear as though the video was shot with people originally speaking the translated / dubbed dialogue. This approach can be applied when permitted by the content owner and speakers.

For example, the following clip shows a video that was dubbed in the conventional way (without lip reanimation):

“Machine Learning Foundations” video clip dubbed from English to Spanish, with voice transfer, but without lip reanimation

Notice how the speaker’s mouth movements don’t seem to move naturally with the voice. The video below shows the same video with lip reanimation, resulting in lip motion that appears more natural with the translated / dubbed dialogue:

The dubbed “Machine Learning Foundations” video clip, with both voice transfer and lip reanimation

For lip reanimation, we train a personalized multistage model that learns to map audio to lip shapes and facial appearance of the speaker, as shown in Figure 2. Using original videos of the speaker for training, we isolate and represent the faces in a normalized space that decouples 3D geometry, head pose, texture, and lighting, as described in this paper. Taking this approach allows our first stage to focus on synthesizing lip-synced 3D geometry and texture compatible with the dubbed audio, without worrying about pose and lighting. Our second stage employs a conditional GAN-based approach to blend these synthesized textures with the original video to generate faces with consistent pose and lighting. This stage is trained adversarially using multiple discriminators to simultaneously preserve visual quality, temporal smoothness and lip-sync consistency. Finally, we refine the output using a custom super-resolution network to generate a photorealistic lip-reanimated video. The comparison videos shown above can also be viewed here.


Lip-Reanimation Pipeline showing inference blocks in blue, training blocks in red.
Figure 2: Lip-Reanimation Pipeline: inference blocks in blue, training blocks in red.

Aligning with our AI Principles

The techniques described here fall into the broader category of synthetic media generation, which has rightfully attracted scrutiny due to its potential for abuse. Photorealistically manipulating videos could be misused to produce fake or misleading information that can create downstream societal harms, and researchers should be aware of these risks. Our use case of video dubbing, however, highlights one potential socially beneficial outcome of these technologies. Our new research in voice dubbing could help make educational lectures, video-blogs, public discourse, and other formats more widely accessible across a global audience. This is also only applied when consent has been given by the content owners and speakers.

During our research, we followed our guiding AI Principles for developing and deploying this technology in a responsible manner. First, we work with the creators to ensure that any dubbed content is produced with their consent, and any generated media is identifiable as such. Second, we are actively working on tools and techniques for attributing ownership of original and modified content using provenance and digital watermarking techniques. Finally, our central goal is fidelity to the source-language video. The techniques discussed herein serve that purpose only — namely, to amplify the potential social benefit to the user, while preserving the content’s original nature, style and creator intent. We are continuing to determine how best to uphold and implement data privacy standards and safeguards before broader deployment of our research.

The Opportunity Ahead

We strongly believe that dubbing is a creative process. With these techniques, we strive to make a broader range of content available and enjoyable in a variety of other languages.

We hope that our research inspires the development of new tools that democratize content in a responsible way. To demonstrate its potential, today we are releasing dubbed content for two online educational series, AI for Anyone and Machine Learning Foundations with Tensorflow on the Google Developers LATAM channel.

We have been actively working on expanding our scope to more languages and larger demographics of speakers — we have previously detailed this work, along with a broader discussion, in our research papers on voice transfer and lip reanimation.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
product-marketing-core:-two-day-in-person-workshop-|-amsterdam-|-january-25-–-26,-2023

Product Marketing Core: Two day in-person workshop | Amsterdam | January 25 – 26, 2023

Next Post
directors-of-content-need-these-4-elements-in-their-content-marketing-team-today

Directors of Content Need These 4 Elements in Their Content Marketing Team Today

Related Posts

Artemis II astronauts speak publicly for first time since successful moon mission

事件导语 近期,阿尔忒弥斯二号(Artemis II)任务的宇航员首次公开露面,分享了他们在成功完成月球任务后的经验和感受。这一事件标志着美国国家航空航天局(NASA)阿尔忒弥斯计划的一个重要里程碑,引发了广泛的关注和讨论。阿尔忒弥斯计划旨在于2025年之前将人类送回月球,并为未来的人类登月和火星探索奠定基础。该任务的成功不仅代表了美国在太空探索领域的重大突破,也对全球太空竞争和地缘政治格局产生了深远影响。 背景深度解析 阿尔忒弥斯计划是美国国家航空航天局为实现人类返回月球和进一步探索火星而启动的综合性计划。该计划的目标包括在2025年之前将第一位女性和下一位男性送到月球南极,之后每年都会有一次类似的任务。阿尔忒弥斯二号任务是该计划中的一个关键步骤,它将验证太空船和航天器的性能,为未来的载人任务提供参考。阿尔忒弥斯计划还涉及与私营公司的合作,例如SpaceX和Blue Origin,共同开发月球着陆器和其他必要的技术。通过这样的合作,美国希望能够加快月球探索的步伐,并在太空技术领域保持领先地位。 阿尔忒弥斯二号任务的成功不仅在于其技术上的突破,也体现了美国在太空探索领域的战略意图。自20世纪60年代的阿波罗计划以来,美国就一直是太空探索的先驱。然而,近年来,其他国家如中国和俄罗斯也开始加大对太空探索的投入,挑战了美国在这一领域的主导地位。因此,阿尔忒弥斯计划不仅是美国太空探索的延续,也是其在全球太空竞争中的重要战略举措。 多方观点与博弈 阿尔忒弥斯二号任务的成功引发了各界的广泛讨论。美国国家航空航天局的官员表示,这一任务标志着阿尔忒弥斯计划的重大进展,并为未来的人类登月任务奠定了坚实的基础。与此同时,科学家和工程师们也对任务中使用的技术和数据进行了详细的分析,认为这些成果将对深空探索产生深远影响。 然而,也有一些批评的声音认为,阿尔忒弥斯计划的成本过高,且在当前的经济环境下,是否值得投入如此大量的资源仍存在争议。另外,一些国家也对美国的太空探索计划表示了担忧,认为这可能会引发新的太空军备竞赛,并加剧全球的紧张局势。 在地缘政治方面,阿尔忒弥斯二号任务的成功也引发了其他国家的关注。中国作为一个正在快速崛起的太空大国,已经展开了多项月球探索任务,并宣布了更为雄心勃勃的火星探索计划。俄罗斯同样也在加速其太空探索的步伐,两国都将美国视为主要竞争对手。这种竞争不仅体现在技术和经济领域,也反映在全球政治和战略格局的变化上。 地缘政治影响 阿尔忒弥斯二号任务的成功对全球地缘政治格局产生了深远影响。首先,它标志着美国在太空探索领域的持续领先地位,这将对其他国家的太空战略产生影响。其次,这一事件也加剧了全球太空竞争的紧张局势,各国将更加重视自己的太空探索计划和技术开发。 在区域层面上,阿尔忒弥斯计划也将对亚太地区的安全局势产生影响。随着中国和俄罗斯在太空领域的实力增强,美国将需要加强与其盟友的合作,以维持区域的平衡和稳定。另外,太空探索的军事应用也将成为一个重要的因素,各国将需要考虑如何在太空领域维护自己的安全利益。 经济与市场反应 阿尔忒弥斯二号任务的成功也对经济和市场产生了积极影响。该任务的成功验证了美国在太空技术领域的优势,这将吸引更多的投资和合作。与此同时,太空探索技术的发展也将带来新的商业机会和就业岗位,促进相关产业的发展。 在股市上,相关公司的股票价格也出现了上涨,反映了投资者对太空探索领域的信心和期待。然而,阿尔忒弥斯计划的高昂成本也引发了人们对其经济可行性的担忧,这将需要美国政府和相关机构进行仔细的成本核算和预算规划。 历史相似案例…
Read More