The Programmer News Hubb
Advertisement Banner
  • Home
  • Technical Insights
  • Tricks & Tutorial
  • Contact
No Result
View All Result
  • Home
  • Technical Insights
  • Tricks & Tutorial
  • Contact
No Result
View All Result
Gourmet News Hubb
No Result
View All Result
Home Technical Insights

How Google improves video voice dubbing through deep learning

admin by admin
December 26, 2022
in Technical Insights


With less than 20% of the world’s population speaking English as their first or second language, Google is ramping up the efficiency of video voice dubbing with technologies for cross-lingual voice transfer and lip reanimation using deep learning and TensorFlow. 

The first technology keeps the voice similar to that of the original speaker and the second adjusts the speaker’s lip movements in the video to better match the audio generated in the target language. 

Google performs cross-lingual voice transfer by creating synthetic voices in the target language that best fits the speaker’s original voice. This technology was made possible by first pre-training a multilingual text-to-speech (TTS) model based on the cross-language voice transfer approach. Then Google, fine-tuned the model parameters by retraining with a fixed mixing ratio of the adaptation data and original multilingual data 

For lip reanimation, Google trained a multistage model that maps audio to lip shapes and the appearance of the speaker. Then they used the original videos of the speaker for training, isolated the frequency and represented the faces in a space that decouples 3D geometry, head pose, texture and lighting.Then a GAN-based approach is used to blend these synthesized textures with the original video before later refinement using a  super-resolution network.

Google’s new system for video dubbing through deep learning uses a combination of natural language processing, speech recognition, and audio-video analysis to create more natural-sounding and accurate voice dubs. 

“We strongly believe that dubbing is a creative process. With these techniques, we strive to make a broader range of content available and enjoyable in a variety of other languages,” Google wrote in a blog post.



Source link

Previous Post

SD Times Q&A: Five things to look for in 2023

Next Post

Google announces innovations in privacy-enhancing technologies

Next Post

Google announces innovations in privacy-enhancing technologies

Recommended

The 10 Best Free Lower Thirds Templates for After Effects

4 months ago

Minimum Viable Architecture – Apiumhub

4 months ago

A Pure CSS Gallery Focus Effect with :not | CSS-Tricks

4 months ago

WordPress Developer Blog | CSS-Tricks

2 months ago

Atlassian to ‘Unleash’ Agile, DevOps best practices at new event

3 weeks ago

OutSystems expands low-code platform with cloud-native development offering

2 months ago

© 2022 The Programmer News Hubb All rights reserved.

Use of these names, logos, and brands does not imply endorsement unless specified. By using this site, you agree to the Privacy Policy and Terms & Conditions.

Navigate Site

  • Home
  • Technical Insights
  • Tricks & Tutorial
  • Contact

Newsletter Sign Up.

No Result
View All Result
  • Home
  • Technical Insights
  • Tricks & Tutorial
  • Contact

© 2022 The Programmer News Hubb All rights reserved.