Giter VIP home page Giter VIP logo

Comments (13)

prajwalkr avatar prajwalkr commented on August 27, 2024 1

Hello, thank you for the clear and detailed description of the issue. I think I clearly understand the issue. It is caused by this line:

Wav2Lip/inference.py

Line 97 in 0600d0f

boxes = get_smoothened_boxes(np.array(results), T=5)

This line smooths bounding boxes over a 5 frame window. This is desirable as face detection can be at times be noisy and change a lot across frames. But, when there is rapid motion, the face position changes so much that the averaging the box location across the frames is undesirable.

We will look into this issue and update you once there is some sort of fix.

from wav2lip.

prajwalkr avatar prajwalkr commented on August 27, 2024 1

Please consider to mention the use of --nosmooththis on the main GitHub in case people would like to use it.

Will do it right away!

from wav2lip.

prajwalkr avatar prajwalkr commented on August 27, 2024

Can you pull the latest commit, and try with the --nosmooth argument and let me know?

from wav2lip.

AlonDan avatar AlonDan commented on August 27, 2024

Just to be clear: I'm a newbie when it comes to GitHub and I'm not a programmer so everything is very new to me, all I can do is to follow instructions and tutorials.
So I hope that I did it correct:

I've downloaded "inference.py" made a backup of the old one, and put this on the main master directory.

Then I used the same files I did for the test of this issue and at the end I've added --nosmooth like this:
python inference.py --checkpoint_path checkpoints/wav2lip_gan.pth --face MyDir/video.avi --audio MyDir/audio.wav --outfile MyDir/result_video.mp4 --nosmooth

but I get this error:
inference.py: error: unrecognized arguments: --nosmooth

I double checked that I put the NEWEST file (from about 30 minutes ago) instead of the old one, but still it seems like it's not recognizing the new command.

Maybe I did something wrong, not sure if I'm the right man to help testing but I can try if it's something simple enough for me to follow, sorry for the mess.

from wav2lip.

dipam7 avatar dipam7 commented on August 27, 2024

Hey, I had a similar issue. I ran it with the --nosmooth argument and it ran till a 100% but then gave me the following error:

Traceback (most recent call last):
  File "inference.py", line 280, in <module>
    main()
  File "inference.py", line 250, in main
    total=int(np.ceil(float(len(mel_chunks))/batch_size)))):
  File "/usr/local/lib/python3.6/dist-packages/tqdm/std.py", line 1127, in __iter__
    for obj in iterable:
  File "inference.py", line 111, in datagen
    face_det_results = face_detect(frames) # BGR2RGB for CNN face detection
  File "inference.py", line 101, in face_detect
    results = [[image[y1: y2, x1:x2], (y1, y2, x1, x2)] for image, (x1, y1, x2, y2) in zip(images, boxes)]
UnboundLocalError: local variable 'boxes' referenced before assignment

I see that line 101 uses boxes

Wav2Lip/inference.py

Line 101 in 8549b79

results = [[image[y1: y2, x1:x2], (y1, y2, x1, x2)] for image, (x1, y1, x2, y2) in zip(images, boxes)]

which we only retrieve if line 100 is true otherwise we don't set it to anything.

Wav2Lip/inference.py

Line 100 in 8549b79

if not args.nosmooth: boxes = get_smoothened_boxes(np.array(results), T=5)

from wav2lip.

prajwalkr avatar prajwalkr commented on August 27, 2024

@AlonDan can you check now if it is better?

from wav2lip.

AlonDan avatar AlonDan commented on August 27, 2024

Sure, I just tried it now it won't start running there is an error right after I press ENTER:

File "inference.py", line 6
    <!DOCTYPE html>
    ^
SyntaxError: invalid syntax

Even if I just run the main command:
python inference.py

I get the same error.

from wav2lip.

prajwalkr avatar prajwalkr commented on August 27, 2024

Hello, I think this is an error from your end, as you can see that the repo does not have that in line 6:

Wav2Lip/inference.py

Line 6 in 11a8eac

from glob import glob

from wav2lip.

AlonDan avatar AlonDan commented on August 27, 2024

Hmm, if that's the case I'm not sure how to fix that as I explained unfortunatly I'm a newbie in this.

f I switch back to the backup (old file version before the issue) everything works, but of course without the --nosmooth

Any suggestions to fix that issue on my side so I can test the new file version?
Thanks ahead.

from wav2lip.

prajwalkr avatar prajwalkr commented on August 27, 2024

Any suggestions to fix that issue on my side so I can test the new file version?

download the latest version once more. or the whole repo once more.

from wav2lip.

AlonDan avatar AlonDan commented on August 27, 2024

GOOD NEWS!
I just downloaded the latest file version again and did a test again, from the SAME video as the original issue, now it's STEADY as it should be, probably need to test on more complicated video situations but from the current one it's IMPRESSIVE!

Great job, thank you for the super quick fix!
here is the current version in case anyone would like to see the difference with --nosmooth
The mouth now following every single frame!

DOWNLOAD same video using --nosmooth as (zip file)

result_video (nosmooth enabled).zip

Suggestion:
Please consider to mention the use of --nosmooththis on the main GitHub in case people would like to use it.

Same Frame with --nosmooth and without:
1a
1b

from wav2lip.

dipam7 avatar dipam7 commented on August 27, 2024

I'm seeing weird artifacts near the lips for some videos even after using --nosmooth. Is this because I am using high res videos?

from wav2lip.

prajwalkr avatar prajwalkr commented on August 27, 2024

It’s hard to say without seeing the video. But, yes, high res videos and faces can produce artefacts.

from wav2lip.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.