There is nothing wrong on your part. RAFT model is far from perfect, so it tries to predict optical flow where there is none, and the diff map are calculated based on that flow. There are several things you can do to eliminate it. You can go to flow_utils.py and remove occlusion mask from the diff calculation of the 79 line like so:
alpha_mask = np.maximum(diff_mask_org * 4, diff_mask_stl * 2)
You may also want to try it like this:
alpha_mask = diff_mask_org * 4

Otherwise you could also try a different model that the RAFT method provides. To do so, you would need to change line 19 of flow_utils.py to one of these:

'model': 'RAFT/models/raft-sintel.pth'
or
'model': 'RAFT/models/raft-kitti.pth'
or
'model': 'RAFT/models/raft-chairs.pth'

If you find out that one of these models works better, please report it here. I haven't tried other models yet.

from sd-cn-animation.

ill try the models

from sd-cn-animation.

Raft-sintel is arguably worse by far

raft-kitti is even worse still

raft-chairs looked like it was going to be better but then fell apart

going back to raft-things but removing the occlusion mask

changing the alpha mask to only use the diff_mask_org * 4

i dont see much difference in any method.

is there a better option than using raft? i know it was implemented to reduce VRAM, but is there a more vram expensive method that would yield a better result?

from sd-cn-animation.

This is very strange that you still has white areas with diff_mask_org * 4. That tells that there are actual differences between frames that become exaggerated because of multiplier.

Regarding the better flow estimation model, this should work much better: https://github.com/autonomousvision/unimatch but I never worked with this method, so I don't know yet how hard/easy it is to incorporate it to the project.

from sd-cn-animation.

I can only assume then that the black frames are somehow filled with noise from the compression method, though i assumed uncompressed AVI was, well... uncompressed.
Ill have to look into the frames themselves and see if i can find some difference in the background, ill also look at implementing unimatch, its about time i got some real coding done

from sd-cn-animation.

ive used ffmpeg to pull the frames out as bmp and the black areas are 100% pure black, frame to frame there is zero data in the black areas.

the only thing i can think is some form of compression

ok, used ffmpeg to put together a file using lib264 and lib264rgb at crf of 0, same thing, there is or should be no variation in the black parts of the image, and yet the optical flow thing is detecting them

from sd-cn-animation.

Im not python programmer but maybe background subtraction can help you ? https://www.geeksforgeeks.org/python-background-subtraction-using-opencv/

from sd-cn-animation.

i implemented a background removal during the processing (its not a great implementation due to MOG2 being what is, and i dont recommend its use) however it shows the same effect.

if someone wants to recommend a better method to remove the background im open to listening, it needs testing, but its a start.

#16

from sd-cn-animation.

There's this background remover, I don't know if this better : https://github.com/danielgatis/rembg

from sd-cn-animation.

I asked chatgpt for fix the problem and add in flow_utils.py these lines :

remove white noise

next_flow[np.abs(next_flow) < 3] = 0
prev_flow[np.abs(prev_flow) < 3] = 0

it seems to have fixed the artifacts problem :

from sd-cn-animation.

The complete file https://www.file.io/q5f2/download/m3VrmDDImSZM

from sd-cn-animation.

@alexfredo Seems reasonable. You should make a pull-request. I guess this kind of threshold should not do any harm.

from sd-cn-animation.

I need to test on several video before for be sure that works well everywhere, also I never made a pull-request but it doesn't seems complicated I can try ^^;

from sd-cn-animation.

I noticed something weird with my modification when I generate the flow.h5 file the flow preview show a bad result, but then when I use the flow file with "py vid2vid.py" the preview show a good result without artifacts

from sd-cn-animation.

Is that in addition to the background remover code I suggested or for the original method?

I hadn't considered just removing the noise lol.

from sd-cn-animation.

The original method

from sd-cn-animation.

The problem is when there's a sudden change of scene the accuracy of the motions become bad, chatgpt said "To improve the accuracy of flow estimation when sudden scene changes occur, one approach is to use post-processing techniques such as filtering and smoothing. " is it true ? the other solution and to separate all the scenes, it require more work...

from sd-cn-animation.

I know when I'm doing AI frame interpolation you get bad data around scene changes, it passes the sniff test.

But if a little white noise removal is all that's needed then sure why not, the noise is unlikely to be consistent frame to frame but motion usually lasts more than one frame so noise removal makes sense.

from sd-cn-animation.

hey, so, funny story... ive been doing some work on this over the last 2 days... made some discoveries...
RAFT shits itself with large areas of solid colour.
If i take a background removed image, and add a checkerboard pattern, most of the noise goes away.
If i add a noise pattern, all of the noise goes away
so it seems that background removal is not actually doing us any favours on its own.

ive played around with noise removal (btw the noise removal needs to be in the estimate flow part of the script rather than the compute diff map, it seems to help with the noise during both the vizualising and the compute diff part so you can see straight away if its going to give you issues.

as for background removal? i would maybe consider stripping it out tbh, the method i put in is so dirty, i did not expect it to be merged tbh, however if you want to keep it, just move the low value noise suggestion into the estimate flow area

useless code blocks in these comments

from sd-cn-animation.

if you want true black areas you need to use WebM, HEVC With Alpha or ProRes4444 with transparency enabled. Every other codec will show compression artifacts in black backgrounds.

from sd-cn-animation.