How does the inference command work for clothing transfer, exactly? about swapnet HOT 4 CLOSED

Hi Jerry, thanks for your question.

Regarding a missing stats file, check out this part of the Readme. I forgot to upload that file with the dataset, and haven't gotten around to reuploading the whole thing to add the single file yet.

For your question about what clothing gets transferred, at the moment, by default it does a one-to-one mapping based on the image index i (sorted alphabetically). So the model will use the clothes_i (Source) to transfer onto body_i (Target). As long as the number of files in Source >= number of files in Target, it should run fine.

In the first sanity check in the Readme, we use this:

python inference.py --checkpoint checkpoints/deep_fashion \
  --dataroot data/deep_fashion \
  --shuffle_data True

to perform inference on the DeepFashion dataset itself. The default one-to-one mapping would mean we'd transfer the clothing onto the same body it belonged to originally. This doesn't make much sense, so we use the --shuffle_data option so that clothes get transferred to random bodies.

For the inference command you mentioned,

python inference.py --checkpoint checkpoints/deep_fashion \
  --cloth_dir [SOURCE] --texture_dir [SOURCE] --body_dir [TARGET]
this is to control which cloth (Source) we want to transfer to what body (Target). 

The first important thing is that if you use the --texture_dir, --cloth_dir, --body_dir options individually, they should point to dataroot/texture, dataroot/cloth, dataroot/body/ folders respectively. They do not each point to just dataroot.

The reason these inputs are folders instead of single images is in case you want to process a video. You would put all the frames from a video in the same folder. If you want to transfer a single image, then just put a single image in each folder.

Hope that helps a bit. I know it's bit complicated, please let me know if you have more questions.

from swapnet.

Another important thing to note about SwapNet's clothing transfer - it's not exactly "transfer" (at least I don't think so). It's more like: learn to repose the source into any arbitrary pose; during inference, just repose the clothes to match that of the target body. The identity of the target user is not preserved at all, and SwapNet resorts to copying over the face pixels from the target to "preserve" identity (I haven't implemented this part yet, see #6). There has definitely been better clothing transfer works published since.

from swapnet.

I see, this makes a lot of sense (I was wondering why the target body directory didn't need a textures directory too - I guess it just doesn't use them if the target isn't preserved since it just needs pose info).

I got the basic inference to work well, but would you know of any open-source code that I can build on for those better transfer networks? Would love to hop on a call to talk more about the topic as well since I'm relatively new to this area of research - you can reach me at [email protected]. Thanks!

from swapnet.

Hi Jerry, we can connect on LinkedIn and set something up. I'm https://www.linkedin.com/in/ajong/. Cheers.

from swapnet.