wzmsltw / painttransformer Goto Github PK

View Code? Open in Web Editor NEW

308.0 10.0 33.0 12.2 MB

Official PaddlePaddle implementation of Paint Transformer

License: Apache License 2.0

Python 100.00%

painttransformer's Introduction

Paint Transformer: Feed Forward Neural Painting with Stroke Prediction

[Paper] [Paddle Implementation]

Update

We have optimized the serial inference procedure to achieve better rendering quality and faster speed.

Overview

This repository contains the official PaddlePaddle implementation of paper:

Paint Transformer: Feed Forward Neural Painting with Stroke Prediction,

Songhua Liu*, Tianwei Lin*, Dongliang He, Fu Li, Ruifeng Deng, Xin Li, Errui Ding, Hao Wang (* indicates equal contribution)

ICCV 2021 (Oral)

Prerequisites

Linux or macOS
Python 3.6+
PaddlePaddle 2.0+ and other dependencies (numpy, cv2, and other common python libs)
```
python -m pip install paddlepaddle-gpu
```

Getting Started

Clone this repository:

git clone https://github.com/wzmsltw/PaintTransformer
cd PaintTransformer

Download pretrained model from Google Drive and move it to inference directory:
```
mv [Download Directory]/paint_best.pdparams inference/
cd inference
```
Inference:
```
python inference.py
```
- Input image path, output path, and etc can be set in the main function.
- Notably, there is a flag serial as one parameter of the main function:
  - If serial is True, strokes would be rendered serially. The consumption of video memory will be low but it requires more time. Serial inference can achieve better rendering quality.
  - If serial is False, strokes would be rendered in parallel. The consumption of video memory will be high but it would be faster.
  - If animated results are required, serial must be True.
Train:
- You can send email to us for the training codes.

More Results

Input	Animated Output

App

Do not want to run the code? Try an App 一刻相册 downloaded from here!

Citation

If you find ideas or codes useful for your research, please cite:

@inproceedings{liu2021paint,
  title={Paint Transformer: Feed Forward Neural Painting with Stroke Prediction},
  author={Liu, Songhua and Lin, Tianwei and He, Dongliang and Li, Fu and Deng, Ruifeng and Li, Xin and Ding, Errui and Wang, Hao},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  year={2021}
}

Contact

For any question, please file an issue or contact

Songhua Liu: [email protected]
Tianwei Lin: [email protected]

painttransformer's People

Contributors

Stargazers

Watchers

painttransformer's Issues

Is it possible to set the number of the frames?

Hi,

Can I specify the iteration number, so for example can I get more total frames than 171?

Thanks!

Stylized paintings

How about the stylized paintings applications of Section 4.4 of the paper?

Brush stroke suggestion

I think the brush stroke should have some variable transparency, not just be 100% opaque. Something similar to what is being done here:
http://3dstereophoto.blogspot.com/2018/07/non-photorealistic-rendering-software.html

This is a stroke based renderer that simulates oil painting/digital painting using traditional methods, which means it probably creates too many strokes.

By having the brushstroke as a grayscale image where white means fully opaque and black fully transparent, some of the color that's on the canvas would still be visible after the brush stroke is applied, which would give in my opinion a more natural/realistic look.

IndexError: index -5178994697005420578 is out of bounds for axis 0 with size 2

Hi, I downloaded the code and ran the inference.py, but got this error
Traceback (most recent call last):
File "inference.py", line 64, in
main(input_path='input/chicago.jpg',
File "inference.py", line 49, in main
final_result_list = render_serial.render_serial(original_img, net_g, meta_brushes)
File "/data/zhang/PaintTransformer/inference/render_serial.py", line 240, in render_serial
get_single_layer_lists(param, decision, expand_img, render_size_x, render_size_y, h, w,
File "/data/zhang/PaintTransformer/inference/render_serial.py", line 20, in get_single_layer_lists
valid_foregrounds = render_utils.param2stroke(param[:, :], render_size_y, render_size_x, meta_brushes)
File "/data/zhang/PaintTransformer/inference/render_utils.py", line 55, in param2stroke
brush = paddle.to_tensor(meta_brushes_resize[index])
IndexError: index -5178994697005420578 is out of bounds for axis 0 with size 2
How could I fix this bug?

How to set do not resize?

I want to do not resize and set resize_h and resize_w as None, but got the error:
ValueError: (InvalidArgument) The 'shape' in Reshape0p is invalid.The input tensor X'size must be equal to the capacity of 'shape'......

index error when above 512

When setting resize_h or resize_w to let's say 1024 instead of 512 I get an error that it's out of index.
Is 512 the maximum size?

The vision of paddlepaddle-gpu.

The vision of paddlepaddle-gpu that I used is 2.1.3, and meet the requirements of version greater than 2.0 . It doesn't work. In the definition of the Transformer function, the number of initial parameters of the TransformerEncoder class does not correspond. The number of initial parameters in the 2.0 and 2.1 versions is 3, but 4 actual parameters are passed in in network.py. I want to know which version of paddlepaddle-gpu you have chosen.

License?

Hi, can you add a license? if I wanted to use output in a project with attribution?

Can you provide Google CoLab support?

As a mac user, I do not have valid GPU to use.
I have downloaded 一刻相机 to try your painting style, but it seems very slow.
Could you please provide online running demo like CoLab? Or does PaddlePaddle has online running feature?
Thanks in advance.

IndexError

environment：

windows python3.6
CUDAv10.2 and Corresponding cdnn、paddle（2.4.0.port102），

where:

render_utils.py line 56：rush = paddle.to_tensor(meta_brushes_resize[index])

IndexError
index 30238054144 is out of bounds for axis 0 with size 2

issue about CUDNN

你好，我根据 readme 文件安装了 paddlepaddle-gpu 2.5.1.post112 在使用您提供的推理代码时出现下面警告
WARNING: device: . The installed Paddle is compiled with CUDNN 8.2, but CUDNN version in your machine is 8.0, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version.
并报错
python: symbol lookup error: /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcublas.so: undefined symbol: init_gemm_select, version libcublasLt.so.11
请问应该作者在实验的时候用的 paddle 版本是什么样的

Training codes

May I ask for the training codes? I've sent 2 emails but no replies yet.