hepesu / linefiller Goto Github PK

Automatic coloring tool can be disassembled into several parts, line-closing tool,floodfill tool, area classification tool, color design tool, shading tool. There are a lot of tools to fill the area, such as MultiFill, but these tools need closing line-art. The deep learning tool can now complete the parts besides shading. As for tools in project HAT, LineRelifer and LineFiller can be used to complete the basic color filling, which fulfill line-closing and floodfill parts. ​​​​

As for full version of LineFiller, it will contain a model performs area classification. And JACS provides a model for color design, as well as unpublished LineShader to complete the shading. There are currently two implementations of LineFiller, one is traditional algorithm for filling and a convolution neural network model for classification, the other one is based on reinforcement learning.

Back to the end-to-end model based on deep learning, in fact if you have tried, your might find that the color design of model is weak, leads to lots of instructions for color suggestion. And sometimes due to incomplete classification of area, more instructions needed. So the total efficiency might be not as good as the paint bucket. If you want to try new color palette, you would have to add more suggestions. Therefore, in actual pipeline, automatic coloring workflow needs to be break into parts, which can be guided by human in any step and if a problem is solved, it will not appear in next step.

As to the coloring style which deep learning models have, full version of LineFiller only to finish the Cel style. However, it is possible to get styled color result without deep learning (see the Blur filter of CelFX). And various style models can be trained, CAN is a potential structure for this task.

To sum up, in this way, every steps are available for human guide, and after finishing the Cel shade, you can get any style with transformation model. It is suitable for production pipeline.

In the first commit, I use a simple algorithm to group the fill results. It works, but not good.

For the next release, I am trying using CNN to extract feature then using cluster method for grouping. This may work, because someone used this for shape segmentation, a similar task. And I am busy doing annotation for training, it will be public soon.

But I am thinking about using more interesting method, maybe reinforcement learning. A little filling game, the agent has some actions, one is floodfill. Actions are inferred with CNN from the filling result. The reward depends on result and if all areas are filled, the game is finished. Training data of second method can be used, but the environment need some code.

Hi, thanks for your work.

While testing your codes, I found an error:

/content/LineFiller/linefiller/trappedball_fill.py in get_border_point(points, rect, max_height, max_width)
    284 
    285     # Get shape.
--> 286     _, contours, _ = cv2.findContours(fill, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    287     approx_shape = cv2.approxPolyDP(contours[0], 0.02 * cv2.arcLength(contours[0], True), True)
    288 

ValueError: not enough values to unpack (expected 3, got 2)

I guess there is a version compatibility issue for opencv, since cv2.findContours may return 2 or 3 values according to the opencv version.

I think the solution is simple:

# as is
_, contours, _ = cv2.findContours(fill, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

# to be
contours = cv2.findContours(fill, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[-2]

Thanks.

Hi hepesu,

Great work on this project, I was amazed at how well your segmentation performs.

Do you have any times on how to get this to run faster? If I pass it an HD image with a lot of segments, it sometimes takes up to 30 seconds on my laptop. It seems like the majority of the time is being spend in the flood_fill_multi and merge_fill functions. Is there anyway these functions can be optimized?

I am also wondering if it would be possible to split up the input images into patches and run segmentation in parallel on each patch. Once each patch is finished you could merge them back together. What do you think of this approach?