Where the neural network stores information about the tags? about deepdanbooru HOT 3 CLOSED

Json files in images folder are used for debugging purpose. It does not need to training. (also tags-xxx.txt are used for debugging too.)

Essential files are these:
Image files in images folder, database sqlite file, tags.txt.
Training data is loaded from images folder and database sqlite file, then model's output dimension is
calculated by using tags.txt.

So if you want to merge multiple databases into single one, you should modify both database.sqlite and tags.txt.

from deepdanbooru.

If the output dimension increase, memory usage will be high and you need more memory and training time.

The best case is that tags.txt has only tags which you want to estimate. In ex, if you want to tag only characters, tags.txt have to contain only that character tags.

If the tag is not present in tags.txt, it will be simply ignored and do not affect training performance.

from deepdanbooru.

Thank you for your detailed answer, I have new questions.

What happens if the tag that is specified in the database record is not present in the tags file.txt?

Speaking of dimension, I'm training a model for tagging on a specific anime universe (naruto), the total number of images will be no more than 200k. And the number of tags in them, about 60 for characters and 1000-3000 for General tags.

That is, in fact, in a project for 200k images can contain 1000-3500 unique tags, while in the file: tags.txt, unique tags will be more than 100k. Will this not affect the training or performance?

It might be better if I write a script that generates tags.txt only from those tags that are in the database? Or am I worrying for nothing?

from deepdanbooru.