Giter VIP home page Giter VIP logo

todds's Introduction

todds

License: MPL 2.0 Contributor Covenant

A CPU-based DDS texture encoder optimized for fast batch conversions with high encoding quality.

Features

ARGS:
  input                       Encode all PNG files inside of this folder as DDS. It can also point to a single PNG file. If this parameter points to a TXT file, it will be processed as a list of PNG files and/or directories. Entries must be on separate lines. Every listed PNG file and those inside listed directories will be encoded as DDS.
  output                      Write DDS files to this folder instead of creating them next to input PNGs. This argument is ignored if input points to a TXT file.

OPTIONS:
  -cl, --clean                Deletes all DDS files matching input PNG files instead of encoding them.
  -f, --format                DDS encoding format.
                                  BC7: High-quality compression supporting alpha. [Default]
                                  BC1: Highly compressed RGB data.
                                  PNG: PNG format, intended for downscaling or metric calculation.
  -af, --alpha-format         Use a different DDS encoding format for files with alpha. Defaults to using the value in --format unconditionally.
                                  BC7: High-quality compression supporting alpha. [Default]
  -q, --quality               Encoder quality level, must be in [0, 7]. Defaults to 6.
  -nm, --no-mipmaps           Disable mipmap generation.
  -fs, --fix-size             Set image width and height to the next multiple of 4.
  -mf, --mipmap-filter        Filter used to resize images during mipmap generation.
                                  LANCZOS: Lanczos interpolation. Preserves edges and details better than other filters when dowsncaling images. [Default]
                                  NEAREST: Nearest neighbor interpolation. Very fast, but it does not produce great results.
                                  LINEAR: Bilinear interpolation. Fast, and with reasonable quality.
                                  CUBIC: Bicubic interpolation. Recommended filter for upscaling images.
                                  AREA: Resampling using pixel area relation. Good for downscaling images and mipmap generation.
  -mb, --mipmap-blur          Blur applied during mipmap generation. Defaults to 0.55.
  -sc, --scale                Scale image size by a value given in %.
  -ms, --max-size             Downscale images with a width or height larger than this threshold to fit into it.
  -sf, --scale-filter         Filter used to scale images when using the scale or max_size parameters.
                                  LANCZOS: Lanczos interpolation. Preserves edges and details better than other filters when dowsncaling images. [Default]
                                  NEAREST: Nearest neighbor interpolation. Very fast, but it does not produce great results.
                                  LINEAR: Bilinear interpolation. Fast, and with reasonable quality.
                                  CUBIC: Bicubic interpolation. Recommended filter for upscaling images.
                                  AREA: Resampling using pixel area relation. Good for downscaling images and mipmap generation.
  -th, --threads              Number of threads used by the parallel pipeline, must be in [1, 32]. Defaults to maximum.
  -d, --depth                 Maximum subdirectory depth to use when looking for source files. Defaults to maximum.
  -o, --overwrite             Convert files even if an output file already exists.
  -on, --overwrite-new        Convert files if an output file exists, but it is older than the input file.
  -vf, --vflip                Flip source images vertically before encoding.
  -t, --time                  Show total execution time.
  -r, --regex                 Process only absolute paths matching this regular expression.
  -dr, --dry-run              Calculate all files that would be affected but do not make any changes.
  -p, --progress              Display progress messages.
  -v, --verbose               Display all input files of the current operation.
  -h, --help                  Show usage information.

ADVANCED OPTIONS:
  -bc1-ab, --bc1-alpha-black  The BC1 encoder will use 3 color blocks for blocks containing black or very dark pixels. Increases texture quality substantially, but programs using these textures must ignore the alpha channel.
  -rp, --report               Prints information about the encoding process of each file.

Quality

BC7 support is implemented using the bc7e.ispc library. BC1 support has been implemented with the rgbcx library.

As seen in the Texture Compression in 2020 blog post, these libraries provide great encoding quality and performance.

Check the Analysis documentation for details.

Performance

todds is optimized for encoding multiple files at the same time, without compromising on encoding quality. The full benefits of todds are reached when it is used to encode large numbers of files, but it also performs well while encoding single files.

Check the Analysis documentation for details.

Building

todds uses CMake presets and a vcpkg submodule to retrieve its dependencies and compile. Compiling also requires the Intel® Implicit SPMD Program Compiler. For more details, check the wiki.

Compiling without vcpkg

To compile todds without vcpkg, all of its dependencies must be available as development libraries. Check the dependencies section for details.

Dependencies

All of the following dependencies are available through vcpkg.

The following third party library dependencies are contained as source code in the thirdparty folder of the todds repository. Each one of these libraries is under its own license which is also included in the repository.

CMake options

  • TODDS_CLANG_ALL_WARNINGS: This option is only available when the clang compiler is in use. This enables almost every Clang warning, except for a few that cause issues with todds. This may trigger unexpected positives when using newer Clang versions. Off by default.
  • TODDS_CLANG_TIDY: If clang-tidy is available, it will be used to analyze the project. Off by default.
  • TODDS_ISPC: Enables use of the bc7e_ispc for encoding BC7 files, which uses SIMD and requires the ispc compiler. On by default. If this setting is disabled, BC7 encoding will take longer and might have decreased quality.
  • TODDS_MIMALLOC_ALLOCATOR: todds will use the mimalloc allocator instead of the standard allocator.
  • TODDS_NEON_SIMD: Use NEON SIMD instructions instead of x64 SIMD instructions. Intended for compiling for ARM platforms.
  • TODDS_PIPELINE_DUMP: Dump the binary data generated by each pipeline stage. Limited to a single file. The dump files will be generated next to the todds executable.
  • TODDS_REGULAR_EXPRESSIONS: Enables the regular expression parameter for todds. Requires the Hyperscan library.
  • TODDS_TBB_ALLOCATOR todds will use the scalable_allocator from oneTBB instead of the standard allocator.
  • TODDS_UNIT_TESTS -> Build todds unit tests. Requires the Catch2 library. Off by default.
  • TODDS_WARNINGS_AS_ERRORS: Treat all compiler warnings as errors. Off by default.
  • TODDS_TRACY: Compiles todds with Tracy Profiler support. todds will use a custom allocator that will expose allocation data to Tracy.

Contributing

todds encourages community involvement and contributions. Check the CONTRIBUTING.md and CODE_OF_CONDUCT.md files for details. You can check all contributors in the contributors list.

License

todds is licensed under the Mozilla Public License, v. 2.0. See the LICENSE file for details. Check the MPL 2.0 FAQ to learn more.

Acknowledgements

Read the ACKNOWLEDGEMENTS.md file for details.

todds's People

Contributors

joseasoler avatar twstagg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

todds's Issues

Impractical error message when loading incorrect paths

Describe the bug
When loading an incorrect path, todds outputs error messages that do not help at all in identifying what is wrong.

To Reproduce
Steps to reproduce the behavior:

  1. Launch todds on Windows. Provide a malformed path such as E:/Games/RimWorld/Mods\animal-gear-master\Textures\Things\Pawn\Animal\Apparel\emptyGear_south.png
  2. todds displays the following errors:

Load PNG file error in E:/Games/RimWorld/Mods\animal-gear-master\Textures\Things\Pawn\Animal\Apparel\emptyGear_south.png Could not load any data for PNG file E:/Games/RimWorld/Mods\animal-gear-master\Textures\Things\Pawn\Animal\Apparel\emptyGear_south.png

Expected behavior
todds displays helpful error messages which help diagnose this issue.

System (please complete the following information):

  • OS: Windows

Suggestion

Can you make it so that we can set how many concurrent jobs can be run for conversion

Exception with corrupted filesystems

A Windows user was experiencing the following exception:

todds has been terminated because of an exception: filesystem::recursive_directory_iterator increment error: The system cannot find the file specified [system:2]

According to this boost::filesystem issue, this exception happens when the filesystem is corrupt. Another fact that points to a corruption as the cause is that I managed to repro this issue at one point using mods from RimWorld as the input, but the issue went away after validating game files and letting Steam update the mods.

Potential fixes are discussed in the boost::filesystem issue mentioned above and in here: boostorg/filesystem@9a14c37

Core detection in Threadripper CPUs

ToDDS (in this case the RimWorld UI version) is only using 16 cores by default on a AMD Ryzen Threadripper 3970X CPU, which has 32 cores.

dnn_superres upscaling support

OpenCV has implementations of specialized upscaling algorithms that todds currently does not support, which are described for example in here: https://towardsdatascience.com/deep-learning-based-super-resolution-with-opencv-4fd736678066. It should be possible to integrate these algorithms into the todds codebase and its pipeline.

Benchmarks and quality metrics: https://docs.opencv.org/4.x/dc/d69/tutorial_dnn_superres_benchmark.html

When upscaling is used, todds could use the original image as the first level mipmap, to minimize quality loss during mipmap calculations.

Ram usage while using Todds

For some reason Todds keep using more and more ram, and after like 5min it crashes Wayland cause there is not enought ram available (no swap)

I didn't look at how Todds work but I assume it inport game asset into memory, are they removed after ?
Note: i only use the one include in rimsort

Pipeline refactor

The pipeline should be better encapsulated, to allow using it as a static library on separate applications.

bitdenfender think todds is a virus

Describe the bug
bitdenfender think todds is a virus

To Reproduce
download todds

System (please complete the following information):
windows 11

Additional context
image

im not the one who got the repport btw

Improve error messages reported to the user

When a filesystem exception happens, png2dds is just printing it instead of reporting a proper error back to the user. This should be improved.

Example of exception: png2dds has been terminated because of an exception: filesystem::recursive_directory_iterator increment error: The system cannot find the file specified [system:2]

Compile bc7e_ispc on ARM platforms

The current CMake script for compiling bc7e_ispc hardcodes the SIMD instruction set and object files that ispc will generate. On ARM platforms, the SIMD instruction set should be --target=neon. This will also only generate bc7e.o, without any additional bc7e_####.o files.

Ideally, the script should be modified to work in all platforms without any extra workarounds.

Generation of binaries compatible with most Linux distributions

Since todds supports static builds, it is possible to generate a binary that contains all of the dependencies. The remaining issue has always been glibc. So far, we have been building binaries using increasingly older Linux distributions, in the hopes of finally finding a combination that works for everyone.

This is time consuming, as it requires setting up a new VM and figuring out how to compile todds in that case. Also, lately we have been reaching a point where using even older distros would mean switching from GCC 13 to older versions, which could potentially cause performance impacts. We need an alternative which addresses these problems.

One potential way to address all of this would be to use crosstool-ng. This is explained in the "Setup 2" part of this excellent Stackoverflow response: https://stackoverflow.com/a/52550158

This setup should give us freedom to choose which glibc version to use without requiring a VM.

Optimize --overwrite-new

When this setting is used, todds takes a noticeable amount of time before the progress bar is shown. It should be possible to optimize these checks. Some feedback to the user would also be helpful.

Total pixels / time

todds already keeps an internal list of data with an entry for every image, which contains the width and height among other data.

By accumulating pixel count and using the total time that is already there, it would be possible to print the total pixels and total pixels by time.

With the new report system, it will also possible to estimate the current pixels / time.

Refactorization of pipeline.cpp

Currently this file contains the implementation of all filters. Changing one of them leads to long compilation times. They should be split into different cpp files to simplify the code and prevent this issue.

Incorrect DDS files on MacOS M1

Describe the bug
todds produces incorrect DDS files on Mac OS M1

To Reproduce

  1. Compile todds in an ARM Mac computer.
  2. Encode any dds texture
  3. Check the texture in TacentView.

Expected behavior
The texture does not have any artifacts.

System (please complete the following information):

  • OS: MacOS
  • CPU model: ARM

Capture

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.