arm-software / astc-encoder Goto Github PK

Greetings astc-encoder developers and contributors,

We’re reaching out because your project is an important part of the open source ecosystem, and we’d like to invite you to integrate with our fuzzing service, OSS-Fuzz. OSS-Fuzz is a free fuzzing infrastructure you can use to identify security vulnerabilities and stability bugs in your project, like #36 and #35. OSS-Fuzz will:

• Continuously run all the fuzzers you write.
• Alert you when it finds issues.
• Automatically close issues after they’ve been fixed by a commit.

Many widely used open source projects like OpenSSL, FFmpeg, LibreOffice, and ImageMagick are fuzzing via OSS-Fuzz, which helps them find and remediate critical issues.

Even though typical integrations can be done in < 100 LoC, we have a reward program in place which aims to recognize folks who are not just contributing to open source, but are also working hard to make it more secure.

We want to stress that anyone who meets the eligibility criteria and integrates a project with OSS-Fuzz is eligible for a reward.

If you're not interested in integrating with OSS-Fuzz, it would be helpful for us to understand why—lack of interest, lack of time, or something else—so we can better support projects like yours in the future.

If we’ve missed your question in our FAQ, feel free to reply or reach out to us at [email protected].

Thanks!

Julien,
OSS-Fuzz Team

Many use cases want predictable results across platforms and for multi-threaded library code. This isn't possible for use cases using std::rand() so we should replace this with a stable RNG which can be seeded per thread.

Hello,everyone! I use my company tools to fuzz astc.I found a abort in astc on Linux64. I think it is due to zbuild_huffman().So I will show you some detail information to help you find the leak.

I am trying to compress my normal maps for my iOS game, but by default I receive some noise even in 4x4 block sizes. After further research I have found -normal_psnr, but it makes texture in green Chanel (image attached).
I am using Mali texture Compression Tool.
What I am doing wrong ?
Sorry for this question, I am pretty new in the field

Input .png image

Settings:



Output ASTC texture:

The astcenc executable has its process priority to High by default causing any lengthy encoding to lock up Windows. A higher priority doesn't make it run any faster, it just prevents other programs from working correctly.

Hi,

Our fuzzer found an assertion failure on the function compute_huffman_codes (the latest commit 5ff4d86 on master). This vulnerability causes the program to abort.

PoC: https://github.com/strongcourage/PoCs/blob/master/astc-encoder_5ff4d86/PoC_af_compute_huffman_codes

ASAN says:

astcenc-asan -c $PoC /dev/null 6x6 -medium 
Encoding settings:

2D Block size: 6x6 (3.56 bpp)
3D Block size: 6x6x1 (3.56 bpp)
Radius for mean-and-stdev calculations: 0 texels
RGB power: 1
RGB base-weight: 1
RGB local-mean weight: 0
RGB local-stdev weight: 0
RGB mean-and-stdev mixing across color channels: 0
Alpha power: 1
Alpha base-weight: 1
Alpha local-mean weight: 0
Alpha local-stdev weight: 0
RGB weights scale with alpha: disabled
Color channel relative weighting: R=1 G=1 B=1 A=1
Block-artifact suppression parameter : 0
Number of distinct partitionings to test: 25 (preset)
PSNR decibel limit: 2D: 40.529411 3D: 40.529411 (preset)
1->2 partition limit: 1.200000
Dual-plane color-correlation cutoff: 0.750000 (preset)
Block Mode Percentile Cutoff: 75.000000 (preset)
Max refinement iterations: 2 (preset)
Thread count : 8 (autodetected)

astcenc-asan: stb_image.c:2182: int compute_huffman_codes(zbuf*): Assertion `c >= 0 && c < 19' failed.
Aborted

Thanks,
Manh Dung

Hi,

Our fuzzer found a crash due to a heap buffer overflow on the function tga_load on the latest commit 5ff4d86 on master..

PoC_hbo_tga_load: https://github.com/strongcourage/PoCs/blob/master/astc-encoder_5ff4d86/PoC_hbo_tga_load

ASAN says:

astcenc -c $PoC /dev/null 6x6 -medium
Encoding settings:

2D Block size: 6x6 (3.56 bpp)
3D Block size: 6x6x1 (3.56 bpp)
Radius for mean-and-stdev calculations: 0 texels
RGB power: 1
RGB base-weight: 1
RGB local-mean weight: 0
RGB local-stdev weight: 0
RGB mean-and-stdev mixing across color channels: 0
Alpha power: 1
Alpha base-weight: 1
Alpha local-mean weight: 0
Alpha local-stdev weight: 0
RGB weights scale with alpha: disabled
Color channel relative weighting: R=1 G=1 B=1 A=1
Block-artifact suppression parameter : 0
Number of distinct partitionings to test: 25 (preset)
PSNR decibel limit: 2D: 40.529411 3D: 40.529411 (preset)
1->2 partition limit: 1.200000
Dual-plane color-correlation cutoff: 0.750000 (preset)
Block Mode Percentile Cutoff: 75.000000 (preset)
Max refinement iterations: 2 (preset)
Thread count : 8 (autodetected)

=================================================================
==10365==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f50735bf438 at pc 0x0000004dbaa0 bp 0x7ffcaf6be450 sp 0x7ffcaf6be440
READ of size 1 at 0x7f50735bf438 thread T0
    #0 0x4dba9f in tga_load /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:3430
    #1 0x4dba9f in stbi_tga_load /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:3452
    #2 0x4dba9f in stbi_load_main /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:552
    #3 0x4dc009 in stbi_load_from_file /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:571
    #4 0x4dc009 in stbi_load /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:562
    #5 0x4877b9 in load_image_with_stb(char const*, int, int*) /home/dungnguyen/gueb-testing/astc-encoder/Source/astc_stb_tga.cpp:66
    #6 0x46bff0 in astc_codec_load_image(char const*, int, int*) /home/dungnguyen/gueb-testing/astc-encoder/Source/astc_image_load_store.cpp:1328
    #7 0x49a3dd in astc_main(int, char**) /home/dungnguyen/gueb-testing/astc-encoder/Source/astc_toplevel.cpp:2329
    #8 0x7f507679982f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #9 0x402738 in _start (/home/dungnguyen/PoCs/astc-encoder_5ff4d86/astcenc-asan+0x402738)

0x7f50735bf438 is located 968 bytes to the left of 7598656-byte region [0x7f50735bf800,0x7f5073cfea40)
allocated by thread T0 here:
    #0 0x7f5077101602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x4d89db in tga_load /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:3289
    #2 0x4d89db in stbi_tga_load /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:3452
    #3 0x4d89db in stbi_load_main /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:552

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:3430 tga_load
Shadow bytes around the buggy address:
  0x0fea8e6afe30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea8e6afe40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea8e6afe50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea8e6afe60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea8e6afe70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0fea8e6afe80: fa fa fa fa fa fa fa[fa]fa fa fa fa fa fa fa fa
  0x0fea8e6afe90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea8e6afea0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea8e6afeb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea8e6afec0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fea8e6afed0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==10365==ABORTING

Thanks,
Manh Dung

I need to sort opaque and transparent/translucent objects differently (front-to-back vs back-to-front). I understand that ASTC decodes to RGBA, but if the original source was RGB, then I know I can treat that object as an opaque object.

I see in astc_codec_internals.h there is an enumeration of endpoint_formats (LUMINANCE, LUMINANCE_ALPHA, RGB, RGBA, etc.) that looks like it could tell me what i'm looking for. Is this stored in the file anywhere? It would not be optimal for the GPU to treat all objects as transparent and sort them back-to-front even if they were really opaque. But without further info, I'm not sure what else to do.

hints?

I use g++ 5.4.0 and AddressSanitizer to build astc-encoder v1.4, this file can cause memory leak when executing this command:

./astcenc -c pngtest.png 1.astc 6x6 -medium

This is the ASAN information:

Encoding settings:

2D Block size: 6x6 (3.56 bpp)
3D Block size: 6x6x1 (3.56 bpp)
Radius for mean-and-stdev calculations: 0 texels
RGB power: 1
RGB base-weight: 1
RGB local-mean weight: 0
RGB local-stdev weight: 0
RGB mean-and-stdev mixing across color channels: 0
Alpha power: 1
Alpha base-weight: 1
Alpha local-mean weight: 0
Alpha local-stdev weight: 0
RGB weights scale with alpha: disabled
Color channel relative weighting: R=1 G=1 B=1 A=1
Block-artifact suppression parameter : 0
Number of distinct partitionings to test: 25 (preset)
PSNR decibel limit: 2D: 40.529411 3D: 40.529411 (preset)
1->2 partition limit: 1.200000
Dual-plane color-correlation cutoff: 0.750000 (preset)
Block Mode Percentile Cutoff: 75.000000 (preset)
Max refinement iterations: 2 (preset)
Thread count : 8 (autodetected)

pngtest.png: 2D LDR image, 91 x 69 x 1, 4 components

192 blocks to process ..
183
=================================================================
==7141==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x7f6a08b546b2 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x996b2)
    #1 0x4c24d6 in main /home/fouzhe/my_fuzz/astc-encoder/Source/astc_toplevel.cpp:2321
    #2 0x7f6a07c5382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Direct leak of 4 byte(s) in 1 object(s) allocated from:
    #0 0x7f6a08b546b2 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x996b2)
    #1 0x4c24a3 in main /home/fouzhe/my_fuzz/astc-encoder/Source/astc_toplevel.cpp:2320
    #2 0x7f6a07c5382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 25116 byte(s) in 1 object(s) allocated from:
    #0 0x7f6a08b546b2 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x996b2)
    #1 0x4773fb in allocate_image(int, int, int, int, int) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_image_load_store.cpp:63
    #2 0x4a34bc in load_image_with_stb(char const*, int, int*) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_stb_tga.cpp:72
    #3 0x482334 in astc_codec_load_image(char const*, int, int*) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_image_load_store.cpp:1328
    #4 0x4c2550 in main /home/fouzhe/my_fuzz/astc-encoder/Source/astc_toplevel.cpp:2329
    #5 0x7f6a07c5382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 552 byte(s) in 1 object(s) allocated from:
    #0 0x7f6a08b546b2 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x996b2)
    #1 0x477372 in allocate_image(int, int, int, int, int) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_image_load_store.cpp:62
    #2 0x4a34bc in load_image_with_stb(char const*, int, int*) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_stb_tga.cpp:72
    #3 0x482334 in astc_codec_load_image(char const*, int, int*) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_image_load_store.cpp:1328
    #4 0x4c2550 in main /home/fouzhe/my_fuzz/astc-encoder/Source/astc_toplevel.cpp:2329
    #5 0x7f6a07c5382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f6a08b54532 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x99532)
    #1 0x4771a1 in allocate_image(int, int, int, int, int) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_image_load_store.cpp:49
    #2 0x4a34bc in load_image_with_stb(char const*, int, int*) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_stb_tga.cpp:72
    #3 0x482334 in astc_codec_load_image(char const*, int, int*) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_image_load_store.cpp:1328
    #4 0x4c2550 in main /home/fouzhe/my_fuzz/astc-encoder/Source/astc_toplevel.cpp:2329
    #5 0x7f6a07c5382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x7f6a08b546b2 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x996b2)
    #1 0x477313 in allocate_image(int, int, int, int, int) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_image_load_store.cpp:61
    #2 0x4a34bc in load_image_with_stb(char const*, int, int*) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_stb_tga.cpp:72
    #3 0x482334 in astc_codec_load_image(char const*, int, int*) /home/fouzhe/my_fuzz/astc-encoder/Source/astc_image_load_store.cpp:1328
    #4 0x4c2550 in main /home/fouzhe/my_fuzz/astc-encoder/Source/astc_toplevel.cpp:2329
    #5 0x7f6a07c5382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: 25720 byte(s) leaked in 6 allocation(s).

Hello,everyone! I use my company tools to fuzz astc.I found a crash in astc on Linux64. I think it is due to parse_png_file().So I will show you some detail information to help you find the leak.

One common request is to support the ASTC codec as a library, rather than a standalone executable. To make this possible:

• Remove all global variables, as they cause problems with parallel usage
• Split out the command line tool behavior from the core codec. Core codec should read and write to memory buffers; no file access.
• Provide support for hooking into custom thread pools (but still allow automatic threading if the user wants it).
• Provide build system support to build the library.

Last time I saw a warehouse that supported ktx, but now it seems to be 404. I can't enter it.

We have a number of paths for loading and storing images in a variety of formats:

• stb_image.c provides the bulk of the formats
• custom logic for TGA files (although stb_image also handles TGA files)
• custom logic for non-standard HTGA files
• custom logic for ASTC files
• custom logic for KTX and DDS files
• support tools needed on the command line for OpenEXR

We have inconsistent interfaces to these modules, and a fair amount of duplicated (or at least very similar) code for shoveling pixels around between container formats. Feels like this could be cleaned up and rationalized, both in terms of implementation and in terms of where the code lives.

As part of this we should also add format load/store tests to the test suite, as currently it's not tested at all.

Currently the only output compressed format of astc-encoder is ASTC format.
I can't find any way to convert from ASTC to KTX.

Please add support for KTX format as it's more widely used, almost all other compressor support KTX as output format.

How can I convert ASTC to KTX (without decompressed it)?

-Wclass-memaccess, which is part of -Wall in gcc 8, triggers a lot of compilation warnings like:

vectypes.h: In function ‘int4 as_int4(uint4)’:
vectypes.h:13691:24: warning: ‘void* memcpy(void*, const void*, size_t)’ writing to an object of non-trivially copyable type ‘int4’ {aka ‘class vtype4<int>’}; use copy-assignment or copy-initialization instead [-Wclass-memaccess]
  memcpy(&val1, &inp, 16);
                        ^
vectypes.h:4210:35: note: ‘int4’ {aka ‘class vtype4<int>’} declared here
 template < typename vtype > class vtype4
                                   ^~~~~~

A possible workaround is to static_cast the destination pointer of memcpy to be a void *

Hello.guys! I use my fuzzing tools. I found a Segmentation fault. I want to show my error information.

I have use ASAN. There is the error informaiton:

Encoding settings:

2D Block size: 6x6 (3.56 bpp)
3D Block size: 6x6x1 (3.56 bpp)
Radius for mean-and-stdev calculations: 0 texels
RGB power: 1
RGB base-weight: 1
RGB local-mean weight: 0
RGB local-stdev weight: 0
RGB mean-and-stdev mixing across color channels: 0
Alpha power: 1
Alpha base-weight: 1
Alpha local-mean weight: 0
Alpha local-stdev weight: 0
RGB weights scale with alpha: disabled
Color channel relative weighting: R=1 G=1 B=1 A=1
Block-artifact suppression parameter : 0
Number of distinct partitionings to test: 25 (preset)
PSNR decibel limit: 2D: 40.529411 3D: 40.529411 (preset)
1->2 partition limit: 1.200000
Dual-plane color-correlation cutoff: 0.750000 (preset)
Block Mode Percentile Cutoff: 75.000000 (preset)
Max refinement iterations: 2 (preset)
Thread count : 64 (autodetected)

ASAN:SIGSEGV
=================================================================
==206812==ERROR: AddressSanitizer: SEGV on unknown address 0x7ffdbfd918b4 (pc 0x0000005a7db7 sp 0x7ffdd7d5fe70 bp 0x7ffdbfd918b4 T0)
    #0 0x5a7db6 in get8 /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/stb_image.c:732
    #1 0x5a7db6 in get16 /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/stb_image.c:796
    #2 0x5a7db6 in get32 /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/stb_image.c:802
    #3 0x5d66b7 in parse_png_file /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/stb_image.c:2826
    #4 0x5eb086 in do_png /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/stb_image.c:2834
    #5 0x5eb086 in stbi_png_load /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/stb_image.c:2857
    #6 0x5eb086 in stbi_load_main /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/stb_image.c:537
    #7 0x60757a in stbi_load_from_file /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/stb_image.c:571
    #8 0x60757a in stbi_load /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/stb_image.c:562
    #9 0x536f14 in load_image_with_stb(char const*, int, int*) /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/astc_stb_tga.cpp:66
    #10 0x4ed8ad in astc_codec_load_image(char const*, int, int*) /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/astc_image_load_store.cpp:1328
    #11 0x404408 in main /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/astc_toplevel.cpp:2329
    #12 0x7f88f515ef44 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)
    #13 0x40a7dc (/home/lx/5_18/astc/ASAN/astc-encoder-master/Source/astcenc+0x40a7dc)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/lx/5_18/astc/ASAN/astc-encoder-master/Source/stb_image.c:732 get8
==206812==ABORTING

Then there is GDB information:

(gdb) bt
#0  0x00000000004580b3 in get32(stbi*) ()
#1  0x000000000045ea8c in parse_png_file(png*, int, int) ()
#2  0x0000000000461542 in stbi_load_main(stbi*, int*, int*, int*, int) ()
#3  0x0000000000464988 in stbi_load ()
#4  0x0000000000441d55 in load_image_with_stb(char const*, int, int*) ()
#5  0x00000000004313b6 in astc_codec_load_image(char const*, int, int*) ()
#6  0x0000000000401f10 in main ()

(gdb) x/8i $pc
=> 0x4580b3 <_ZL5get32P4stbi+387>:	movzbl (%rax),%ebp
   0x4580b6 <_ZL5get32P4stbi+390>:	shl    $0x8,%ebp
   0x4580b9 <_ZL5get32P4stbi+393>:	cmp    %r8,%rcx
   0x4580bc <_ZL5get32P4stbi+396>:	jae    0x457fbc <_ZL5get32P4stbi+140>
   0x4580c2 <_ZL5get32P4stbi+402>:	lea    0x1(%rcx),%rdi
   0x4580c6 <_ZL5get32P4stbi+406>:	mov    %rdi,0xb8(%rbx)
   0x4580cd <_ZL5get32P4stbi+413>:	movzbl (%rcx),%eax
   0x4580d0 <_ZL5get32P4stbi+416>:	add    %eax,%ebp

(gdb) i r
rax            0x7fffec5b0fce	140737158778830
rbx            0x7fffffffd650	140737488344656
rcx            0x7fffec5b0fcf	140737158778831
rdx            0x74	116
rsi            0x7fffffffd688	140737488344712
rdi            0x7fffffffd650	140737488344656
rbp            0xffffffffec5b393a	0xffffffffec5b393a
rsp            0x7fffffffbfb0	0x7fffffffbfb0
r8             0x7fffffffd708	140737488344840
r9             0xfdf7a67adda06d4e	-146465416231752370
r10            0xc9e005181b084210	-3900111676211969520
r11            0x968	2408
r12            0xb5c196af	3049363119
r13            0x2945	10565
r14            0x7fffffffd650	140737488344656
r15            0x7fffffffd5e0	140737488344544
rip            0x4580b3	0x4580b3 <get32(stbi*)+387>
eflags         0x10287	[ CF PF SF IF RF ]
cs             0x33	51
ss             0x2b	43
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0

I realize that input validation is not currently a priority; however, in case you'd ever like to work on it, I let a fuzzer have its way with astcenc -d for more-or-less twenty hours, and it shook out these 173 address boundary errors, which may be mostly related.

This link refers to a gzipped tarball containing the test cases it found. Each test case is trimmed to the smallest form that still reliably produces the crash.
http://marumie.magnifi.ca/astcenc/crashes.tgz

If any of these are architecture-specific, my test platform was AMD64.

Have a nice morning in Cambridge. :- )

texture is y-flipped when load an ASTC-encoded (.astc) file, the astc data write to a texture's memory directly. But when I try to load ktx or pvr format file using astc-encoded, texture's content is normally. Is this a bug or a implicit rule ?

I know -normal_psnr roughly maps to BC3nm and you use red and alpha channels in the shader.

To get a single channel output, is it correct to use "-ch 1 0 0 0 -esw r000-dsw r000" as the set of switches? Is the -mask switch going to be of any help here?

Thanks

The original reference codec was implemented for simplicity, so allocates worst-case memory sizes for many buffers. For example, we allocate scratch space for 216 texels per block (needed for a 6x6x6 3D block), even though the common use cases are 64 texels (8x8 block 2D block) or fewer.

Experiments show that statically reducing the max allocation sizes to 64 texels gives significant (5-10%) improvements in compressor performance due to improved memory locality, so we should look to improve this while still supporting the large block footprints when needed.

While compressing my normal map texture, it always crash with this error, seems that it happens because of Exhaustive compression mode (with other everything is good).

Currently decompressed HDR outputs are written to .htga, a simple floating point tga-like format invented solely for use by this compressor. It would be more useful if outputs were written in to something more standard like OpenEXR or Radiance format.

[C++] astc_partition_tables.cpp
astc_percentile_tables.cpp:77:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
61, 84,
^~~~~~
{ }
astc_percentile_tables.cpp:78:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
184, 141,
^~~~~~~~
{ }
astc_percentile_tables.cpp:79:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
0, 53,
^~~~~
{ }
astc_percentile_tables.cpp:80:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
percentile_arr_4x4_0, percentile_arr_4x4_1
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{ }
astc_percentile_tables.cpp:117:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
91, 104,
^~~~~~~
{ }
astc_percentile_tables.cpp:118:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
322, 464,
^~~~~~~~
{ }
astc_percentile_tables.cpp:119:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
0, 202,
^~~~~~
{ }
astc_percentile_tables.cpp:120:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
percentile_arr_5x4_0, percentile_arr_5x4_1
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{ }
astc_percentile_tables.cpp:165:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
129, 126,
^~~~~~~~
{ }
astc_percentile_tables.cpp:166:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
258, 291,
^~~~~~~~
{ }
astc_percentile_tables.cpp:167:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
0, 116,
^~~~~~
{ }
astc_percentile_tables.cpp:168:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
percentile_arr_5x5_0, percentile_arr_5x5_1
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{ }
astc_percentile_tables.cpp:220:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
165, 145,
^~~~~~~~
{ }
astc_percentile_tables.cpp:221:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
388, 405,
^~~~~~~~
{ }
astc_percentile_tables.cpp:222:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
0, 156,
^~~~~~
{ }
astc_percentile_tables.cpp:223:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
percentile_arr_6x5_0, percentile_arr_6x5_1
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{ }
astc_percentile_tables.cpp:282:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
206, 164,
^~~~~~~~
{ }
astc_percentile_tables.cpp:283:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
769, 644,
^~~~~~~~
{ }
astc_percentile_tables.cpp:284:2: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
0, 256,
^~~~~~
{ }
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [astc_percentile_tables-avx2.o] Error 1

make failed to build some targets (15 seconds)

Is there a way to get the encoder configuration about best configuration and total configurations?
configuration include:partition count & partition index ,color mode & decimation

Compressing a single pixel image as sRGB (-cs) where the pixel color is:

• (43,172,0)

... and then decompressing as sRGB (-ds) results in an sRGB+A output

• (43,171,0,254)

The 171 isn't a data precision problem in the compressed format; changing the input to (43,171,0) results in (43,170,0,254), so it looks like high values are getting rounded down.

I think the color part of this should be getting represented exactly - there are enough bits for this in a constant color block.

The alpha channel isn't sRGB encoded at all, so it's certainly odd to gain a non-1 alpha channel when there isn't any transparency in the source image.

There is a difference here between LDR and sRGB profiles - only sRGB images have this problem.

Decompressing an LDR image as LDR returns no alpha, so there is no alpha in the end point. Decompressing the same image as an sRGB image returns an alpha of 254, so the problem is in the decompressor, not the compressor.

The current command line is quite complex and includes many options which were useful for the initial compressor research, but not for a typical user. Given feedback that understanding the command line is too complex, these should be rationalized.

• Remove indirect block sizes specified as a bitrate - just support explicit block sizes.
• Remove -veryfast quality preset - it's too low quality to be useful, and -fast is already good enough in terms of speed for state space exploration.
• Remove -normal_pnsr
• Rename -normal_percep to -normalmap
• Remove -hdr_log and -hdra_log; known to give blocky output.
• Remove -force_hdr and -force_hdra - overlaps with -hdr and -hdra.
• Remove -rn - overlaps with -normalmap.
• Rename -srgb as it doesn't do the obvious thing (i.e. you should not use it to encode sRGB textures in sRGB mode).
• Clarify interaction of -c and HDR textures; even if the input is an HDR format (.hdr, .exr) you still need to specify the -hdr flag which seems odd, given that other formats tends to autodetect, and that the -c opmode is explicitly for HDR profile encoding (-cl and -cs are for LDR profile). Probably therefore rename it -ch.
• Remove -time and -showpsnr options; they should be enabled by default.
• Rename -silentmode to just -silent
• Review -b; current help docs say 0 does nothing, higher values to improve, but this makes it different to every other weighting parameter. Make it a scaling factor like other weight modifiers, but limit to 1.0 or higher?
• Review the low-level performance-quality options; the naming is inconsistent.

We've removed the legacy code left over from the research codec related to selecting non-standard block footprint sizes, but we still have percentile tables for them in astc_percentile_tables.cpp.

These redundant tables add up to a considerable percentage of the binary size at with (~20 at 8KB each), so they should get removed.

Hi,

Our fuzzer found a crash due to a heap buffer overflow on the function load_image_with_stb on the latest commit 5ff4d86 on master.

PoC_hbo_load_image_with_stb: https://github.com/strongcourage/PoCs/blob/master/astc-encoder_5ff4d86/PoC_hbo_load_image_with_stb

ASAN says:

astcenc -c $PoC /dev/null 6x6 -medium
Encoding settings:

2D Block size: 6x6 (3.56 bpp)
3D Block size: 6x6x1 (3.56 bpp)
Radius for mean-and-stdev calculations: 0 texels
RGB power: 1
RGB base-weight: 1
RGB local-mean weight: 0
RGB local-stdev weight: 0
RGB mean-and-stdev mixing across color channels: 0
Alpha power: 1
Alpha base-weight: 1
Alpha local-mean weight: 0
Alpha local-stdev weight: 0
RGB weights scale with alpha: disabled
Color channel relative weighting: R=1 G=1 B=1 A=1
Block-artifact suppression parameter : 0
Number of distinct partitionings to test: 25 (preset)
PSNR decibel limit: 2D: 40.529411 3D: 40.529411 (preset)
1->2 partition limit: 1.200000
Dual-plane color-correlation cutoff: 0.750000 (preset)
Block Mode Percentile Cutoff: 75.000000 (preset)
Max refinement iterations: 2 (preset)
Thread count : 8 (autodetected)

=================================================================
==7392==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f0083af2628 at pc 0x000000487b80 bp 0x7ffccd4bc4f0 sp 0x7ffccd4bc4e0
WRITE of size 1 at 0x7f0083af2628 thread T0
    #0 0x487b7f in load_image_with_stb(char const*, int, int*) /home/dungnguyen/gueb-testing/astc-encoder/Source/astc_stb_tga.cpp:82
    #1 0x46bff0 in astc_codec_load_image(char const*, int, int*) /home/dungnguyen/gueb-testing/astc-encoder/Source/astc_image_load_store.cpp:1328
    #2 0x49a3dd in astc_main(int, char**) /home/dungnguyen/gueb-testing/astc-encoder/Source/astc_toplevel.cpp:2329
    #3 0x7f00870ec82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #4 0x402738 in _start (/home/dungnguyen/PoCs/astc-encoder_5ff4d86/astcenc-asan+0x402738)

0x7f0083af2628 is located 0 bytes to the right of 11578920-byte region [0x7f0082fe7800,0x7f0083af2628)
allocated by thread T0 here:
    #0 0x7f0087a556b2 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x996b2)
    #1 0x462b41 in allocate_image(int, int, int, int, int) /home/dungnguyen/gueb-testing/astc-encoder/Source/astc_image_load_store.cpp:63

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/dungnguyen/gueb-testing/astc-encoder/Source/astc_stb_tga.cpp:82 load_image_with_stb(char const*, int, int*)
Shadow bytes around the buggy address:
  0x0fe090756470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe090756480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe090756490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe0907564a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe0907564b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0fe0907564c0: 00 00 00 00 00[fa]fa fa fa fa fa fa fa fa fa fa
  0x0fe0907564d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe0907564e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe0907564f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe090756500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe090756510: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==7392==ABORTING

Thanks,
Manh Dung

We ship prebuilt 32-bit binaries for Windows, which made sense in 2011 for the first releases, but the world is now increasingly a 64-bit one. All major game engine editors now require 64-bit Windows, so this ticket proposes removing 32-bit builds from the VS solution file and removal of the 32-bit binaries from master.

Hello,everyone! I use my company tools to fuzz astc.I found a crash in astc on Linux64.So I will show you some detail information to help you find the leak.

Hi,

Is astcenc able to convert 16bit/24bit png's ?
It gives a pretty clear error message when trying ("8 bit only.") but I was wondering if that was a core limitation of the algorithm or if there were ways around that.

Thanks

The validity of 3D void-extent blocks depends in part on the values of the extent. If not all-1's, any block where the minimum s,t, or r coordinate is greater than or equal to the corresponding maximum coordinate is defined as invalid, and should decode to the error color.

The code that reads the min and max s,t and r values is in error - the maxima are read from the wrong bit positions. This may lead to some void-extent blocks being marked as valid when they are not, and vice-versa. This can be verified by looking at table Table C.2.30 - 3D Void-Extent Block Layout Overview in the extension spec.

When mipmapping a 3D image with it is possible that the Z dimension in the input data collapses to one for a downscaled image because that happens to be the smallest dimension in the original data.

The current encoder will automatically convert this into a 2D image with an X x Y block size, ignoring the requested 3D block dimension. This breaks mipmapping, because all images in the chain must use the same block size.

Hi,

Our fuzzer found an assertion failure on the function stbi__create_png_image_raw (the latest commit 8cbe698 on master). This vulnerability causes the program to abort.

PoC: https://github.com/strongcourage/PoCs/blob/master/astc-encoder_8cbe698/PoC_af_stbi__create_png_image_raw
Command: astcenc -c $PoC /dev/null 6x6 -medium

Valgrind says:

astcenc: stb_image.h:4424: int stbi__create_png_image_raw(stbi__png*, stbi_uc*, stbi__uint32, int, stbi__uint32, stbi__uint32, int, int): Assertion `img_width_bytes <= x' failed.
==14817== 
==14817== Process terminating with default action of signal 6 (SIGABRT)
==14817==    at 0x5778428: raise (raise.c:54)
==14817==    by 0x577A029: abort (abort.c:89)
==14817==    by 0x5770BD6: __assert_fail_base (assert.c:92)
==14817==    by 0x5770C81: __assert_fail (assert.c:101)
==14817==    by 0x44750A: stbi__create_png_image_raw(stbi__png*, unsigned char*, unsigned int, int, unsigned int, unsigned int, int, int) [clone .isra.23] (stb_image.h:4424)
==14817==    by 0x4521E1: stbi__create_png_image (stb_image.h:4623)
==14817==    by 0x4521E1: stbi__parse_png_file(stbi__png*, int, int) (stb_image.h:4915)
==14817==    by 0x452EA1: stbi__do_png (stb_image.h:4966)
==14817==    by 0x452EA1: stbi__png_load (stb_image.h:4996)
==14817==    by 0x452EA1: stbi__load_main(stbi__context*, int*, int*, int*, int, stbi__result_info*, int) [clone .isra.32] (stb_image.h:1008)
==14817==    by 0x4578A3: stbi__load_and_postprocess_8bit(stbi__context*, int*, int*, int*, int) (stb_image.h:1115)
==14817==    by 0x458523: stbi_load_from_file (stb_image.h:1229)
==14817==    by 0x458523: stbi_load (stb_image.h:1219)
==14817==    by 0x45B53C: load_image_with_stb(char const*, int, int*) (astc_stb_tga.cpp:65)
==14817==    by 0x42FDC5: astc_codec_load_image(char const*, int, int*) (astc_image_load_store.cpp:1283)
==14817==    by 0x4617A7: astc_main(int, char**) (astc_toplevel.cpp:2266)

Thanks,
Manh Dung

I am using the test mode you describe in the wiki
*./astcenc -t /images/example.png /images/example-decompressed.tga 6x6 -medium
(to test what compression and decompression are like),
but i cannot view the resulting image - compression/de-compression succeeds as i see from log. I am using the binary provided on OS X, tried tga/png as the resulting image

I am able to use the binary ./astcenc as i am not able to get the Binary repository.
Kindly help.

Hi,

Our fuzzer found a stack buffer overflow on the function build_huffman (the latest commit 5ff4d86 on master).

PoC: https://github.com/strongcourage/PoCs/blob/master/astc-encoder_5ff4d86/PoC_sbo_build_huffman

ASAN says:

astcenc -c $PoC /dev/null 6x6 -medium
Encoding settings:

2D Block size: 6x6 (3.56 bpp)
3D Block size: 6x6x1 (3.56 bpp)
Radius for mean-and-stdev calculations: 0 texels
RGB power: 1
RGB base-weight: 1
RGB local-mean weight: 0
RGB local-stdev weight: 0
RGB mean-and-stdev mixing across color channels: 0
Alpha power: 1
Alpha base-weight: 1
Alpha local-mean weight: 0
Alpha local-stdev weight: 0
RGB weights scale with alpha: disabled
Color channel relative weighting: R=1 G=1 B=1 A=1
Block-artifact suppression parameter : 0
Number of distinct partitionings to test: 25 (preset)
PSNR decibel limit: 2D: 40.529411 3D: 40.529411 (preset)
1->2 partition limit: 1.200000
Dual-plane color-correlation cutoff: 0.750000 (preset)
Block Mode Percentile Cutoff: 75.000000 (preset)
Max refinement iterations: 2 (preset)
Thread count : 8 (autodetected)

=================================================================
==15504==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff00ef8040 at pc 0x0000004ae78a bp 0x7fff00ef43b0 sp 0x7fff00ef43a0
WRITE of size 1 at 0x7fff00ef8040 thread T0
    #0 0x4ae789 in build_huffman /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:1009
    #1 0x4b9af3 in process_marker /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:1478
    #2 0x4ba85d in decode_jpeg_header /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:1609
    #3 0x4c2440 in decode_jpeg_header /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:1605
    #4 0x4c2440 in decode_jpeg_image /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:1625
    #5 0x4c2440 in load_jpeg_image /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:1816
    #6 0x4c2440 in stbi_jpeg_load /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:1912
    #7 0x4d55be in stbi_load_main /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:536
    #8 0x4dc009 in stbi_load_from_file /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:571
    #9 0x4dc009 in stbi_load /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:562
    #10 0x4877b9 in load_image_with_stb(char const*, int, int*) /home/dungnguyen/gueb-testing/astc-encoder/Source/astc_stb_tga.cpp:66
    #11 0x46bff0 in astc_codec_load_image(char const*, int, int*) /home/dungnguyen/gueb-testing/astc-encoder/Source/astc_image_load_store.cpp:1328
    #12 0x49a3dd in astc_main(int, char**) /home/dungnguyen/gueb-testing/astc-encoder/Source/astc_toplevel.cpp:2329
    #13 0x7fe73da5082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #14 0x402738 in _start (/home/dungnguyen/PoCs/astc-encoder_5ff4d86/astcenc-asan+0x402738)

Address 0x7fff00ef8040 is located in stack of thread T0 at offset 14704 in frame
    #0 0x4c1bbf in stbi_jpeg_load /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:1909

  This frame has 5 object(s):
    [32, 64) 'coutput'
    [96, 224) 'data'
    [256, 384) 'data'
    [416, 608) 'res_comp'
    [640, 14704) 'j' <== Memory access at offset 14704 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow /home/dungnguyen/gueb-testing/astc-encoder/Source/stb_image.c:1009 build_huffman
Shadow bytes around the buggy address:
  0x1000601d6fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000601d6fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000601d6fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000601d6fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000601d6ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1000601d7000: 00 00 00 00 00 00 00 00[f4]f4 f3 f3 f3 f3 00 00
  0x1000601d7010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000601d7020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000601d7030: f1 f1 f1 f1 00 00 00 00 f2 f2 f2 f2 00 00 00 06
  0x1000601d7040: f2 f2 f2 f2 04 f4 f4 f4 f3 f3 f3 f3 00 00 00 00
  0x1000601d7050: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==15504==ABORTING

Thanks,
Manh Dung

Decompression from .astc files, e.g.:

./astcenc-avx2 -dl ../out.astc ../out.png

... is failing with:

Failed to load image ../out.astc
Reason: unknown image type

Compression (-c) is working for all types, and round-tripping (-t) is working for all types, so this just seems to impact decompression.

Is there any reason to convert symbolic compressed block to physical compressed block and back again at the end of compress_symbolic_block()? Will it differs from original scb?

Anyway, right after that scb->pcb->scb convertion it would be again converted scb->pcb in case of !pack_and_unpack , and just decompressed otherwise. Is it safe just to delete those two lines

Hi,

Recently I've been working on creating an open source CLI tool for texture compressing in all common formats (ASTC, ETC, PVR and S3TC) for WebGL. I've looked in the source code but I haven't been able to find an option to generate mipmaps with Astcenc. Is there a way to pass a flag to enable mipmapping and if not, is this something on the roadmap? I have noticed that PVRTexTool is able to generate mipmaps but unfortunately I don't think I have access to the source code.

If you are interested, the compression tool can be found here: https://github.com/TimvanScherpenzeel/texture-compressor/blob/master/lib/compressWithASTC.js

Many thanks in advance,

Tim

The license file in the repo doesn't seem to be one of the standard Open Source License (like Apache or GPL).

Could you please clarify if it permits the usage of the binaries in this repo for converting images from PNG to ASTC format? And using the resulting ASTC textures in a commercial application - essential not shipping the binaries but rather the ASTC output generated by it

In section 3.16 "Data Size Determination", the sample code has the line:

config_bits = 24 + 3*num_partitions;

but referring to table 15 "Table 15 –Multi-Partition Color Endpoint Modes" shows that there are 25 bits of data in locations 24:0, so the line should be:

config_bits = 25 + 3*num_partitions;

This is touted in all of the talks and discussions about ATSC, but it doesn't help if the encoders don't generate the data. Does the encoder support these modes? I can sometimes get 3 channels from the encoder, but most often it defaults to 4 channels. The encoders I've found all use 3 or 4 channels, but isn't the compression considerable with 1 or 2 if you have a 1 or 2 channel texture? I'm looking for the equivalent of BC4 and BC5, which ATSC was supposed to encompass.

The current 2.0 prototype branch is statically enabling various levels of SSE support at build-time, and we'd like the option to use SSE4.2, POPCNT, and AVX2 features for critical data paths. We can assume that SSE2 is always available (mandatory for x86-64), but anything higher than that should be dynamically selected at runtime to avoid the need for per-target builds.

For testing purposes it must be possible to forcefully select a specific path; either at build time or runtime, so we can ensure good test coverage.

A common developer need is to compress entire mipmap chains. Currently astcenc cannot do that; you have to compress single mip levels and then repack them as a post-process activity. This issue looks at adding mipmap support for compression and decompression.

For compression workflows:

• Mipmap generation
• Mipmap compression
• Storage as a KTX format output.

For decompression workflows:

• Storage as any uncompressed format, with a "_level" postfix.

Mipmap generation

The most interesting part of this issue is the mipmap generation, as there are multiple aspects to consider.

What filters to expose?

• Weighted box filer with bilinear sample weighting for odd sizes.
• Box with random sample (useful for e.g. noise textures, where you want to preserve dynamic range and frequency and not low-pass filter the noise)
• N-sample lanczos.
• N-sample kaiser.
• For windowed filters support filtering wraps for tiled textures.

What tweaks to apply?

• Always downsample from original vs downsample from level N-1? In principal downsampling from the original each time avoids accumulating error from each level's approximation for e.g. lanczos filter windows.
• sRGB - filter in linear space to preserve gamma intensity.
• Normals - renormalize each level.

In astc_toplevel.cpp, the call to CreateThread at line 50 is missing its matching CloseHandle call, causing the created thread objects to leak. I propose adding the missing call in pthread_join, like so:

	int pthread_join(pthread_t thread, void **value)
	{
		WaitForSingleObject(thread, INFINITE);
		CloseHandle(thread); // <= Missing
		return 0;
	}

The current codec always uses DX/Vulkan coordinate system (0,0 top left), which is different to the convention used by OpenGL and OpenGL ES and container formats such as KTX (0,0 bottom left).

Users can manually flip first, but it's painful, so we should support a command line option to flip the input image on load, and on store for decompression passes.

I'm getting compilation -Werror=shadow errors on GCC 7.4.0

Analysis shows that we get increased cache pressure in compute_lowest_and_highest_weight because these two:

float error_from_forcing_weight_down[60];
float error_from_forcing_weight_either_way[60];

... are stored as separate arrays and accessed inside a hot loop. We always access the same index in each array at the same time, so we should store this as a single array of structs to reduce cache pressure.

The compare mode doesn't cope with missing command line parameters.