aomediacodec / av1-avif Goto Github PK
View Code? Open in Web Editor NEWAV1 Image File Format Specification - ISO-BMFF/HEIF derivative
Home Page: https://aomediacodec.github.io/av1-avif/
License: BSD 2-Clause "Simplified" License
AV1 Image File Format Specification - ISO-BMFF/HEIF derivative
Home Page: https://aomediacodec.github.io/av1-avif/
License: BSD 2-Clause "Simplified" License
Probably the same profile or lower than the associated image sequences
Hi all --
In having some discussions with @wantehchang, we were curious if there was an official fallback value for the matrixCoefficients
enum, when it is unspecified. Should we fallback to BT709 or BT601?
The current libavif falls back to BT709 coefficients (1) if it is unspecified, but I believe traditional video paths (ffmpeg) tend to fallback to BT601 (6). Is there anywhere in the HEIF or AVIF standard we can use that'd guide us here?
I'm working on AVIF support for Firefox, and the sample images in https://github.com/AOMediaCodec/av1-avif/tree/master/testFiles have been very helpful, but given the dearth of renderers in the current ecosystem, it's difficult to have confidence in the correctness of output.
For example, I'm pretty confident that Netflix/avif/cosmos_frame01000_yuv420_8bpc_bt709_g22_qlossless.avif is rendering correctly based on this screenshot of my prototype rendering in browser:
But I assumed the same frame in 10-bit (cosmos_frame01000_yuv420_10bpc_bt2020_hlg_qlossless.avif) should look essentially the same, causing me to doubt my output when it did not:
However, the 10-bit version rendered by VLC nightly looks quite similar:
As does the version rendered by Windows:
Though I have my doubts about relying on the correctness of other renderers based on the Windows output for cosmos_frame01000_yuv444_8bpc_bt709_g22_qlossless.avif:
For comparison here's that same input as rendered by Firefox:
If it would be possible to see what the input was, it would be a great help in validating the correctness of the output.
What are the right levels for the profiles?
What should we do with profile 31 (unrestricted)?
There is value in having conformance files that have odd frame sizes. Currently, all test image files have a frame size that is divisible by 8.
By default, AV1/libaom will internally use a frame size aomImage->w, aomImage->h
that is a multiple of 8(?). However, it also exposes the to-be-displayed framesize aomImage->d_w, aomImage->d_h
and the to-be-rendered framesize aomImage->r_w, aomImage->r_h
. (I don't know if there's an important difference between d_w
and r_w
)
Adding the conformance files makes sure that the right dimensions are referenced for the output. I suggest the following resolutions:
Maybe we want to also include a derived grid image with the "sub images" having odd resolutions?
According to #68, it is valid AVIF file whose bit-depth of primary image and that of alpha plane are different.
(For example, AVIF file with 8bpc primary image and 10bpc alpha plane).
However, as @joedrago says
I think this creates a lot of pain points, corner cases, and complexity for very little gain, if I'm being honest.
AOMediaCodec/libavif#86
I really agree with him. In fact, I support those image in cavif and davif, but it made the code a little bit complex and make very difficult to write test (because there are many combinations to test...).
How about add additional constraints, such as "The bit-depth of the primary image and that of alpha image should be the same".
Current spec is not clear about the status of scalable images. An item is said to be a AV1 Sync Sample but this is allowed to contain multiple frames from different layers. This should probably be made explicit and maybe restricted in profiles
I just worked on my first PR and both my local bikeshed
copy as well as the Curl API https://tabatkins.github.io/bikeshed/#curl generate an docs/index.html
with a lot of changes vs the status-quo.
If would be great to have either (or both) of the following
Hi. I am currently supporting AVIF/AVIFS images in Chrome, and implementing AVIFS in cavif and davif.
According to Wan-Teh, a chrome developer, it is required to all frames in an animated image to be the same size in Chrome(widthxheight).
I think it's reasonable constraints.
FYI: In WebP, it looks all frame have the same size.
https://github.com/webmproject/libwebp/blob/master/src/webp/mux.h#L277-L312
When HEVC is used in a HEIF image sequence, non-IDR frames are allowed. I would expect the same to be allowed with AV1.
The spec should define file extensions and MIME types. It currently does not.
In this spec it says that the AV1CodecConfigurationBox shall not contain any Sequence Header OBUs. But in the AV1 ISOBMFF mapping spec, it says the opposite. In that spec, the Sequence Header is mandatory and it shall be the first one.
This discrepancy is problematic because ideally the AV1F spec should just be a superset of the AV1-ISOBMFF mapping spec. It should not define things that directly conflict with the AV1-ISOBMFF mapping spec.
It gets particularly confusing with the AV1F file contains an image sequence. As defined in the HEIF spec, an image sequence is stored in a 'pict' track, which is defined just like a video track, except that the HandlerType is 'pict'. But an AV1 video track would require sequence Sequence Header OBUs in the AV1CodecConfigurationBox. The result is that within the same AV1F file, there will be conflicting requirements for the AV1CodecConfigurationBox depending on whether AV1CodecConfigurationBox is used for a still image vs. a picture track. This is not a good situation, IMO.
Hi.
I would like to add link for AVIF polyfill to the wiki. Seems like it's locked for non-collaborators so could you please put it on Implementations page?
Thank you.
Motivation
Allowing 4:4:4 Chroma sub-sampling is critical to many application areas (e.g. graphical images or photographic content with overlays). AVIF should allow for storing image information with 4:4:4 chroma sub-sampling.
Comparison with other formats:
Main 4:4:4 Intra
profile (see https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding#Version_2_profiles)Proposed change
Change required reader profile to "High" or higher to allow 4:4:4 bit streams.
From the AV1 spec https://aomediacodec.github.io/av1-spec/av1-spec.pdf, p635:
The Main profile supports YUV 4:2:0 or monochrome bitstreams with bit depth equal to 8 or 10. The Highprofile further adds support for 4:4:4 bitstreams with the same bit depth constraints. Finally, the Professional profile extends support over the High profile to also bitstreams with bit depth equal to 12, and also adds support for the 4:2:2 video format
Other
I'd suspect that 444 would play particularly well to the strength of the "chroma from luma prediction" in AV1: https://people.xiph.org/~xiphmont/demo/av1/demo1.shtml
See related issue: #15
In the Implementations section, please update the libavif link to point here:
https://github.com/AOMediaCodec/libavif
(the official location now)
Hi, I'm currently working on supporting AVIF and AVIFS, and I have a question.
How can I determine the repetition count (infinite loop, fixed time loop, no loop) for the AVIFS image sequence?
I checked ISOBMFF, MIAF, HEIF, and av1-isobmff, but I can't find how many times the avifs should be repeated.
(It looks sample_delta
field in stts
box determine the display time, but it is unsigned, so I think it can't be used to represent infinite loop.)
Would be nice to have a conformance file with one or more embedded thumbnails
Based on the AOM call and discussion on the Chrome bug for AVIF, there may be interest in allowing a constrained decoder to only support AVIF in fragmented MP4, as opposed to requiring unfragmented MP4 support.
This could be similar to the way the decoder constraint for reduced / intra-only tooling is handled. Not sure if a flag is required or if the file can simply be inspected.
HEIF 2nd Edition Amd1 specifies new property types:
AVIF current refers to MIAF for the first 2 (https://aomediacodec.github.io/av1-avif/#other-item-property) and does not mention the third one.
For example derived image items.
One of the primary concerns for adopting new image formats is if they are patent encumbered or have non-compatible licenses with what businesses need. The current documentation does not mention patents or licenses.
Even if this is a decision that is planned to be made after the format becomes more stable, that should be mentioned.
For easy integration and experimentation a reference library and command line tools for AVIF should be provided somewhere. It could have a similar structure to webp, which provides a libary libwebp
and command line tools cwebp
and dwebp
.
I would suggest the following 6 packages to be created:
.avif (single frame)
libavif
: Encoding and decoding .avif librarycavif
: Command line .avif encoding tooldavif
: Command line .avif decoding tool.avifs (multi frame / animation)
libavifs
: Encoding and decoding .avifs librarycavifs
: Command line .avifs encoding tooldavifs
: Command line .avifs decoding toolIn this configuration, the avif tools could be very lightweight, since only the AV1 intra encoding and decoding tools are needed. the avifs tools extend avif with the inter en- and decoding code. cavifs
could replace both gif2webp and img2webp.
As discussed during today's AOM Image call:
cavid avif encoder, using libaom directly.
https://github.com/AOMediaCodec/av1-avif/wiki
Our encoder is "cavif (Coding AVIF)". Could you fix a typo?
From pretty early on in AV1 development it was assumed that it would replace VP8 in WebP, so why not do that. There are lots of people (including myself) who are not a fan of the trend of creating a new container for every use case and every combination of codecs, especially when it is just a limited subset of one that already exists (why so many people initialy disliked WebM). The way I see it it should either be based on WebP in which case call it WebP or it be based on HEIF in which case call it HEIF, this should be a specification for the "carriage of AV1" in an already existing container (like the av1-isobmff specification is doing).
Edit: Actually why not both.
Edit 2: The simple version for those that are confused; use a common file extension so regular users don't get scared and use some online "converter" to "convert" to jpeg (what the vast majority of people think pictures should be).
HEIF 2nd Ed Amd 1 defines a lot of new item properties, which can be categorized as metadata properties:
We should determine if we want to update the spec to mention/mandate some of them.
Section 6.1: "The FileTypeBox should declare at least one profile that enables decoding of the primary image item, or one of its alternates. The profile should allow decoding of any associated auxiliary images, unless it is acceptable to decode the image item without its auxiliary images."
How can we know if it is "acceptable to decode the image item without its auxiliary images"?
HEIF 2nd Ed Amd 1 defines a set of new properties for the purpose of distinguishing images in a group (bracketing or panorama):
We should determine if we want to do something special with these properties.
MIAF currently specifies:
For RGB encoding, sRGB should be assumed as indicated by colour_primaries equal to 1, transfer_characteristics equal to 13, matrix_coefficients equal to 0, and full_range_flag equal to 1.
In this context, MPEG/ITU issued a "Defect report for Video CICP (ISO/IEC 23091-2)" on 2019-10-11, N18912.
This raises 2 questions (at least):
Just like other codec mappings:
heic/heix vs hevc/hevx
avci vs avcs
Since the format allows lossless encoding, it should have these to distinguish it from lossy images.
In this case of course the container should only allow such image parameters that are in line of the image being lossless for the file to be valid. (Not sure what else could be done to guarantee that the image is lossless beside a checksum/hashcode)
Otherwise the image should be rejected.
Lossless images of course have some distinct use cases. And it helps if the user can tell whether he acquired a lossless image, or saved a lossless image just by choosing the lossless AVIF variant. One expects them to be editable and re-encodable without quality loss.
AVIF is based on HEIF, which means it can also contain audio (sic!) and subtitles. Why not make a new format and name it like AOM (AOMedia)? AV1 for video, AA1 for audio, AS1 for subtitles. Yes, you will need to re-invent, but you will immediately make a normal format, which will not be embarrassing to let in the masses. There will be a single ecosystem that does not oblige to support all that obsolete garbage from the WEBM.
Wishes:
Let's do better than HEVC, PNG, JPEG, GIF, FLIF, ASS (SSA) and any audio format. We will reconsider everything that we have done before, we will do better. And we will spread this format by all means, because AOMedia contains companies that everyone has heard.
HEIF 2nd Edition AMD 1 specifies the ability to store predicted images (inter frames) as items. The item type of such image is the same as for intra images. The way to detect that an image is predicted is by search for an essential property of type pred
.
We should determine if we want to have support for AV1 inter frames or not. If so, should we wait for a possible MIAF update?
We should raise awareness of implementers that they should not process image items containing essential properties that they don't understand.
https://github.com/AOMediaCodec/av1-avif/wiki#demuxers--players makes no mention of nightly VLC, but it is able to display (albeit with some issues) both AVIF and AVIS files.
I'd have just submitted a PR, but for changes to a wiki, GitHub doesn't seem to have a simple workflow.
.avif
file extension is very misleading. It looks very similar to .avi
, and a user may expect that it should be a video, but it is not. Why .webp
extension is not utilized? If .webp
can't be used because the internal container has been changed (from RIFF to ISOBMFF), we could utilize .webi
(Web Image) or .aomi
(AOMedia Image), for example.
Hi.
I would like to add link for cavif and davif to the wiki.
They are used to generate test images I prepared and decode them again to compare to the original images.
While libavif supports many encoders and decoders, cavif and davif just uses libaom to encode and dav1d to decode. Instead, cavif and davif let you control each encoder and decoder in detail.
Thank you.
The spec currently treats picture tracks and video tracks as they are the same. I think a picture track can be a sequence of burst images, for example, and would be generated "picture module" of a camera. But a video track is just a plain video track that would be generated by the "video module".
So, while it would be OK to have special restrictions on "picture tracks" (for example, I-frames only), "video tracks" should be mostly unrestricted as they would be created by the camera's normal video module.
As discussed last week, we should consider whether we want to define a brand to identify when image sequences are intra-only coded.
As of today, you have to inspect the track and detect that all samples are 'sync' samples. This might be a bit late and could result in files being downloaded while not being playable, if the decoder only supports intra.
It might be better defined also within MPEG, as this is codec-independent.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.