vade / ffmpegmediaextension Goto Github PK

Use macOS Sequia's new Media Extensions to allow playback of FFMPEG supported media via AVFoundation

C 0.13% Swift 5.25% Objective-C 94.62%

ffmpegmediaextension's Introduction

FFMPEGMediaExtension

This is an attempt to build a macOS 15.0 MediaExtension for libAV* which in theory would allow for native AVFoundation container parsing and decode.

Since MediaExtensions are shipped in a host app, this has also provides a Host app.

We have basic frame decoding working!

But there is a ton left to do!

Asset / Track Metadata
GOP Support
Additional Codecs

Goal

Provide additional media format playback support for any macOS app which leverages AVFoundation / Core Media as its underlying playback / decode engine

Finder
Quicklook
Quicktime Player
Any other app which opts in by calling:
- MTRegisterProfessionalVideoWorkflowFormatReaders() for format parsing support
- VTRegisterProfessionalVideoWorkflowVideoDecoders() for additional decoder support

Requirements:

MacOS 15 Beta 4 +
XCode 16 Beta 4 +

Building:

If you want to build FFMPEG libAVFormat and libAVCodec:

brew install automake nasm pkg-config

FFMPEG is installed as a submodule - so ensure you git checkout the submodule and init

cd ffmpeg

./configure --prefix=/usr/local --enable-static --disable-shared --disable-programs --disable-doc --disable-avdevice --disable-avfilter --disable-postproc --disable-swscale --disable-swresample --arch=arm64 --disable-avfoundation

make -j$(sysctl -n hw.ncpu)

This will build you the statically linked libAVFormat.a and libAVCodec.a libries

Design

MEFormatReader (LibAVFOrmatReader)

MEVideoDecoder

Not yet started
This means today we only add container support, not yet additional codecs
Stay tuned

Todo:

Strategy

We Eventually require 2 Bundle targets that are Media Extensions.

A Media Extention that implements MEFormatReader, which will allow container parsing via libavformat and provide container metadata, track metadata, and compressed sample metadata.
A Media Extension that impelments MEVideoDecoder - a Video Decompressor which will allow for codec decompression via libavcodec and provide pixel buffers from appropriate samples.

FFMPEGs library more or less follows similar design patterns to AVFoundation. The following isnt totally accurate, but more or less gets the point across:

CMTime <~> AVRational
CMMediaType <~> AVMediaType
CMFormatDescription <~> AVCodecParameters
CMSampleBuffer <~> AVPacket
CMBlockBuffer <~> AVBuffer
CVPixelBuffer <-> AVFrame (decoded video)
AVAsset <~> AVFormatContext
AVAssetTrack <~> AVStream
AVSampleCursor <~> AVIOContext

Leverage AVIOContext to allow a MEByteSource to driven by a AVFormatContext to get us container parsing and metadata

Running Notes:

Weirdly MediaExtension headers imply macOS 14 supported?
entitlements for Media Extensions

ffmpegmediaextension's People

Contributors

Stargazers

Watchers

ffmpegmediaextension's Issues

<<<< AudioFormatDescription >>>> figAudioFormatDescriptionCreateFormatList: Unexpected error: AudioFormatGetProperty(FormatList) returned an error(2003329396) despite AudioFormatGetPropertyInfo(FormatList) succeeded

Determine Sync metadata for samples from libavformat / libavcodec without decoding

It seems like our Media Format Extension is super close, but we need to understand how to provide the MESampleCursor implementation with some metadata about key frame, sync and dependencies

AVSampleCursorSyncInfo
AVSampleCursorDependencyInfo
MEHEVCDependencyInfo
decodeTimeOfLastSampleReachableByForwardSteppingThatIsAlreadyLoadedByByteSource
samplesWithEarlierDTSsMayHaveLaterPTSsThanCursor
samplesWithLaterDTSsMayHaveEarlierPTSsThanCursor

Implement libav* object reference counting correctly and memory management stuff

I've been sort of YOLO'ing it - should pay a bit more attention here.

av_packet_ref
av_packet_unref

default 12:50:23.966175-0400 LibAVExtensionHost <<< PluginFormatReader >>> wrapperSampleCursor_StepInPresentationOrderAndReportStepsTaken: Bad response from plugin. Plugin sample cursor took +1 steps when -1 were requested. Now at pts: {100/1000 = 0.100}

Support GOP Codecs

So while we have our first frame, this only works right now for I frame codecs - ie we aren't properly supporting other frame types just yet.

Playback appears to be requiring some additional metadata we aren't supplying (see #6 )

But we also will need to update the function

- (void)loadSampleBufferContainingSamplesToEndCursor:(nullable id<MESampleCursor>)endSampleCursor completionHandler:(void (^)(CMSampleBufferRef _Nullable newSampleBuffer, NSError * _Nullable error))completionHandler

To support the end sample cursor and 'accrue' samples in the CMBlockBuffer and populate the sample timing info.

The header docs for CMSampleBufferCreateReady has some good insight there.

Determine if Dispatch Queue is in fact necessary

While having a sexy multi threaded thingy-majigy sounds cool, because we have multiple objects handling the same underlying 'asset' and sharing pointers for format contexts, streams, codec parsers managing that seems stupid and error prone.

Lets see if we can KISS.

Next Steps - Populate Metadata

From what I can tell, we also need to actually implement some metadata reporting

<<<< MEFormatReaderWrapper >>>> objcWrapperPluginFormatReader_CopyProperty signalled err=-12784 (kCMBaseObjectError_PropertyNotFound) (can not find metatdata info) at ExtensionFormatReaderWrapper.m:535


<<<< AVAssetInspectorLoader >>>> -[AVFigAssetInspectorLoader loadValuesAsynchronouslyForKeys:keysForCollectionKeys:completionHandler:]: called (self: 0x600001dac310, keys: (
    commonMetadata,
    hasProtectedContent,
    composable,
    streaming,
    availableMetadataFormats,
    metadata,
    compatibleWithAirPlayVideo,
    availableMediaCharacteristicsWithMediaSelectionOptions,
    availableChapterLocales,
    maximumVideoResolution,
    availableVideoDynamicRanges,
    tracks,
    assetCache
), keysForCollectionKeys: (null), handler: non-nil)

<<<< AVAssetInspectorLoader >>>> -[AVFigAssetInspectorLoader loadValuesAsynchronouslyForKeys:keysForCollectionKeys:completionHandler:]: Calling FigAssetLoadValuesAsyncForProperties for properties (
    availableMediaCharacteristicsWithMediaSelectionOptions,
    "assetProperty_QuickTimeUserData",
    "assetProperty_HLSMetadata",
    "assetProperty_ID3Metadata",
    "assetProperty_CommonMetadata",
    "assetProperty_ISOUserData",
    Chapters,
    "assetProperty_AVCHDMetadata",
    "assetProperty_FormatSpecificMetadata",
    "assetProperty_QuickTimeMetadata",
    "assetProperty_Tracks",
    "assetProperty_AvailableVideoDynamicRanges",
    "assetProperty_AssetType",
    "assetProperty_MaximumVideoResolution",
    "assetProperty_CompatibleWithAirPlayVideo",
    "assetProperty_iTunesMetadata",
    "assetProperty_AvailableMetadataFormats",
    "assetProperty_ChapterGroupInfo",
    "assetProperty_ProtectedStatus"
)