Repository navigation
Add audio rendering support (continued) - #65
Merged
Merged
Conversation
Add a complete audio pipeline parallel to the existing video pipeline: - Decode audio from source media files using FFmpeg (libavcodec + libswresample) - Walk OTIO timeline audio tracks: clips, gaps, transitions (crossfade), time warps, track mixing - Extract audio from video clips that contain audio streams - Mux audio alongside video in output containers (MP4, MOV) - Write audio to separate files via -afile option New CLI options: -acodec, -arate, -achannels, -afile, -no_audio New files: - AudioBuffer.h: interleaved float32 audio buffer type - FFmpegAudioRead.h/cpp: audio decoder using FFmpeg + swresample - AudioGraph.h/cpp: OTIO timeline walker for audio (mirrors ImageGraph) - FFmpegAudioWrite.h/cpp: standalone audio file writer Modified files: - FFmpeg.h/cpp: AudioCodec enum (pcm_s16le, pcm_s24le, pcm_f32le, flac, aac) - FFmpegWrite.h/cpp: audio+video muxing in combined writer - App.h/cpp: CLI options and audio render loop integration - TimelineAlgo.h/cpp: getAudioClips() helper - CMakeLists.txt: new sources + libswresample linking - BuildFFmpeg.cmake: AAC codec in minimal build Signed-off-by: Daniel Weinmann <daniel@seasoned.cc> (cherry picked from commit 3b9bf3c)
The render loop asked for the same number of samples every frame, the sample rate over the frame rate rounded to an integer. At 24 fps that is exact. At 29.97 it is 1602 where the frame is 1601.6 samples long, so the audio ran long by the difference every frame, and the reader, which compared its own clock against the frame time and seeked whenever they disagreed, seeked on every frame: a flush, a resampler reset, and a backward seek to a packet boundary whose leading samples were then played again. On a 440 Hz tone that was 86 audible jumps in ten seconds. The count for a frame is now the difference between its rounded start and end positions in samples, so the counts alternate and sum to the timeline's length. The reader keeps its position in samples, seeks only when asked for somewhere it is not, and after a seek drops the samples before the target so the output picks up at the sample asked for rather than at the packet the seek landed on. Verified against a tone at 24, 29.97 and 59.94 fps, with a clip that starts mid-file, a gap, a transition and a second audio track: the output matches the source sample for sample. Signed-off-by: Darby Johnston <darbyjohnston@yahoo.com>
FFmpeg 7.1 deprecated the codec's sample format list in favor of avcodec_get_supported_config(), which the writers now ask. The unused getAudioClips() goes, and the README says what the audio options do. Signed-off-by: Darby Johnston <darbyjohnston@yahoo.com>
The minimal build carries the codecs that ship without a patent license, and AAC is not one of them. ALAC is: Apache-licensed, encoded and decoded by FFmpeg itself, and played by QuickTime and most everything else inside a MOV, so it is the compressed choice there. AAC stays in the codec list for a full FFmpeg build; the options shown are the encoders the build has. Signed-off-by: Darby Johnston <darbyjohnston@yahoo.com>
A video clip's media may carry audio, and it was mixed in alongside the audio tracks. Now it is left alone: the timeline says what is heard the way it says what is seen, and a timeline with no audio tracks writes a movie with no audio stream. That also drops the probe that opened every video clip's media on start-up to find out whether it had audio. Signed-off-by: Darby Johnston <darbyjohnston@yahoo.com>
7 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The original PR this work is based on: #59
Some additional notes:
Assisted-by: Claude:claude-fable-5-1 [patch] [debugging]