Apparently FFmpeg probes WAV files for 32 packets before returning the
stream info. With the default of 19200 bytes per packet, this could end
up waiting for up to 5 seconds of data to be downloaded.
Setting the max_size option only affects WAV and W64 in our build of
libavformat. No other formats we care about will be affected.
Unfortunately, this necessarily involves parsing codec frames to get
their durations. Ogg's granule positions indicate the last sample of
the last complete frame of the page, so the navigator has to determine
the durations of every packet in the page to offset it back to its
start.
Vorbis is an especially involved codec to determine that for, since it
has initial data defining a mapping of indices to big or small block
sizes that has to be held onto, and that mapping is preceded by a bunch
of conditionally parsed bits. Also, a frame's block size calculation
involves the block size of the previous frame for an overlap add. To
avoid bringing that complexity directly into LibMedia, we delegate the
parsing/calculation to FFmpeg's av_vorbis_parse functions.
A new MP3Navigator class is added, which determines timestamps for byte
positions by resyncing to a frame and then interpolating between known
points on either side. The known points start out as the first frame's
position in the file at timestamp 0, and EOF at the timestamp for
FFmpeg's file duration estimate. New buffered ranges are interpolated
between those two points, but also between the end of a prior range and
the start of the next.
Since MP3 can have variable bitrate without declaring it in the file
header, we have to allow buffered ranges to shift forward as new data
arrives to make room for underestimated durations. This is done for all
ranges following the first that has been appended to, keeping the start
of the current range consistent, so that subsequent seeks within that
range remain consistent.
Seeking is also implemented within the navigator to ensure that the
byte<->timestamp mapping is consistent and the buffered ranges begin
exactly where the seek landed.
We could end up seeked within AVIOContext's internal buffer without it
checking whether new data could be buffered without hitting EOF again,
so we could get an unexpected read error.
Add a new FLACNavigator class that can scan for frames both forward and
backward from a byte offset to get the exact buffered ranges. The reads
are done in chunks, which keeps each call pretty cheap.
The ranges themselves are resolved by its ScanningContainerNavigator
base class, which reconciles cached ranges against the incoming ones to
avoid repeating work unnecessarily. With those combined optimizations,
the calls normally take under 1 microsecond.
The default mode is to grab the index and use it to determine the start
and end of the buffered ranges. This works well for MP4.
For WAV, the index is incomplete, so instead use a much simpler method,
just determining the ranges based on the constant bitrate of the file.
This is implemented through a new virtual ContainerNavigator class.
Previously, we weren't too consistent about the definition of frame and
sample when it relates to raw audio data. This brings all the usages in
the context of raw data in line (hopefully), with samples referring to
a single PCM value, and frames referring to the multiple samples that
make up an instant's audio across all channels.
This is the new way of handling fast seeks. Instead of delegating the
logic all the way down the pipeline to the decoder thread's seek
handler, we can just determine the timestamp we want to seek to ahead
of time.
This will allow reuse of the allocations, instead of reallocating a
FixedArray for every block that changes size. Generally, it will be
possible to reuse AudioBlock memory throughout most of the pipeline
at least while in a single process.
Decoded video frames should own their planar YUV data and color space
directly. Keeping that storage behind ImmutableBitmap gave a
still-image abstraction media-specific behavior and made calls like
bitmap() potentially allocate and convert a whole video frame.
Move YUV ownership into Media::VideoFrame, where the lifetime naturally
follows media playback, and remove the YUV-backed mode from
ImmutableBitmap. This commit intentionally keeps the visible Web paint
path on ExternalContentSource by converting the current frame back to
an ImmutableBitmap where Web still expects one.
Callers that need pixels now ask the frame to convert explicitly. That
preserves behavior for canvas and bitmap consumers while making the
expensive YUV-to-pixel path visible at the call site instead of
hiding it behind ImmutableBitmap::bitmap().
Video frames are about to be shared between the decoder, the data
provider, the display sink, and Web painting code. Passing them by
value keeps ownership tied to the old bitmap-shaped pipeline and makes
later lifetime changes harder to reason about.
Make VideoFrame ref-counted and return NonnullRefPtr from the decoder
and media queues. This changes ownership only: a VideoFrame still wraps
an ImmutableBitmap at this point, so playback behavior remains
unchanged while later commits can move storage and painting
independently.
Doing this in FFmpegVideoDecoder meant that the fallback to BT.709 with
unspecified transfer characteristics was taking precedence. Instead of
overriding unspecified to sRGB there, change the switch statement in
ColorSpace::from_cicp() so that the unspecified fallback and override
both live in the same place.
This should fix the contrast on a lot of YouTube videos.
Note that this now also affects images, which is consistent with
Chromium.
Storing these was pointless, since they're only used briefly to provide
the info needed to upload the buffers to the GPU.
For BT.2020 coefficients, we can just say that the bit depth is 16 bits
since we're always bit replicating to that for Skia's shaders.
PlaybackManager then intersects all enabled tracks' buffered time
ranges. This will be used by the media element for the buffered
attribute and to update the ready state.
Apparently this function uses a bitrate heuristic to determine which
track is best. We don't want or need that, so just select the first
track with default disposition (e.g. FlagDefault=1 in Matroska).
...giving tracks a kind attribute, and renaming name to label.
Demuxers will need to determine the kind attribute, since the spec for
sourcing tracks requires us to select based on info we don't expose.
libavcodec apparently holds onto any error that is not AVERROR_EOF when
a read fails. This means that reading until EOF after an aborted read
results in us receiving an AVERROR_EXIT in FFmpegDemuxer instead of
AVERROR_EOF, which causes the playback system to enter an error state
without decoding all frames in the file.
Instead, just always return AVERROR_EOF, and check if the read was
aborted in FFmpegDemuxer instead to return the correct error category
from there.
For web audio, I reckon an occasional misjudged channel layout is
better than more frequent exceptions.
Signed PCM is normalized with unsigned max divided by 2, not
signed max. If you divide by the signed max (32767), you get headroom
that can exceed the threshold below -1.0. It's not audible, this mostly
matters for tests that assume correct normalization. But it turns out
there's no shortage of "golden ears" jackholes out there who swear they
can hear the difference.
The way that other classes interact with IncrementallyPopulatedStream
is now through a virtual interface MediaStream and MediaStreamCursor.
This way, we can have simpler implementations of reading media data
that will not require an RB tree and synchronization.
...and abstract away the stream/cursor blocking/aborting functionality
so that demuxers can implement or ignore those methods as they see fit.
This is a step towards implementing a wrapper demuxer for MSE streams.
It's not necessary to keep around an instance of AVFormatContext in
FFmpegDemuxer, so instead just copy out the info we need for our
implementation and then destroy it so that our stream cursor is freed.
This saves us from having our own color conversion code, which was
taking up a fair amount of time in VideoDataProvider. With this change,
we should be able to play high resolution videos without interruptions
on machines where the CPU can keep up with decoding.
In order to make this change, ImmutableBitmap is now able to be
constructed with YUV data instead of an RBG bitmap. It holds onto a
YUVData instance that stores the buffers of image data, since Skia
itself doesn't take ownership of them.
In order to support greater than 8 bits of color depth, we normalize
the 10- or 12-bit color values into a 16-bit range.
When a seek is requested while a previous seek is still blocked waiting
for not yet available bytes, we want to abandon the old request
immediately and start processing the new one.
Refactor the FFmpeg and Matroska demuxers to consume data through
`IncrementallyPopulatedStream::Cursor` instead of a pointer to fully
buffered.
This change establishes a new rule: each track must be initialized with
its own cursor. Data providers now explicitly create a per-track context
via `Demuxer::create_context_for_track(track, cursor)`, and own pointer
to that cursor. In the upcoming changes, holding the cursor in the
provider would allow to signal "cancel blocking reads" so an
in-flight seek can fail immediately when a newer seek request arrives.
Audio blocks now contain a sample specification with the sample rate
and channel map for the audio data they contain. This will facilitate
conversion from one sample specification to another in order to allow
playback on devices with more or less speakers than the audio data
contains.
