Frames are considered late if the time is ahead by half their duration.
The fudging is necessary because Matroska (and perhaps other formats)
store their frame durations in different time units than their
timestamps.
Skipping these should make it clear when decoding is running behind,
instead of displaying the video in slow motion while audio runs at a
normal rate.
To give an accurate counting for video playback quality when it is
implemented, we'll most likely want to count all pulled frames in an
update as dropped if the last frame is dropped. Otherwise the frame
drop count will only increase at the display rate when decoding is
continually running behind.
This is already handled by the seeking state enum later. We could end
up displaying nothing if MovedPosition isn't immediately followed by a
displayable frame.
This allows nodes downstream of the mixer to know the exact frame at
which EOS is reached.
More concepts have been introduced into PipelineStatus.h to make each
condition in the pipeline clearer about its intent.
This could cause an upstream transition from Pending to HaveData to
never unblock the audio processor thread if that transition happened
between setting the wait flag and re-checking the status for
MovedPosition at the start of the processor loop.
This wasn't observable with the existing pipeline simply because it was
near impossible to run down the upstream data to get a Pending status.
Pass direct Core::EventLoop references through media producer and
playback plumbing that posts work back to the main thread. The browser
process main loops stay alive for the process lifetime, so these paths
no longer need weak event loop references.
Update LibMedia tests to pass their stack event loop directly to the
producer helpers.
Fixes an extremely rare flake in the resize-event-during-playback.html
test when seeks are resolved in DisplayingVideoSink::update(). When
they are resolved there, the media element would queue the `seeked`
event before the `resize` event, causing the event ordering in the test
to be incorrect.
This would normally not be hit due to seeks primarily being resolved by
the wake handler of the input.
We could potentially have a pull() that would output audio after a
status query said that data was pending. Instead, ensure that we keep
the subsequent pull() from providing anything in such cases.
Also, rewrite HaveData to Pending when it outputs nothing. When
multiple audio tracks were enabled, it was possible to reach the empty
block branch with a HaveData status, since it takes priority over
Pending. This fixes a crash in pull() verifying that the combined
status is not HaveData.
We were clearing the current frame before we knew that we had new data
to complete an ongoing seek. Instead, we can eagerly clear the next
frame to allow the new one to come in, but keep the current frame until
that new data is available.
We already have an optimization in demuxers to skip seeking to random
access points when the closest available is not closer than the current
timestamp in the file. This commit extends that same concept to the
rest of the pipeline.
DecodedVideoProducer now tracks a time range that is currently within
the queue. If the seeked time falls within that range, handle_seek()
is skipped, since the downstream can just skip forward to that
timestamp synchronously.
Similarly, DisplayingVideoSink checks whether seeks land in the range
between the current frame and the next frame end, and if so, the seek
never gets sent upstream in the pipeline.
Both use a fudging factor of 1.5x the duration to determine the end of
the available range, since Matroska/WebM doesn't use the same time
units for timestamps and durations, and therefore can leave a gap
between frames.
Seeks don't always move a decoded data producer's head, so we need to
make sure not to remove queued data downstream when that is the case.
To communicate this, the producers can now be queried before pulling
data, allowing them to have an in-band signal to clear the queued data
after a seek has moved the producer and broken monotonicity.
This fixes a flake in HTMLVideoElement-resize-event-during-playback.
Previously, we weren't too consistent about the definition of frame and
sample when it relates to raw audio data. This brings all the usages in
the context of raw data in line (hopefully), with samples referring to
a single PCM value, and frames referring to the multiple samples that
make up an instant's audio across all channels.
With this last piece done, we can now connect a DecodedAudioProducer
directly to an AudioPlaybackSink instead of being forced to go through
the mixer. This should open up options for other pipeline setups if
needed in the future.
Now, all nodes are connected through Sink::connect_input() and
disconnect_input().
AudioMixer now derives from a base AudioProcessor class that inherits
from both AudioSink and AudioProducer. It is the only current node that
can accept multiple inputs, tracking each one by its pointer identity.
Seeking is now unified under one single method signature implemented by
all producers and transmitted through the pipeline by all sinks. By
doing it this way, we can simply instantaneously notify each node of
the pipeline that it needs to stop what it's doing and seek. For nodes
that are threaded (particularly the source providers), this causes them
to stop pushing data to their queue immediately, so that no stale data
makes it through to the output. Then, when new data does come through,
that is a clear indication that the seek has completed.
Note that track enablement is now through the pipeline as well, which
means that SuspendedStateHandler no longer has a way to suspend newly-
enabled tracks. Decoder suspension will need to be reworked to fit into
this new pipeline, sleeping/disposing and restarting entirely based on
the pull() timing in the producers.
Using a callback shared by all producers in the pipeline, notify the
AudioPlaybackSink when it needs to wake up and start processing data
again. Prior to this commit, it was simply burning CPU spinning until
data was produced.
This is an intermediate step towards unifying the pipeline around new
Producer/Sink interfaces. Producers now have a pull() method that gets
the next piece of data from them. The pull() method returns a status
that can indicate whether it has current data, and if not, why it's
unavailable. This signal will be passed down the pipeline to the final
sink, which can expose the signal to its user, which in the normal
playback pipeline is PlaybackManager. The signal can be used to
transition between playback states. Currently, this is only hooked up
to the buffering state, but should be used later for ending playback
as well as decoding error propagation.
Buffering is now determined solely based on whether the pipeline is
blocked on incomplete data, so the ready state for video now progresses
past HAVE_METADATA immediately after playback manager initializes. This
will change when files have buffered ranges.
- Provider -> producer
- (Audio|Video)DataProvider -> Decoded(Audio|Video)Producer
- MediaTimeProvider remains suffixed Provider, moves out of the
Providers folder to the root of LibMedia
This brings the naming more in line with the intended split
functionality split between different nodes in the pipeline.
This doesn't actually change things too much from the prior commit, but
acts as a step towards making mixing into a sink/provider combo in the
new pipeline model.
While mixing doesn't cost much time at the moment, moving audio
processing off the callback thread means that the callback itself can
block for a minimal amount of time regardless of the amount of time
needed to actually process the audio data.
This matters a lot for AudioUnit, where the entire daemon can start to
glitch if delays occur.
The goal of this change is to split mixing off into a separate class,
and process audio on both ends of the mixer synchronously. This should
allow a more declarative approach to processing audio, where inserting
a time stretching processor, for example, becomes much simpler.
In this commit, the thread spin-waits for new upstream data. Making it
wait on a condition variable here would overcomplicate this commit, so
it's deferred until a later commit establishes that forward wakeup.
The playback stream is never recreated currently, so this wasn't
actually doing anything. This is now made explicit by never resetting
the playback stream creation flag when it succeeds.
Instead of skipping setting this if the playback stream isn't ready, we
need to store the time so that the stream creation resolution callback
can begin the seek later.
Recalculating after suspend isn't really useful if the PlaybackStream
implementations ensure that the time doesn't advance past the data that
has been written so far.
The second paragraph of this comment was outdated, and the comment
isn't really necessary to understand the purpose of the function, when
it returns a status indicative of that purpose. To ensure the value is
checked, add [[nodiscard]].
This commit splits out synchronization primitives from LibThreading into
LibSync. This is because LibThreading depends on LibCore, while LibCore
needs the synchronization primitives from LibThreading. This worked
while they were header only, but when I tried to add an implementation
file it ran into the circular dependency. To abstract away the pthread
implementation using cpp files is necessary so the synchronization
primitives were moved to a separate library.
The display queue used TimedImage even though the media pipeline is
selecting decoded video frames. That naming hid the real object being
handed from LibMedia to Web and kept the queue interface coupled to
bitmap-style painting.
Rename the wrapper to TimedVideoFrame and pass ref-counted VideoFrame
objects through the provider and display sink. Web still reads the
ImmutableBitmap from the frame for painting in this commit, so rendered
output and conversion behavior stay the same while the playback-facing
interfaces become frame-shaped.
Previously, we would just listen to the single video track for
buffering, so if for some reason the audio data runs ahead of the
video, we would drop some audio until the video buffered. Instead,
stop playing audio at the last available sample when any provider is
blocked.
Also, PlaybackManager now starts in the Buffering state, so that it can
wait for enough data to be ready to play without interruption. When the
end of the stream is reached, the buffering state is exited to ensure
that we don't get stuck buffering at the end of a media file.
Hook up a callback in AudioMixingSink to notify PlaybackManager if the
output fails to be initialized. Then, when that happens, swap out the
time provider for GenericTimeProvider and continue without audio.
Fixes#8071
This allows us to avoid returning a PlaybackStream in cases where the
async initialization fails.
This is a step towards more graceful fallbacks when audio fails in
AudioMixingSink.
If a set_time()'s callbacks were still pending, but a resume() went
through, the audio timing could get a bit confused. Instead, check if
m_temporary_time is set before resuming, since that would indicate that
the set_time() callbacks will call resume() when the time is stable
again.
Also, multiple set_time() calls will now only start one set of tasks,
and m_temporary_time is used to store the time passed to the last call.
Thus, set_time() doesn't conflict with itself, but no calls to
set_time() are ignored still.
We could hit a VERIFY in RefPtr if there was a seek in flight while the
PlaybackManager was being destroyed, since finishing a seek would run
DisplayingVideoSink::resume_updates() which would then check if there
is a new frame to display.
In order to free up memory when a video is paused for an extended
period, we add a new Suspended state to PlaybackManager which tells the
data providers to suspend. The data providers will handle this signal
by disposing of their entire decoded data queue and flushing their
decoder.
When initially creating a PlaybackManager, and when resuming to a
paused state, the delay before suspension will be much lower than when
pausing from any other state. This is intended to prevent media
elements from consuming memory for long when decoding the first frame
for display, as well as to allow the data providers to suspend much
more quickly after a seek while paused.
Currently, resuming playback doesn't display much of a delay on my
MacBook, though that may change once we completely tear down the
decoder in the suspended state. It may also be exacerbated by using
hardware decoders due more complex decoder initialization.
This saves us from having our own color conversion code, which was
taking up a fair amount of time in VideoDataProvider. With this change,
we should be able to play high resolution videos without interruptions
on machines where the CPU can keep up with decoding.
In order to make this change, ImmutableBitmap is now able to be
constructed with YUV data instead of an RBG bitmap. It holds onto a
YUVData instance that stores the buffers of image data, since Skia
itself doesn't take ownership of them.
In order to support greater than 8 bits of color depth, we normalize
the 10- or 12-bit color values into a 16-bit range.
`IncrementallyPopulatedStream::Cursor` now tracks whether it's currently
blocked inside a wait for more bytes, allowing higher layers to
distinguish "no frames yet" from "decoder is idle".
Enter buffering when `DisplayingVideoSink` runs out of frames and the
associated `VideoDataProvider` is blocked waiting for data to arrive.
Exit buffering once decoding refills the frame queue.
For now, buffering behaves like paused, but it gives us an explicit
state to hook UI into.
