Instead of tracking in-flight seeks across all the sinks in the seeking
state handler, move the logic to PlaybackManager to determine the
overall status and then notify the state of that status to potentially
trigger resumption.
The buffering state handler can then share essentially the same logic
instead of having the playback manager specifically track the blocked
tracks for it.
This has no effect on the user experience, and likely also produces the
same timestamps during a seek as before. However, it is needed in order
to ensure that the future Ended state is always at the duration.
The algorithm is ported over from Chromium's, which produces very good
results for speech, while being not objectionable for music, especially
in the background.
Other algorithms were tested.
Phase vocoders:
- Bungee
- Signalsmith Stretch
- pvdoneright
All three of these exhibited the usual phase shifting artifacts,
causing speech to sound slightly shifted into the high end. Speech is
the main thing we want to optimize for, so these aren't ideal.
Sonic (TD-PSOLA) performs better than WSOLA for speech, especially at
rates higher than 2x, but makes background sounds/music garbled and
unpleasant. It is still worth considering for speech clarity, and could
be added as an optional feature.
Pass direct Core::EventLoop references through media producer and
playback plumbing that posts work back to the main thread. The browser
process main loops stay alive for the process lifetime, so these paths
no longer need weak event loop references.
Update LibMedia tests to pass their stack event loop directly to the
producer helpers.
Instead of using a playback state to initiate and wake from suspension,
suspend the decoders themselves based on the time since the last status
check or pull. This allows audio and video to suspend independently,
and fixes videos getting stuck after suspension due to the waking seeks
being skipped downstream of the decoder, leaving the decoder suspended.
This is an intermediate step towards unifying the pipeline around new
Producer/Sink interfaces. Producers now have a pull() method that gets
the next piece of data from them. The pull() method returns a status
that can indicate whether it has current data, and if not, why it's
unavailable. This signal will be passed down the pipeline to the final
sink, which can expose the signal to its user, which in the normal
playback pipeline is PlaybackManager. The signal can be used to
transition between playback states. Currently, this is only hooked up
to the buffering state, but should be used later for ending playback
as well as decoding error propagation.
Buffering is now determined solely based on whether the pipeline is
blocked on incomplete data, so the ready state for video now progresses
past HAVE_METADATA immediately after playback manager initializes. This
will change when files have buffered ranges.
- Provider -> producer
- (Audio|Video)DataProvider -> Decoded(Audio|Video)Producer
- MediaTimeProvider remains suffixed Provider, moves out of the
Providers folder to the root of LibMedia
This brings the naming more in line with the intended split
functionality split between different nodes in the pipeline.
This doesn't actually change things too much from the prior commit, but
acts as a step towards making mixing into a sink/provider combo in the
new pipeline model.
This commit splits out synchronization primitives from LibThreading into
LibSync. This is because LibThreading depends on LibCore, while LibCore
needs the synchronization primitives from LibThreading. This worked
while they were header only, but when I tried to add an implementation
file it ran into the circular dependency. To abstract away the pthread
implementation using cpp files is necessary so the synchronization
primitives were moved to a separate library.
This state will indicate to the media element that it's not guaranteed
to have a frame yet, for the purposes of determining the ready state.
JavaScript should be sure that video elements with a ready state of
HAVE_CURRENT_DATA or greater represent the current video frame already.
To allow the state to be exited if audio is disabled, audio tracks are
now only added to the buffering set on enable if the audio sink exists,
since without the sink starting the data provider, it will never be
removed.
This is a step towards making video ref tests.
This allows us to differentiate between having no data available yet,
having current data, and having future data. The main purpose of this
is to allow a new starting state to explicitly force HAVE_METADATA
instead of >= HAVE_CURRENT_DATA.
Note that the SeekingStateHandler returns Current instead of None. This
is deliberate, since the buffered ranges from the demuxer(s) can be
used to inform whether the possibly-current data is actually available
at the seek target.
Audio output on macOS was consuming Core Audio resources until the
PlaybackStream creation took well over the timeout for some tests.
This was observed in media-source-buffered.html, where it would time
out due to the long-running callback on the main thread to create the
PlaybackStream for AudioMixingSink.
However, the AudioUnit init should definitely not be blocking the main
thread, so I've added a FIXME there.
This is delegated to the state handlers, but it essentially amounts to
`state() != Buffering && state() != Seeking`. If the PlaybackManager is
in either state, we know that there is no future data yet, as it should
exit those states as soon as the data is ready.
PlaybackManager then intersects all enabled tracks' buffered time
ranges. This will be used by the media element for the buffered
attribute and to update the ready state.
This will allow us to pass in a class implementing Demuxer for each
track owned by a MediaSource.
We'll also use the new ThreadPool class instead of a dedicated media
initialization thread. We shouldn't spin up a new thread for such a
trivial operation.
Previously, we would just listen to the single video track for
buffering, so if for some reason the audio data runs ahead of the
video, we would drop some audio until the video buffered. Instead,
stop playing audio at the last available sample when any provider is
blocked.
Also, PlaybackManager now starts in the Buffering state, so that it can
wait for enough data to be ready to play without interruption. When the
end of the stream is reached, the buffering state is exited to ensure
that we don't get stuck buffering at the end of a media file.
Hook up a callback in AudioMixingSink to notify PlaybackManager if the
output fails to be initialized. Then, when that happens, swap out the
time provider for GenericTimeProvider and continue without audio.
Fixes#8071
PlaybackManager's ref counting was only used to keep it alive in a few
callbacks. Instead, the callbacks can use weak references that can only
be used from the thread that the PlaybackManager was created on, to
ensure that the PlaybackManager can't be destroyed while being
accessed.
This ensures that:
- The PlaybackManager is destroyed immediately when it is reassigned
by HTMLMediaElement
- No callbacks are invoked after that point
This fixes the crash initially being addressed by #8081. The test from
that PR has been included as a regression test.
In order to free up memory when a video is paused for an extended
period, we add a new Suspended state to PlaybackManager which tells the
data providers to suspend. The data providers will handle this signal
by disposing of their entire decoded data queue and flushing their
decoder.
When initially creating a PlaybackManager, and when resuming to a
paused state, the delay before suspension will be much lower than when
pausing from any other state. This is intended to prevent media
elements from consuming memory for long when decoding the first frame
for display, as well as to allow the data providers to suspend much
more quickly after a seek while paused.
Currently, resuming playback doesn't display much of a delay on my
MacBook, though that may change once we completely tear down the
decoder in the suspended state. It may also be exacerbated by using
hardware decoders due more complex decoder initialization.
Refactor the FFmpeg and Matroska demuxers to consume data through
`IncrementallyPopulatedStream::Cursor` instead of a pointer to fully
buffered.
This change establishes a new rule: each track must be initialized with
its own cursor. Data providers now explicitly create a per-track context
via `Demuxer::create_context_for_track(track, cursor)`, and own pointer
to that cursor. In the upcoming changes, holding the cursor in the
provider would allow to signal "cancel blocking reads" so an
in-flight seek can fail immediately when a newer seek request arrives.
Posting callbacks to the main thread is now predicated on whether the
event loop reference is alive, preventing a stack-use-after-return.
The data providers will also check if they've been requested to exit
before calling deferred_invoke, though this is not going to be the case
unless the media element gets GCed while the media is playing.
Demuxer creation and track+duration extraction are moved to a separate
thread so that the media data byte buffer is no longer accessed from the
main thread. This will be important once the buffer is populated
incrementally, as having the main thread both populate and read from the
same buffer could easily lead to deadlocks. Aside from that, moving
demuxer creation off the main thread helps to be more responsive.
`VideoDataProvider` and `AudioDataProvider` now accept the main thread
event loop pointer as they are constructed from the thread responsible
for demuxer creation.
This implementation allows:
- Accurate seeking to an exact timestamp
- Seeking to the keyframe before a timestamp
- Seeking to the keyframe after a timestamp
These three options will be used to satisfy the playback position
selection in the media element's seeking steps.
This time provider can later be swapped out for the AudioMixingSink
when it implements the MediaTimeProvider interface, so that frame
timing can be driven by audio when it is present.
This commit implements the functionality to play back audio through
PlaybackManager.
To decode the audio data, AudioDataProviders are created for each track
in the provided media data. These providers will fill their audio block
queue, then sit idle until their corresponding tracks are enabled.
In order to output the audio, one AudioMixingSink is created which
manages a PlaybackStream which requests audio blocks from multiple
AudioDataProviders and mixes them into one buffer with sample-perfect
precision.
With this commit, all PlaybackManager can do is autoplay a file from
start to finish, with no pausing or seeking functionality.
All audio playback functionality has been removed from HTMLMediaElement
and HTMLAudioElement in anticipation of PlaybackManager taking that
over, for both audio-only and audio/video.
This commit removes the -Wno-unusued-private-field flag, thus
reenabling the warning. Unused field were either removed or marked
[[maybe_unused]] when unsure.
