Instead of tracking in-flight seeks across all the sinks in the seeking
state handler, move the logic to PlaybackManager to determine the
overall status and then notify the state of that status to potentially
trigger resumption.
The buffering state handler can then share essentially the same logic
instead of having the playback manager specifically track the blocked
tracks for it.
This has no effect on the user experience, and likely also produces the
same timestamps during a seek as before. However, it is needed in order
to ensure that the future Ended state is always at the duration.
The algorithm is ported over from Chromium's, which produces very good
results for speech, while being not objectionable for music, especially
in the background.
Other algorithms were tested.
Phase vocoders:
- Bungee
- Signalsmith Stretch
- pvdoneright
All three of these exhibited the usual phase shifting artifacts,
causing speech to sound slightly shifted into the high end. Speech is
the main thing we want to optimize for, so these aren't ideal.
Sonic (TD-PSOLA) performs better than WSOLA for speech, especially at
rates higher than 2x, but makes background sounds/music garbled and
unpleasant. It is still worth considering for speech clarity, and could
be added as an optional feature.
This has two benefits:
- Pausing and then disabling a track no longer has lingering audio from
the disabled track
- The mixer starts mixing from the current time even after it has
advanced past EOS, so it doesn't have to mix audio that won't play
Pass direct Core::EventLoop references through media producer and
playback plumbing that posts work back to the main thread. The browser
process main loops stay alive for the process lifetime, so these paths
no longer need weak event loop references.
Update LibMedia tests to pass their stack event loop directly to the
producer helpers.
This ensures that the media element unpauses after a seek so that when
EOF detection is complete, it can continue playback and jump to the end
of the timeline and end playback.
With this last piece done, we can now connect a DecodedAudioProducer
directly to an AudioPlaybackSink instead of being forced to go through
the mixer. This should open up options for other pipeline setups if
needed in the future.
Now, all nodes are connected through Sink::connect_input() and
disconnect_input().
AudioMixer now derives from a base AudioProcessor class that inherits
from both AudioSink and AudioProducer. It is the only current node that
can accept multiple inputs, tracking each one by its pointer identity.
Seeking is now unified under one single method signature implemented by
all producers and transmitted through the pipeline by all sinks. By
doing it this way, we can simply instantaneously notify each node of
the pipeline that it needs to stop what it's doing and seek. For nodes
that are threaded (particularly the source providers), this causes them
to stop pushing data to their queue immediately, so that no stale data
makes it through to the output. Then, when new data does come through,
that is a clear indication that the seek has completed.
Note that track enablement is now through the pipeline as well, which
means that SuspendedStateHandler no longer has a way to suspend newly-
enabled tracks. Decoder suspension will need to be reworked to fit into
this new pipeline, sleeping/disposing and restarting entirely based on
the pull() timing in the producers.
This is an intermediate step towards unifying the pipeline around new
Producer/Sink interfaces. Producers now have a pull() method that gets
the next piece of data from them. The pull() method returns a status
that can indicate whether it has current data, and if not, why it's
unavailable. This signal will be passed down the pipeline to the final
sink, which can expose the signal to its user, which in the normal
playback pipeline is PlaybackManager. The signal can be used to
transition between playback states. Currently, this is only hooked up
to the buffering state, but should be used later for ending playback
as well as decoding error propagation.
Buffering is now determined solely based on whether the pipeline is
blocked on incomplete data, so the ready state for video now progresses
past HAVE_METADATA immediately after playback manager initializes. This
will change when files have buffered ranges.
- Provider -> producer
- (Audio|Video)DataProvider -> Decoded(Audio|Video)Producer
- MediaTimeProvider remains suffixed Provider, moves out of the
Providers folder to the root of LibMedia
This brings the naming more in line with the intended split
functionality split between different nodes in the pipeline.
This doesn't actually change things too much from the prior commit, but
acts as a step towards making mixing into a sink/provider combo in the
new pipeline model.
This state will indicate to the media element that it's not guaranteed
to have a frame yet, for the purposes of determining the ready state.
JavaScript should be sure that video elements with a ready state of
HAVE_CURRENT_DATA or greater represent the current video frame already.
To allow the state to be exited if audio is disabled, audio tracks are
now only added to the buffering set on enable if the audio sink exists,
since without the sink starting the data provider, it will never be
removed.
This is a step towards making video ref tests.
This allows us to differentiate between having no data available yet,
having current data, and having future data. The main purpose of this
is to allow a new starting state to explicitly force HAVE_METADATA
instead of >= HAVE_CURRENT_DATA.
Note that the SeekingStateHandler returns Current instead of None. This
is deliberate, since the buffered ranges from the demuxer(s) can be
used to inform whether the possibly-current data is actually available
at the seek target.
Having PlaybackManager start in Buffering was causing us to report
a media element readyState of HAVE_CURRENT_DATA. HAVE_CURRENT_DATA
doesn't make a whole lot of sense for local files, since we should have
all the data immediately when we process the metadata. This is
reflected in the buffered attribute, so let's not limit the ready state
unecessarily.
This fixes a crash when a track is enabled and then disabled while a
seek is in progress.
The logic in SeekingStateHandler is reworked to keep track of the
tracks that are currently being seeked, and when a track is disabled,
it is no longer counted against the seek completion. Any seek
completion callback that was instated is cleared by calling seek with
a null callback.
It may be worth making a separate function on the data providers to
clear the current seek instead, to avoid the extra work of seeking, but
this scenario is a very rare one unless someone intentionally triggers
it, and the cost is minimal unless the toggles are spammed.
A crash test is included, which both tests for the crash, and would
also time out if the failing VERIFY in on_track_enabled() was avoided
with the previous seeking implementation, due to the originally-enabled
video track's seek callback being clobbered by on_track_enabled()'s
seek.
Audio output on macOS was consuming Core Audio resources until the
PlaybackStream creation took well over the timeout for some tests.
This was observed in media-source-buffered.html, where it would time
out due to the long-running callback on the main thread to create the
PlaybackStream for AudioMixingSink.
However, the AudioUnit init should definitely not be blocking the main
thread, so I've added a FIXME there.
This is delegated to the state handlers, but it essentially amounts to
`state() != Buffering && state() != Seeking`. If the PlaybackManager is
in either state, we know that there is no future data yet, as it should
exit those states as soon as the data is ready.
PlaybackManager then intersects all enabled tracks' buffered time
ranges. This will be used by the media element for the buffered
attribute and to update the ready state.
This will allow us to pass in a class implementing Demuxer for each
track owned by a MediaSource.
We'll also use the new ThreadPool class instead of a dedicated media
initialization thread. We shouldn't spin up a new thread for such a
trivial operation.
Previously, we would just listen to the single video track for
buffering, so if for some reason the audio data runs ahead of the
video, we would drop some audio until the video buffered. Instead,
stop playing audio at the last available sample when any provider is
blocked.
Also, PlaybackManager now starts in the Buffering state, so that it can
wait for enough data to be ready to play without interruption. When the
end of the stream is reached, the buffering state is exited to ensure
that we don't get stuck buffering at the end of a media file.
Hook up a callback in AudioMixingSink to notify PlaybackManager if the
output fails to be initialized. Then, when that happens, swap out the
time provider for GenericTimeProvider and continue without audio.
Fixes#8071
Previously, we would call the frame end callbacks every time a frame
was decoded. However, the only use case for the callback was to update
the media duration. Instead, cache the duration in the data providers,
and only invoke the callback (renamed to duration_change_handler) when
the duration actually increases. Hopefully this will reduce wasted
work, and possibly some allocations as well.
We don't use EndOfStream errors as a way to determine the end of the
media anyway. It's still appropriate to keep those errors around in
data providers, though, since they are used in simpler usecases like
tests.
PlaybackManager's ref counting was only used to keep it alive in a few
callbacks. Instead, the callbacks can use weak references that can only
be used from the thread that the PlaybackManager was created on, to
ensure that the PlaybackManager can't be destroyed while being
accessed.
This ensures that:
- The PlaybackManager is destroyed immediately when it is reassigned
by HTMLMediaElement
- No callbacks are invoked after that point
This fixes the crash initially being addressed by #8081. The test from
that PR has been included as a regression test.
We only need to take a strong reference to the main event loop when
an error occurred in order to invoke the callback on the main thread.
By taking this lock for the entire duration of the thread, we were
preventing the main thread from exiting if the init thread hangs.
...and abstract away the stream/cursor blocking/aborting functionality
so that demuxers can implement or ignore those methods as they see fit.
This is a step towards implementing a wrapper demuxer for MSE streams.
In order to free up memory when a video is paused for an extended
period, we add a new Suspended state to PlaybackManager which tells the
data providers to suspend. The data providers will handle this signal
by disposing of their entire decoded data queue and flushing their
decoder.
When initially creating a PlaybackManager, and when resuming to a
paused state, the delay before suspension will be much lower than when
pausing from any other state. This is intended to prevent media
elements from consuming memory for long when decoding the first frame
for display, as well as to allow the data providers to suspend much
more quickly after a seek while paused.
Currently, resuming playback doesn't display much of a delay on my
MacBook, though that may change once we completely tear down the
decoder in the suspended state. It may also be exacerbated by using
hardware decoders due more complex decoder initialization.
By sniffing specifically for MP4 and WebM, we were precluding
PlaybackManager from playing any other formats. Instead, use
MatroskaDemuxer if the media has a `matroska` or `webm` EBML doctype,
and fall back to FFmpeg for all others.
We'll need to limit the containers that FFmpeg is able to open at some
point, but for now, this allows us to play the formats we could before.
`IncrementallyPopulatedStream::Cursor` now tracks whether it's currently
blocked inside a wait for more bytes, allowing higher layers to
distinguish "no frames yet" from "decoder is idle".
Enter buffering when `DisplayingVideoSink` runs out of frames and the
associated `VideoDataProvider` is blocked waiting for data to arrive.
Exit buffering once decoding refills the frame queue.
For now, buffering behaves like paused, but it gives us an explicit
state to hook UI into.
When media data is fully buffered, we can just try Matroska first and
fall back to FFmpeg. With incremental fetching, that approach becomes
wasteful: we may repeatedly attempt demuxer construction before enough
bytes are available, and FFmpeg in particular tends to produce noisy
logs while probing partial input.
Add lightweight container sniffing for WebM and MP4 that operates on
`IncrementallyPopulatedStream::Cursor`,
`prepare_playback_from_media_data()` now blocks until there is enough
data to decide the container type, then constructs the appropriate
demuxer directly instead of probing both.
Co-authored-by: Zaggy1024 <Zaggy1024@gmail.com>
Refactor the FFmpeg and Matroska demuxers to consume data through
`IncrementallyPopulatedStream::Cursor` instead of a pointer to fully
buffered.
This change establishes a new rule: each track must be initialized with
its own cursor. Data providers now explicitly create a per-track context
via `Demuxer::create_context_for_track(track, cursor)`, and own pointer
to that cursor. In the upcoming changes, holding the cursor in the
provider would allow to signal "cancel blocking reads" so an
in-flight seek can fail immediately when a newer seek request arrives.
