Frames are considered late if the time is ahead by half their duration.
The fudging is necessary because Matroska (and perhaps other formats)
store their frame durations in different time units than their
timestamps.
Skipping these should make it clear when decoding is running behind,
instead of displaying the video in slow motion while audio runs at a
normal rate.
To give an accurate counting for video playback quality when it is
implemented, we'll most likely want to count all pulled frames in an
update as dropped if the last frame is dropped. Otherwise the frame
drop count will only increase at the display rate when decoding is
continually running behind.
This is already handled by the seeking state enum later. We could end
up displaying nothing if MovedPosition isn't immediately followed by a
displayable frame.
Instead of comparing the current time to the duration, the playback
manager now has an explicit Ended state that jumps to the duration. The
element simply reacts to that to trigger the ended event and attribute,
along with all the other steps involved.
This moves the ended event to fire after the seeked event, which
matches other browsers' behavior. The spec doesn't explicitly say which
order they should fire in.
Instead of tracking in-flight seeks across all the sinks in the seeking
state handler, move the logic to PlaybackManager to determine the
overall status and then notify the state of that status to potentially
trigger resumption.
The buffering state handler can then share essentially the same logic
instead of having the playback manager specifically track the blocked
tracks for it.
This has no effect on the user experience, and likely also produces the
same timestamps during a seek as before. However, it is needed in order
to ensure that the future Ended state is always at the duration.
This matches the mixer's behavior, it won't mix data once it reaches
any Pending status on any input.
Doing this will allow seeking to also rely on the combined status
function to determine when to resolve.
We don't need to proactively reset this anymore with the break out of
halting being determined by seek IDs instead. This will ensure that a
halting status upon a fast-path seek will still remain correct.
Checking for a seek within the error wait loop wasn't necessary, and
unlocking/relocking could actually lose a wake anyway. Callers always
break out of the decode loop into the thread loop, so this will still
seek immediately as before.
Otherwise, we can lose the signal if a seek is aborted after it has
moved the demuxer, making the sink not output until it reaches the time
of the last data it pulled.
The algorithm is ported over from Chromium's, which produces very good
results for speech, while being not objectionable for music, especially
in the background.
Other algorithms were tested.
Phase vocoders:
- Bungee
- Signalsmith Stretch
- pvdoneright
All three of these exhibited the usual phase shifting artifacts,
causing speech to sound slightly shifted into the high end. Speech is
the main thing we want to optimize for, so these aren't ideal.
Sonic (TD-PSOLA) performs better than WSOLA for speech, especially at
rates higher than 2x, but makes background sounds/music garbled and
unpleasant. It is still worth considering for speech clarity, and could
be added as an optional feature.
This has two benefits:
- Pausing and then disabling a track no longer has lingering audio from
the disabled track
- The mixer starts mixing from the current time even after it has
advanced past EOS, so it doesn't have to mix audio that won't play
This allows nodes downstream of the mixer to know the exact frame at
which EOS is reached.
More concepts have been introduced into PipelineStatus.h to make each
condition in the pipeline clearer about its intent.
This could cause an upstream transition from Pending to HaveData to
never unblock the audio processor thread if that transition happened
between setting the wait flag and re-checking the status for
MovedPosition at the start of the processor loop.
This wasn't observable with the existing pipeline simply because it was
near impossible to run down the upstream data to get a Pending status.
This allows the downstream node to query status and sleep again
synchronously, which is needed for nodes that need multiple input
blocks before they can produce output and wake their downstream nodes.
Pass direct Core::EventLoop references through media producer and
playback plumbing that posts work back to the main thread. The browser
process main loops stay alive for the process lifetime, so these paths
no longer need weak event loop references.
Update LibMedia tests to pass their stack event loop directly to the
producer helpers.
Enable -Wexit-time-destructors for all in-tree library targets and
update process-lifetime library statics so they no longer register
exit-time destructors. Long-lived caches, lookup tables, singleton
registries, and generated constants now use NeverDestroyed or leaked
references where the data is intended to live until process exit.
Update LibWeb, LibLine, and the binding generators so regenerated
sources follow the same rule instead of reintroducing destructed
statics.
Fixes an extremely rare flake in the resize-event-during-playback.html
test when seeks are resolved in DisplayingVideoSink::update(). When
they are resolved there, the media element would queue the `seeked`
event before the `resize` event, causing the event ordering in the test
to be incorrect.
This would normally not be hit due to seeks primarily being resolved by
the wake handler of the input.
Apparently FFmpeg probes WAV files for 32 packets before returning the
stream info. With the default of 19200 bytes per packet, this could end
up waiting for up to 5 seconds of data to be downloaded.
Setting the max_size option only affects WAV and W64 in our build of
libavformat. No other formats we care about will be affected.
Unfortunately, this necessarily involves parsing codec frames to get
their durations. Ogg's granule positions indicate the last sample of
the last complete frame of the page, so the navigator has to determine
the durations of every packet in the page to offset it back to its
start.
Vorbis is an especially involved codec to determine that for, since it
has initial data defining a mapping of indices to big or small block
sizes that has to be held onto, and that mapping is preceded by a bunch
of conditionally parsed bits. Also, a frame's block size calculation
involves the block size of the previous frame for an overlap add. To
avoid bringing that complexity directly into LibMedia, we delegate the
parsing/calculation to FFmpeg's av_vorbis_parse functions.
A new MP3Navigator class is added, which determines timestamps for byte
positions by resyncing to a frame and then interpolating between known
points on either side. The known points start out as the first frame's
position in the file at timestamp 0, and EOF at the timestamp for
FFmpeg's file duration estimate. New buffered ranges are interpolated
between those two points, but also between the end of a prior range and
the start of the next.
Since MP3 can have variable bitrate without declaring it in the file
header, we have to allow buffered ranges to shift forward as new data
arrives to make room for underestimated durations. This is done for all
ranges following the first that has been appended to, keeping the start
of the current range consistent, so that subsequent seeks within that
range remain consistent.
Seeking is also implemented within the navigator to ensure that the
byte<->timestamp mapping is consistent and the buffered ranges begin
exactly where the seek landed.
We could end up seeked within AVIOContext's internal buffer without it
checking whether new data could be buffered without hitting EOF again,
so we could get an unexpected read error.
Add a new FLACNavigator class that can scan for frames both forward and
backward from a byte offset to get the exact buffered ranges. The reads
are done in chunks, which keeps each call pretty cheap.
The ranges themselves are resolved by its ScanningContainerNavigator
base class, which reconciles cached ranges against the incoming ones to
avoid repeating work unnecessarily. With those combined optimizations,
the calls normally take under 1 microsecond.
The default mode is to grab the index and use it to determine the start
and end of the buffered ranges. This works well for MP4.
For WAV, the index is incomplete, so instead use a much simpler method,
just determining the ranges based on the constant bitrate of the file.
This is implemented through a new virtual ContainerNavigator class.
This was unintentionally clobbering the prior modification to the
request position. Requesting data slightly earlier in the file is
intended to ensure that we have data if the demuxer decides to seek
backwards.
Returning early if an added chunk didn't overlap the previous one could
result in overlapping chunks or 0-byte gaps if a chunk was after but
within the new one. Instead, fall through to allow the merging to
continue as normal.
Also, make the joining condition check if the new chunk spans multiple
existing chunks to merge/remove those.
