DecodedImageData::paint() used to take both a destination and a
clip rectangle even though most callers passed the same value. SVG
image painting used that API to wrap every nested SVG display list in
save/add-clip/restore, which put an unbounded command in front of
the bounded nested-list command and made offscreen SVG image content
harder to cull.
Move clipping to ImagePaintable, where the object-fit destination can
be compared with the replaced element box. CSS image and marker
painting continue to draw into their destination rect, while repeated
background images keep their explicit tile clip. The scaled decoded
image display-list command now stores only its destination rect and
uses that as its bounds; playback still clips decoded images to that
rect so bitmap rendering stays unchanged.
Future compositor-process work needs to push video frame updates without
going through VideoFrameSource. That object cannot be shared through
IPC, so video display-list resources now use stable VideoFrameResourceId
values and the current frame is sent through explicit resource/update
commands.
DecodedImageFrame only wraps a ref-counted Bitmap and color-space
metadata. The frame object itself does not provide shared mutable
state or lifetime ownership beyond those members, so ref-counting it
adds an unnecessary layer of indirection.
The Paintable tree and its supplemental painting data structures were
GC allocated because that was the easiest way to manage it and avoid
leaks introduced by ref cycles. This included the Paintable subclasses
themselves plus StackingContext, ChromeWidget, Scrollbar, ResizeHandle,
and scroll-frame state.
We are now trying to reduce GC allocation churn on layout and painting
updates, so keeping this short-lived rendering tree outside the JS heap
is a better fit. Move Paintable to RefCountedTreeNode, make painting
helpers ref-counted or weakly reference Paintables, and update the
layout and event-handler call sites to use RefPtr/WeakPtr ownership.
Decoded image data should not continue to traffic in ImmutableBitmap now
that the bitmap wrapper is being retired. Introduce DecodedImageFrame as
the paintable decoded-image unit and store a Bitmap plus ColorSpace in
it directly.
Thread the new frame type through decoded image data, display-list
image commands, filters, canvas drawImage, patterns, WebGL texture
upload, and CSS/SVG image consumers. ImmutableBitmap remains only at
the legacy boundaries that still need it, such as HTML video snapshots
and callers that explicitly ask for a bitmap snapshot.
This keeps color-space ownership with the decoded frame while making
the expensive or legacy ImmutableBitmap path explicit at the few call
sites that still need it.
A video element should record video as video, not as generic external
bitmap content. Add VideoFrameSource and a dedicated display-list
command so the display-list player receives the current
Media::VideoFrame directly.
The Skia player can now upload YUV pixmaps from the frame when a GPU
context is available, without teaching the ordinary ImmutableBitmap
image cache about media formats. If GPU upload is unavailable, the
fallback explicitly converts the frame through YUVData::to_bitmap().
This gives video painting a clear extension point for future frame
backends, such as hardware frames or other planar formats, while
keeping bitmap drawing focused on immutable pixel snapshots.
This tightens the implementation of video element sizing to the spec by
implementing two spec concepts:
- The media resource's natural width and height, and
- The video element's natural width and height.
The element's natural dimensions change based on the representation,
which has many inputs, so update checks are triggered from many
locations.
The resize event is fired when the media resource's natural dimensions
change, and the layout is invalidated if the element's natural
dimensions change.
Tests for a few important resize triggers have been added.
Instead of using a custom paintable to draw the controls for video and
audio elements, we build them out of plain old HTML elements within a
shadow root.
This required a few hacks in the previous commits in order to allow a
replaced element to host children within a shadow root, but it's
fairly self-contained.
A big benefit is that we can drive all the UI updates off of plain old
DOM events (except the play button overlay on videos, which uses the
video element representation), so we can test our media and input event
handling more thoroughly. :^)
The control bar visibility is now more similar to how other browsers
handle it. It will show upon hovering over the element, but if the
cursor is kept still for more than a second, it will hide again. While
dragging, the controls remain visible, and will then hide after the
mouse button is released.
The icons have been redesigned from scratch, and the mute icon now
visualizes the volume level along with indicating the mute state.
Publish new video frames to an ExternalContentSource, and switch
VideoPaintable from draw_scaled_immutable_bitmap to
draw_external_content.
Because DrawExternalContent reads the latest bitmap at replay time,
frame-only updates (no timeline or control change) now call
set_needs_display(InvalidateDisplayList::No) — skipping display list
rebuilds entirely. This addresses problem 2 from the previous commit.
This saves us from having our own color conversion code, which was
taking up a fair amount of time in VideoDataProvider. With this change,
we should be able to play high resolution videos without interruptions
on machines where the CPU can keep up with decoding.
In order to make this change, ImmutableBitmap is now able to be
constructed with YUV data instead of an RBG bitmap. It holds onto a
YUVData instance that stores the buffers of image data, since Skia
itself doesn't take ownership of them.
In order to support greater than 8 bits of color depth, we normalize
the 10- or 12-bit color values into a 16-bit range.
The `Bitmap` type was referring to to its internal pixel format by a
name that represents the order of the color components as they are layed
out in memory. Contrary, the `Color` type was using a naming that where
the name represents the order of the components from most to least
significant byte when viewed as a unsigned 32bit integer. This is
confusing as you have to keep remembering which mental model to use
depending on which code you work with.
To unify the two, the naming of RGBA-like colors in the `Color` type has
been adjusted to match the one from the Bitmap type. This seems to be
generally in line with how web APIs think about these types:
* `ImageData.pixelFormat` can be `rgba-8unorm` backed by a
`Uint8ClamedArray`, but there is no pixel format backed by a 32bit
unsigned type.
* WebGL can use format `RGBA` with type `UNSIGNED_BYTE`, but there is no
such format with type `UNSIGNED_INT`.
Additionally, it appears that other browsers and browser-adjacent
libraries also think similarly about these types:
* Firefox:
https://github.com/mozilla-firefox/firefox/blob/main/gfx/2d/Types.h
* WebKit:
https://github.com/WebKit/WebKit/blob/main/Source/WebCore/platform/graphics/PixelFormat.h
* Skia:
https://chromium.googlesource.com/skia/+/refs/heads/main/include/core/SkColorType.h
This has the not so nice side effect that APIs that interact with these
types through 32bit unsigned integers now have the component order
inverted due to little-endian byte order. E.g. specifying a color as hex
constant needs to be done as `0xAABBGGRR` if it is to be treated as
RGBA8888.
We could alleviate this by providing endian-independent APIs to callers.
But I suspect long-term we might want to think differently about bitmap
data anyway, e.g. to better support HDR in the future. However, such
changes would be more involved than just unifying the naming as done
here. So I considered that out of scope for now.
With this commit, all PlaybackManager can do is autoplay a file from
start to finish, with no pausing or seeking functionality.
All audio playback functionality has been removed from HTMLMediaElement
and HTMLAudioElement in anticipation of PlaybackManager taking that
over, for both audio-only and audio/video.
The extra representations of a video element are unnecessary to specify
in VideoPaintable, as the playback system itself should take care of
all the details of the representation as specified.
PaintContext dates back to a time when display lists didn't exist and it
truly represented "paint context". Renaming it to better align with its
current role.
Instead of trying to manually determine which parts of a bitmap fall
within the box of the `<img>` element, just draw the whole bitmap and
let Skia clip the draw-area to the correct rectangle.
This fixes a bug where the entire bitmap was squashed into the rectangle
of the image box instead of being clipped.
With this change, image rendering is now correct enough to import some
of the WPT tests for object-fit and object-position. To get some good
coverage I have imported all tests for the `<img>` tag. I also wanted to
import a subset of the tests for the `<object>` tag, since those are
passing as well now. Unfortunately, they are flaky for unknown reasons.
This is the second attempt at this bugfix. The prior one was e055927ead
and broke image rendering whenever the page was scrolled. It has
subsequently been reverted in 16b14273d1. Hopefully this time it is not
horribly broken.
Instead of trying to manually determine which parts of a bitmap fall
within the box of the `<img>` element, just draw the whole bitmap and
let Skia clip the draw-area to the correct rectangle.
This fixes a bug where the entire bitmap was squashed into the rectangle
of the image box instead of being clipped.
With this change, image rendering is now correct enough to import some
of the WPT tests for object-fit and object-position. To get some good
coverage I have imported all tests for the `<img>` tag. I also wanted to
import a subset of the tests for the `<object>` tag, since those are
passing as well now. Unfortunately, they are flaky for unknown reasons.
This reduces the number of `.cpp` files that need to be recompiled when
one of the below header files changes as follows:
Painting/Command.h: 1030 -> 61
Painting/DisplayList.h: 1030 -> 60
Painting/DisplayListRecorder.h: 557 -> 59
Resulting in a massive rename across almost everywhere! Alongside the
namespace change, we now have the following names:
* JS::NonnullGCPtr -> GC::Ref
* JS::GCPtr -> GC::Ptr
* JS::HeapFunction -> GC::Function
* JS::CellImpl -> GC::Cell
* JS::Handle -> GC::Root
Now that the heap has no knowledge about a JavaScript realm and is
purely for managing the memory of the heap, it does not make sense
to name this function to say that it is a non-realm variant.
