Store CSS token payloads in a variant so each token only carries the
state needed by its type. Keep delimiter, number, hash, string, and
dimension data separate instead of storing every possible payload on
every token.
Use a smaller component-value token for function and block boundary
metadata. These component values only need token type, original source
text, and source positions, so avoid embedding full token payload
storage inside every Function and SimpleBlock.
Shrink CSS source positions to explicit 32-bit counters. Guard the C++
and Rust tokenizer paths against overflow. Add size assertions for the
hot Token and ComponentValue types so future growth is intentional.
Now that Badge can have multiple types, and a Badge of a derived class
can convert into a Badge of the superclass, we can simplify a few method
signatures and overloads.
test-css-tokenizer is updated to run both the C++ and Rust tokenizers
and compare their output, to ensure they behave identically. The Parser
still uses the C++ Tokenizer.
The LibWeb crate, FFI layer etc are all based on the existing ones for
other libraries.
This is a direct AI translation to get us started, and not idiomatic
Rust. Future work can be done to make it more sensible.
This matches the behavior of other browsers. We did the equivalent
change for <integer> in b86377b
We continue to store these as doubles for the extra precision.
In a future commit we will be applying clamping to numeric (i.e.
number, percentage, dimension) values which will be done in the
appropriate `Token` getters so accessing the underlying number value
would be a potential footgun
Compare `Vector<Parser::ComponentValue>` directly instead of
serializing them into strings first.
This is required for the upcoming changes where we would compare
previous and new sets of custom properties to figure out whether we need
to invalidate descendant elements. Without this change `equals()` would
show up being hot in profiles.
Add `create_foo()` static methods for the missing Token::Types, and use
them in the Tokenizer. This means we slightly deviate from the spec now:
it says "create foo token... set its bar to 32", but we now just wait
and construct the Token fully-formed. But those cases are short so it
should still be clear what we're doing.
This makes it possible to construct all kinds of Token elsewhere, such
as for testing purposes.
