Keep decoded CSS text separate from tokenizer byte input. CSSOM and
already-decoded stylesheet text preserve code point preprocessing, so a
lone surrogate maps to one replacement character instead of being
re-decoded as malformed UTF-8 bytes.
Decode tokenizer byte input with the requested encoding unless that
encoding is UTF-8 and the byte stream is strictly valid UTF-8. Keep the
fast path by constructing the decoded string without validating twice
after strict validation succeeds.
Preserve UTF-8 decoder behavior on the byte fast path by stripping an
initial UTF-8 BOM and rejecting encoded surrogate bytes. Invalid UTF-8
still goes through the decoder. Add tokenizer coverage for both the C++
and Rust backends across decoded text, UTF-8 aliases, BOM-prefixed
input, invalid UTF-8, and non-UTF requested encodings.
When we invoke Tokenizer::consume_a_url_token, we have already consumed
the "url(" text from the source. Thus we would set the original source
text to the text starting just after those consumed code points. Let's
instead pass in the known starting position from the caller.
This fixes a bug seen on https://lichess.org, where they perform a
`substring(4)` on the property value to remove the "url(" text. This
would strip away the "http" part of the URL, and we would try to load
"s://lichess.org/image.svg". With this fixed, we can play chess games
on this site.
Add `create_foo()` static methods for the missing Token::Types, and use
them in the Tokenizer. This means we slightly deviate from the spec now:
it says "create foo token... set its bar to 32", but we now just wait
and construct the Token fully-formed. But those cases are short so it
should still be clear what we're doing.
This makes it possible to construct all kinds of Token elsewhere, such
as for testing purposes.
