By default, `rustfmt` persists the import granularity. In practice, most
Rust code has import granularity "Module" due to LSP's actions.
"Item" gets rid of import groupings and achieves cleaner diffs and
better conflict resolution. Better greppability is a positive side
effect.
Note: it's an unstable rustfmt feature. `cargo +nightly fmt` must be
used instead of `cargo fmt`.
Split the single cbindgen::generate call into a multi-config setup
using cbindgen::Config::from_file and generate_with_config. Move
the namespace and export.include settings that were in cbindgen.toml
into the build.rs CSS config. No functional change — the generated
RustFFI.h is identical.
This prepares for adding a second cbindgen config for the
encoding-detection FFI symbols in a follow-up commit.
Replace the C++ HTML tokenizer with a Rust implementation behind the
existing HTMLTokenizer API.
Keep the parser-facing integration points for streaming input,
insertion points, document.write(), EOF insertion, parser aborts,
speculative parser input, and last start tag tracking. The generated
FFI handle stays an implementation detail of HTMLTokenizer, so callers
keep a single tokenizer class.
Preserve duplicate attributes through FFI so C++ token normalization can
record the duplicate-attribute signal used by CSP nonce checks. Keep
bulk tag-name and attribute scans capped at the active insertion point
so streamed parser input is spliced at the right offset.
Use generated DAFSA tables for named character references and intern
common tag and attribute names to reduce FFI marshalling overhead. This
also fixes attribute name source positions, nested old insertion points,
and aborted fast-path handling.
TestHTMLTokenizer covers duplicate attributes and insertion points in
fast tag-name, attribute-name, and quoted-value scans. A CSP text test
covers duplicate nonce attributes on parser-created script elements.
The tokenizer dump fixtures still match, TestHTMLTokenizer passes, and
the full release test-web run passes with 6981 tests and 226 skipped.
Avoid building a temporary Rust token vector before calling back into
C++. The tokenizer now invokes the callback as each token is produced,
while borrowing the already-filtered input for source slices.
Reserve an initial C++ token capacity from the input size so the common
path avoids repeated growth while appending the converted tokens.
With this change, the Rust CSS tokenizer is now ~1.3x faster than the
C++ CSS tokenizer at churning through all the https://vercel.com/ CSS.
test-css-tokenizer is updated to run both the C++ and Rust tokenizers
and compare their output, to ensure they behave identically. The Parser
still uses the C++ Tokenizer.
The LibWeb crate, FFI layer etc are all based on the existing ones for
other libraries.
This is a direct AI translation to get us started, and not idiomatic
Rust. Future work can be done to make it more sensible.
