Switch bytecode cache source identity from decoded UTF-16 source text to
the original encoded response bytes plus the effective source encoding.
Store the decoded source length in the cache blob header so warm loads
can build lazy SourceCode objects without decoding the source before
checking the sidecar.
This removes the main-thread decoded_source_text_info pass from valid
warm-cache script and module loads. The source is only decoded on cache
miss, or when a rejected sidecar falls back to source compilation.
Add Decoder::process_code_points() so callers can stream decoded code
points without first materializing a UTF-8 string. Implement the UTF-16
variants by walking code units directly and emitting replacement code
points for malformed surrogate pairs.
Introduce a StreamingDecoder wrapper that lets callers feed bytes to a
Decoder one chunk at a time. It buffers any incomplete trailing byte
sequence at the end of a chunk and prepends it to the next chunk, so a
multi-byte code point split across a chunk boundary is decoded correctly
once the next chunk arrives.
To support that, add an incomplete_tail_length() virtual on Decoder
returning the number of trailing bytes that form an incomplete sequence
per the Encoding Standard's decoder handler byte ranges, with overrides
for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP,
Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps
single-byte legacy decoders correct.
This is the foundation for the upcoming incremental HTML parser, which
needs to decode network response bodies as they arrive.
This is preparation for removing the endianness override, since it was
only used by a single client: LibTextCodec.
While here, add helpers and make use of simdutf for fast conversion.
We're starting with a very basic decoding API and only ISO-8859-1 and
UTF-8 decoding (and UTF-8 decoding is really a no-op since String is
expected to be UTF-8.)
