Treat trailing UTF-8 prefixes with an invalid second byte as complete
input for streaming decode, so replacement characters are emitted in the
current chunk instead of being held until later input or finish. Keep
valid incomplete prefixes buffered across chunk boundaries.
Keep TextDecoderStream from holding continuation bytes after an invalid
lead byte at a chunk boundary. Add LibTextCodec and TextDecoderStream
coverage for invalid tails, valid split sequences, EOF partials,
surrogate sequences, and malformed continuation tails.
Reject UTF-8 second bytes outside the Encoding Standard's per-lead-byte
bounds before consuming the rest of each sequence. This keeps surrogate
and out-of-range sequences from collapsing multiple malformed bytes into
one replacement character.
Also report an odd trailing UTF-16 byte as U+FFFD through the streaming
code point path and route UTF-16 to_utf8() through the same logic. This
keeps lazy and eager script decoding aligned for bytecode cache source
hashes.
Cover the malformed UTF-8 and UTF-16 cases in LibTextCodec, TextDecoder,
and bytecode-cache source decoding tests.
Consume truncated UTF-8 tails as one malformed sequence while preserving
existing behavior for encoded surrogate code points. This keeps lazy
SourceCode decoding and bytecode cache source hashing aligned with eager
source text decoding for invalid cached script source bytes.
Continue stripping an initial UTF-8 byte order mark in
UTF8Decoder::to_utf8() so HTML parsing keeps matching the previous
String-based implementation. Cover the shared decoder behavior and the
TextDecoder API surface.
Decode malformed UTF-8 consistently in UTF8Decoder::process() and
UTF8Decoder::to_utf8(). This keeps lazy SourceCode decoding and bytecode
cache source hashing in step with eager source text decoding when cached
script source bytes contain invalid UTF-8.
Cover UTF-8 encoded surrogate code points and overlong byte sequences in
LibTextCodec, and add lazy SourceCode coverage for both cases.
Add Decoder::process_code_points() so callers can stream decoded code
points without first materializing a UTF-8 string. Implement the UTF-16
variants by walking code units directly and emitting replacement code
points for malformed surrogate pairs.
Introduce a StreamingDecoder wrapper that lets callers feed bytes to a
Decoder one chunk at a time. It buffers any incomplete trailing byte
sequence at the end of a chunk and prepends it to the next chunk, so a
multi-byte code point split across a chunk boundary is decoded correctly
once the next chunk arrives.
To support that, add an incomplete_tail_length() virtual on Decoder
returning the number of trailing bytes that form an incomplete sequence
per the Encoding Standard's decoder handler byte ranges, with overrides
for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP,
Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps
single-byte legacy decoders correct.
This is the foundation for the upcoming incremental HTML parser, which
needs to decode network response bodies as they arrive.
This is preparation for removing the endianness override, since it was
only used by a single client: LibTextCodec.
While here, add helpers and make use of simdutf for fast conversion.
There were two problems:
1. They didn't handle surrogates
2. They used signed chars, leading to eg 0x00e4 being treated as 0xffe4
Also add a basic test that catches both issues.
There's some code duplication with Utf16CodePointIterator::operator*(),
but let's get things working first.
Each of these strings would previously rely on StringView's char const*
constructor overload, which would call __builtin_strlen on the string.
Since we now have operator ""sv, we can replace these with much simpler
versions. This opens the door to being able to remove
StringView(char const*).
No functional changes.
