Commit graph

5 commits

Author SHA1 Message Date
Andreas Kling
01cec162c8 LibTextCodec: Stop buffering invalid UTF-8 tails
Treat trailing UTF-8 prefixes with an invalid second byte as complete
input for streaming decode, so replacement characters are emitted in the
current chunk instead of being held until later input or finish. Keep
valid incomplete prefixes buffered across chunk boundaries.

Keep TextDecoderStream from holding continuation bytes after an invalid
lead byte at a chunk boundary. Add LibTextCodec and TextDecoderStream
coverage for invalid tails, valid split sequences, EOF partials,
surrogate sequences, and malformed continuation tails.
2026-05-18 14:08:22 +02:00
Andreas Kling
45da0e4a0e LibTextCodec: Preserve malformed decoder replacements
Reject UTF-8 second bytes outside the Encoding Standard's per-lead-byte
bounds before consuming the rest of each sequence. This keeps surrogate
and out-of-range sequences from collapsing multiple malformed bytes into
one replacement character.

Also report an odd trailing UTF-16 byte as U+FFFD through the streaming
code point path and route UTF-16 to_utf8() through the same logic. This
keeps lazy and eager script decoding aligned for bytecode cache source
hashes.

Cover the malformed UTF-8 and UTF-16 cases in LibTextCodec, TextDecoder,
and bytecode-cache source decoding tests.
2026-05-18 14:08:22 +02:00
Andreas Kling
c92079893a LibTextCodec: Match UTF-8 replacement handling
Consume truncated UTF-8 tails as one malformed sequence while preserving
existing behavior for encoded surrogate code points. This keeps lazy
SourceCode decoding and bytecode cache source hashing aligned with eager
source text decoding for invalid cached script source bytes.

Continue stripping an initial UTF-8 byte order mark in
UTF8Decoder::to_utf8() so HTML parsing keeps matching the previous
String-based implementation. Cover the shared decoder behavior and the
TextDecoder API surface.
2026-05-18 09:18:35 +02:00
Timothy Flynn
01ebf1eb07 AK: Replace surrogates in String::from_utf8_with_replacement_character
We are expected to replace lonely surrogates with U+FFFD when decoding
UTF-8 text.
2025-07-06 04:30:17 +12:00
Bastiaan van der Plaat
0104225d9b LibWeb: Add TextEncoder encodeInto 2023-10-29 21:44:53 +01:00