dmaax/ladybird
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ladybird/Tests/LibTextCodec/TestTextDecoders.cpp

251 lines
12 KiB
C++
Raw Permalink Normal View History

Tests: Add a basic UTF-8 to UTF-8 LibTextCodec test
2022-03-27 16:41:51 -03:00
/*
* Copyright (c) 2022, the SerenityOS developers.
*
* SPDX-License-Identifier: BSD-2-Clause
*/
LibTextCodec+Everywhere: Port Decoders to new Strings
2023-02-17 17:15:10 -03:00
#include <AK/String.h>
Tests: Add a basic UTF-8 to UTF-8 LibTextCodec test
2022-03-27 16:41:51 -03:00
#include <AK/Vector.h>
#include <LibTest/TestCase.h>
#include <LibTextCodec/Decoder.h>
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
static ReadonlyBytes bytes(Vector<u8> const& data)
{
return data.span();
}
static TextCodec::Decoder& decoder_for(StringView encoding)
{
auto decoder = TextCodec::decoder_for(encoding);
VERIFY(decoder.has_value());
return decoder.value();
}
LibTextCodec: Allow processing decoded UTF-16 code points Add Decoder::process_code_points() so callers can stream decoded code points without first materializing a UTF-8 string. Implement the UTF-16 variants by walking code units directly and emitting replacement code points for malformed surrogate pairs.
2026-05-17 14:05:53 -03:00
static Vector<u32> process_code_points(TextCodec::Decoder& decoder, StringView input)
{
Vector<u32> code_points;
MUST(decoder.process_code_points(input, [&](u32 code_point) -> ErrorOr<void> {
TRY(code_points.try_append(code_point));
return {};
}));
return code_points;
}
Tests: Add a basic UTF-8 to UTF-8 LibTextCodec test
2022-03-27 16:41:51 -03:00
TEST_CASE(test_utf8_decode)
{
Tests/LibTextCodec: Do not refer to concrete encoders/decoders Which allows for their removal from the header file once the implementation is ported to rust.
2026-06-19 18:47:01 -03:00
auto& decoder = decoder_for("UTF-8"sv);
Tests: Add a basic UTF-8 to UTF-8 LibTextCodec test
2022-03-27 16:41:51 -03:00
// Bytes for U+1F600 GRINNING FACE
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 14:32:29 -03:00
auto test_string = "\xf0\x9f\x98\x80"sv;
Tests: Add a basic UTF-8 to UTF-8 LibTextCodec test
2022-03-27 16:41:51 -03:00
LibTextCodec: Remove decoder validation API All users instead rely on the 'Fatal' option being passed through.
2026-06-22 16:58:47 -03:00
EXPECT(!decoder.to_utf8(test_string, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Fatal).is_error());
LibTextCodec: Use AK facilities to validate and convert UTF-16 to UTF-8 This allows LibTextCodec to make use of simdutf, and also reduces the number of places with manual UTF-16 implementations.
2024-07-18 13:07:37 -03:00
Tests/LibTextCodec: Do not refer to concrete encoders/decoders Which allows for their removal from the header file once the implementation is ported to rust.
2026-06-19 18:47:01 -03:00
auto processed_code_points = process_code_points(decoder, test_string);
Tests: Add a basic UTF-8 to UTF-8 LibTextCodec test
2022-03-27 16:41:51 -03:00
EXPECT(processed_code_points.size() == 1);
EXPECT(processed_code_points[0] == 0x1F600);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT(MUST(decoder.to_utf8(test_string, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)) == test_string);
Tests: Add a basic UTF-8 to UTF-8 LibTextCodec test
2022-03-27 16:41:51 -03:00
}
LibTextCodec: Make utf-16be and utf-16le codecs actually work There were two problems: 1. They didn't handle surrogates 2. They used signed chars, leading to eg 0x00e4 being treated as 0xffe4 Also add a basic test that catches both issues. There's some code duplication with Utf16CodePointIterator::operator*(), but let's get things working first.
2023-01-22 14:59:00 -03:00
LibTextCodec: Allow processing decoded UTF-16 code points Add Decoder::process_code_points() so callers can stream decoded code points without first materializing a UTF-8 string. Implement the UTF-16 variants by walking code units directly and emitting replacement code points for malformed surrogate pairs.
2026-05-17 14:05:53 -03:00
TEST_CASE(test_utf8_process_code_points)
{
Tests/LibTextCodec: Do not refer to concrete encoders/decoders Which allows for their removal from the header file once the implementation is ported to rust.
2026-06-19 18:47:01 -03:00
auto& decoder = decoder_for("UTF-8"sv);
LibTextCodec: Allow processing decoded UTF-16 code points Add Decoder::process_code_points() so callers can stream decoded code points without first materializing a UTF-8 string. Implement the UTF-16 variants by walking code units directly and emitting replacement code points for malformed surrogate pairs.
2026-05-17 14:05:53 -03:00
auto test_string = "A\xf0\x9f\x98\x80"sv;
EXPECT_EQ(process_code_points(decoder, test_string), (Vector<u32> { 0x41, 0x1F600 }));
}
LibTextCodec: Preserve UTF-8 replacement decoding Decode malformed UTF-8 consistently in UTF8Decoder::process() and UTF8Decoder::to_utf8(). This keeps lazy SourceCode decoding and bytecode cache source hashing in step with eager source text decoding when cached script source bytes contain invalid UTF-8. Cover UTF-8 encoded surrogate code points and overlong byte sequences in LibTextCodec, and add lazy SourceCode coverage for both cases.
2026-05-17 16:09:29 -03:00
TEST_CASE(test_utf8_process_code_points_replaces_surrogates)
{
Tests/LibTextCodec: Do not refer to concrete encoders/decoders Which allows for their removal from the header file once the implementation is ported to rust.
2026-06-19 18:47:01 -03:00
auto& decoder = decoder_for("UTF-8"sv);
LibTextCodec: Preserve UTF-8 replacement decoding Decode malformed UTF-8 consistently in UTF8Decoder::process() and UTF8Decoder::to_utf8(). This keeps lazy SourceCode decoding and bytecode cache source hashing in step with eager source text decoding when cached script source bytes contain invalid UTF-8. Cover UTF-8 encoded surrogate code points and overlong byte sequences in LibTextCodec, and add lazy SourceCode coverage for both cases.
2026-05-17 16:09:29 -03:00
auto utf8_encoded_surrogate_bytes = Vector<u8> { 0xed, 0xa0, 0x80 };
auto utf8_encoded_surrogate = StringView(bytes(utf8_encoded_surrogate_bytes));
LibTextCodec: Remove decoder validation API All users instead rely on the 'Fatal' option being passed through.
2026-06-22 16:58:47 -03:00
EXPECT(decoder.to_utf8(utf8_encoded_surrogate, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Fatal).is_error());
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(decoder.to_utf8(utf8_encoded_surrogate, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)), "\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd"sv);
LibTextCodec: Preserve malformed decoder replacements Reject UTF-8 second bytes outside the Encoding Standard's per-lead-byte bounds before consuming the rest of each sequence. This keeps surrogate and out-of-range sequences from collapsing multiple malformed bytes into one replacement character. Also report an odd trailing UTF-16 byte as U+FFFD through the streaming code point path and route UTF-16 to_utf8() through the same logic. This keeps lazy and eager script decoding aligned for bytecode cache source hashes. Cover the malformed UTF-8 and UTF-16 cases in LibTextCodec, TextDecoder, and bytecode-cache source decoding tests.
2026-05-18 07:53:22 -03:00
EXPECT_EQ(process_code_points(decoder, utf8_encoded_surrogate), (Vector<u32> { 0xfffd, 0xfffd, 0xfffd }));
LibTextCodec: Preserve UTF-8 replacement decoding Decode malformed UTF-8 consistently in UTF8Decoder::process() and UTF8Decoder::to_utf8(). This keeps lazy SourceCode decoding and bytecode cache source hashing in step with eager source text decoding when cached script source bytes contain invalid UTF-8. Cover UTF-8 encoded surrogate code points and overlong byte sequences in LibTextCodec, and add lazy SourceCode coverage for both cases.
2026-05-17 16:09:29 -03:00
}
LibTextCodec: Match UTF-8 replacement handling Consume truncated UTF-8 tails as one malformed sequence while preserving existing behavior for encoded surrogate code points. This keeps lazy SourceCode decoding and bytecode cache source hashing aligned with eager source text decoding for invalid cached script source bytes. Continue stripping an initial UTF-8 byte order mark in UTF8Decoder::to_utf8() so HTML parsing keeps matching the previous String-based implementation. Cover the shared decoder behavior and the TextDecoder API surface.
2026-05-17 20:47:40 -03:00
TEST_CASE(test_utf8_process_code_points_replaces_truncated_tail_as_single_error)
{
Tests/LibTextCodec: Do not refer to concrete encoders/decoders Which allows for their removal from the header file once the implementation is ported to rust.
2026-06-19 18:47:01 -03:00
auto& decoder = decoder_for("UTF-8"sv);
LibTextCodec: Match UTF-8 replacement handling Consume truncated UTF-8 tails as one malformed sequence while preserving existing behavior for encoded surrogate code points. This keeps lazy SourceCode decoding and bytecode cache source hashing aligned with eager source text decoding for invalid cached script source bytes. Continue stripping an initial UTF-8 byte order mark in UTF8Decoder::to_utf8() so HTML parsing keeps matching the previous String-based implementation. Cover the shared decoder behavior and the TextDecoder API surface.
2026-05-17 20:47:40 -03:00
auto truncated_tail_bytes = Vector<u8> { 0xf0, 0x9f, 0x98 };
auto truncated_tail = StringView(bytes(truncated_tail_bytes));
LibTextCodec: Remove decoder validation API All users instead rely on the 'Fatal' option being passed through.
2026-06-22 16:58:47 -03:00
EXPECT(decoder.to_utf8(truncated_tail, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Fatal).is_error());
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(decoder.to_utf8(truncated_tail, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)), "\xef\xbf\xbd"sv);
LibTextCodec: Match UTF-8 replacement handling Consume truncated UTF-8 tails as one malformed sequence while preserving existing behavior for encoded surrogate code points. This keeps lazy SourceCode decoding and bytecode cache source hashing aligned with eager source text decoding for invalid cached script source bytes. Continue stripping an initial UTF-8 byte order mark in UTF8Decoder::to_utf8() so HTML parsing keeps matching the previous String-based implementation. Cover the shared decoder behavior and the TextDecoder API surface.
2026-05-17 20:47:40 -03:00
EXPECT_EQ(process_code_points(decoder, truncated_tail), (Vector<u32> { 0xfffd }));
}
LibTextCodec: Preserve UTF-8 replacement decoding Decode malformed UTF-8 consistently in UTF8Decoder::process() and UTF8Decoder::to_utf8(). This keeps lazy SourceCode decoding and bytecode cache source hashing in step with eager source text decoding when cached script source bytes contain invalid UTF-8. Cover UTF-8 encoded surrogate code points and overlong byte sequences in LibTextCodec, and add lazy SourceCode coverage for both cases.
2026-05-17 16:09:29 -03:00
TEST_CASE(test_utf8_process_code_points_replaces_overlong_sequences)
{
Tests/LibTextCodec: Do not refer to concrete encoders/decoders Which allows for their removal from the header file once the implementation is ported to rust.
2026-06-19 18:47:01 -03:00
auto& decoder = decoder_for("UTF-8"sv);
LibTextCodec: Preserve UTF-8 replacement decoding Decode malformed UTF-8 consistently in UTF8Decoder::process() and UTF8Decoder::to_utf8(). This keeps lazy SourceCode decoding and bytecode cache source hashing in step with eager source text decoding when cached script source bytes contain invalid UTF-8. Cover UTF-8 encoded surrogate code points and overlong byte sequences in LibTextCodec, and add lazy SourceCode coverage for both cases.
2026-05-17 16:09:29 -03:00
auto overlong_null_bytes = Vector<u8> { 0xc0, 0x80 };
auto overlong_null = StringView(bytes(overlong_null_bytes));
LibTextCodec: Remove decoder validation API All users instead rely on the 'Fatal' option being passed through.
2026-06-22 16:58:47 -03:00
EXPECT(decoder.to_utf8(overlong_null, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Fatal).is_error());
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(decoder.to_utf8(overlong_null, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)), "\xef\xbf\xbd\xef\xbf\xbd"sv);
LibTextCodec: Preserve UTF-8 replacement decoding Decode malformed UTF-8 consistently in UTF8Decoder::process() and UTF8Decoder::to_utf8(). This keeps lazy SourceCode decoding and bytecode cache source hashing in step with eager source text decoding when cached script source bytes contain invalid UTF-8. Cover UTF-8 encoded surrogate code points and overlong byte sequences in LibTextCodec, and add lazy SourceCode coverage for both cases.
2026-05-17 16:09:29 -03:00
EXPECT_EQ(process_code_points(decoder, overlong_null), (Vector<u32> { 0xfffd, 0xfffd }));
}
LibTextCodec: Preserve malformed decoder replacements Reject UTF-8 second bytes outside the Encoding Standard's per-lead-byte bounds before consuming the rest of each sequence. This keeps surrogate and out-of-range sequences from collapsing multiple malformed bytes into one replacement character. Also report an odd trailing UTF-16 byte as U+FFFD through the streaming code point path and route UTF-16 to_utf8() through the same logic. This keeps lazy and eager script decoding aligned for bytecode cache source hashes. Cover the malformed UTF-8 and UTF-16 cases in LibTextCodec, TextDecoder, and bytecode-cache source decoding tests.
2026-05-18 07:53:22 -03:00
TEST_CASE(test_utf8_process_code_points_restores_invalid_second_byte)
{
Tests/LibTextCodec: Do not refer to concrete encoders/decoders Which allows for their removal from the header file once the implementation is ported to rust.
2026-06-19 18:47:01 -03:00
auto& decoder = decoder_for("UTF-8"sv);
LibTextCodec: Preserve malformed decoder replacements Reject UTF-8 second bytes outside the Encoding Standard's per-lead-byte bounds before consuming the rest of each sequence. This keeps surrogate and out-of-range sequences from collapsing multiple malformed bytes into one replacement character. Also report an odd trailing UTF-16 byte as U+FFFD through the streaming code point path and route UTF-16 to_utf8() through the same logic. This keeps lazy and eager script decoding aligned for bytecode cache source hashes. Cover the malformed UTF-8 and UTF-16 cases in LibTextCodec, TextDecoder, and bytecode-cache source decoding tests.
2026-05-18 07:53:22 -03:00
auto overlong_three_byte_sequence_bytes = Vector<u8> { 0xe0, 0x80, 0x80 };
auto overlong_three_byte_sequence = StringView(bytes(overlong_three_byte_sequence_bytes));
auto out_of_range_four_byte_sequence_bytes = Vector<u8> { 0xf4, 0x90, 0x80, 0x80 };
auto out_of_range_four_byte_sequence = StringView(bytes(out_of_range_four_byte_sequence_bytes));
LibTextCodec: Remove decoder validation API All users instead rely on the 'Fatal' option being passed through.
2026-06-22 16:58:47 -03:00
EXPECT(decoder.to_utf8(overlong_three_byte_sequence, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Fatal).is_error());
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(decoder.to_utf8(overlong_three_byte_sequence, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)), "\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd"sv);
LibTextCodec: Preserve malformed decoder replacements Reject UTF-8 second bytes outside the Encoding Standard's per-lead-byte bounds before consuming the rest of each sequence. This keeps surrogate and out-of-range sequences from collapsing multiple malformed bytes into one replacement character. Also report an odd trailing UTF-16 byte as U+FFFD through the streaming code point path and route UTF-16 to_utf8() through the same logic. This keeps lazy and eager script decoding aligned for bytecode cache source hashes. Cover the malformed UTF-8 and UTF-16 cases in LibTextCodec, TextDecoder, and bytecode-cache source decoding tests.
2026-05-18 07:53:22 -03:00
EXPECT_EQ(process_code_points(decoder, overlong_three_byte_sequence), (Vector<u32> { 0xfffd, 0xfffd, 0xfffd }));
LibTextCodec: Remove decoder validation API All users instead rely on the 'Fatal' option being passed through.
2026-06-22 16:58:47 -03:00
EXPECT(decoder.to_utf8(out_of_range_four_byte_sequence, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Fatal).is_error());
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(decoder.to_utf8(out_of_range_four_byte_sequence, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)), "\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd"sv);
LibTextCodec: Preserve malformed decoder replacements Reject UTF-8 second bytes outside the Encoding Standard's per-lead-byte bounds before consuming the rest of each sequence. This keeps surrogate and out-of-range sequences from collapsing multiple malformed bytes into one replacement character. Also report an odd trailing UTF-16 byte as U+FFFD through the streaming code point path and route UTF-16 to_utf8() through the same logic. This keeps lazy and eager script decoding aligned for bytecode cache source hashes. Cover the malformed UTF-8 and UTF-16 cases in LibTextCodec, TextDecoder, and bytecode-cache source decoding tests.
2026-05-18 07:53:22 -03:00
EXPECT_EQ(process_code_points(decoder, out_of_range_four_byte_sequence), (Vector<u32> { 0xfffd, 0xfffd, 0xfffd, 0xfffd }));
}
LibTextCodec: Make utf-16be and utf-16le codecs actually work There were two problems: 1. They didn't handle surrogates 2. They used signed chars, leading to eg 0x00e4 being treated as 0xffe4 Also add a basic test that catches both issues. There's some code duplication with Utf16CodePointIterator::operator*(), but let's get things working first.
2023-01-22 14:59:00 -03:00
TEST_CASE(test_utf16be_decode)
{
Tests/LibTextCodec: Do not refer to concrete encoders/decoders Which allows for their removal from the header file once the implementation is ported to rust.
2026-06-19 18:47:01 -03:00
auto& decoder = decoder_for("UTF-16BE"sv);
LibTextCodec: Make utf-16be and utf-16le codecs actually work There were two problems: 1. They didn't handle surrogates 2. They used signed chars, leading to eg 0x00e4 being treated as 0xffe4 Also add a basic test that catches both issues. There's some code duplication with Utf16CodePointIterator::operator*(), but let's get things working first.
2023-01-22 14:59:00 -03:00
// This is the output of `python3 -c "print('säk😀'.encode('utf-16be'))"`.
auto test_string = "\x00s\x00\xe4\x00k\xd8=\xde\x00"sv;
LibTextCodec: Remove decoder validation API All users instead rely on the 'Fatal' option being passed through.
2026-06-22 16:58:47 -03:00
auto utf8 = MUST(decoder.to_utf8(test_string, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Fatal));
LibTextCodec: Use AK facilities to validate and convert UTF-16 to UTF-8 This allows LibTextCodec to make use of simdutf, and also reduces the number of places with manual UTF-16 implementations.
2024-07-18 13:07:37 -03:00
EXPECT_EQ(utf8, "säk😀"sv);
LibTextCodec: Make utf-16be and utf-16le codecs actually work There were two problems: 1. They didn't handle surrogates 2. They used signed chars, leading to eg 0x00e4 being treated as 0xffe4 Also add a basic test that catches both issues. There's some code duplication with Utf16CodePointIterator::operator*(), but let's get things working first.
2023-01-22 14:59:00 -03:00
}
LibTextCodec: Allow processing decoded UTF-16 code points Add Decoder::process_code_points() so callers can stream decoded code points without first materializing a UTF-8 string. Implement the UTF-16 variants by walking code units directly and emitting replacement code points for malformed surrogate pairs.
2026-05-17 14:05:53 -03:00
TEST_CASE(test_utf16be_process_code_points)
{
Tests/LibTextCodec: Do not refer to concrete encoders/decoders Which allows for their removal from the header file once the implementation is ported to rust.
2026-06-19 18:47:01 -03:00
auto& decoder = decoder_for("UTF-16BE"sv);
LibTextCodec: Allow processing decoded UTF-16 code points Add Decoder::process_code_points() so callers can stream decoded code points without first materializing a UTF-8 string. Implement the UTF-16 variants by walking code units directly and emitting replacement code points for malformed surrogate pairs.
2026-05-17 14:05:53 -03:00
EXPECT_EQ(process_code_points(decoder, StringView(bytes({ 0xfe, 0xff, 0x00, 'A', 0xd8, 0x3d, 0xde, 0x00 }))), (Vector<u32> { 0x41, 0x1F600 }));
EXPECT_EQ(process_code_points(decoder, StringView(bytes({ 0xd8, 0x3d, 0x00, 'A', 0xde, 0x00 }))), (Vector<u32> { 0xfffd, 0x41, 0xfffd }));
LibTextCodec: Preserve malformed decoder replacements Reject UTF-8 second bytes outside the Encoding Standard's per-lead-byte bounds before consuming the rest of each sequence. This keeps surrogate and out-of-range sequences from collapsing multiple malformed bytes into one replacement character. Also report an odd trailing UTF-16 byte as U+FFFD through the streaming code point path and route UTF-16 to_utf8() through the same logic. This keeps lazy and eager script decoding aligned for bytecode cache source hashes. Cover the malformed UTF-8 and UTF-16 cases in LibTextCodec, TextDecoder, and bytecode-cache source decoding tests.
2026-05-18 07:53:22 -03:00
EXPECT_EQ(process_code_points(decoder, StringView(bytes({ 0x00, 'A', 0xff }))), (Vector<u32> { 0x41, 0xfffd }));
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(decoder.to_utf8(StringView(bytes({ 0x00, 'A', 0xff })), TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)), "A\xef\xbf\xbd"sv);
LibTextCodec: Allow processing decoded UTF-16 code points Add Decoder::process_code_points() so callers can stream decoded code points without first materializing a UTF-8 string. Implement the UTF-16 variants by walking code units directly and emitting replacement code points for malformed surrogate pairs.
2026-05-17 14:05:53 -03:00
}
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
TEST_CASE(test_streaming_decoder_utf8_mid_sequence)
{
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder = TextCodec::StreamingDecoder { "UTF-8"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 'a', 0xc3 }))), "a"sv);
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0xa9, 'b' }))), "éb"sv);
EXPECT_EQ(MUST(streaming_decoder.finish()), ""sv);
}
TEST_CASE(test_streaming_decoder_finishes_incomplete_sequence)
{
auto& decoder = decoder_for("UTF-8"sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder = TextCodec::StreamingDecoder { "UTF-8"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
auto incomplete_sequence = Vector<u8> { 0xc3 };
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes(incomplete_sequence))), ""sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(streaming_decoder.finish()), MUST(decoder.to_utf8(StringView(bytes(incomplete_sequence)), TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)));
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
}
LibTextCodec: Stop buffering invalid UTF-8 tails Treat trailing UTF-8 prefixes with an invalid second byte as complete input for streaming decode, so replacement characters are emitted in the current chunk instead of being held until later input or finish. Keep valid incomplete prefixes buffered across chunk boundaries. Keep TextDecoderStream from holding continuation bytes after an invalid lead byte at a chunk boundary. Add LibTextCodec and TextDecoderStream coverage for invalid tails, valid split sequences, EOF partials, surrogate sequences, and malformed continuation tails.
2026-05-18 07:24:35 -03:00
TEST_CASE(test_streaming_decoder_utf8_invalid_second_byte_tail)
{
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder_for_overlong_three_byte_sequence = TextCodec::StreamingDecoder { "UTF-8"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Stop buffering invalid UTF-8 tails Treat trailing UTF-8 prefixes with an invalid second byte as complete input for streaming decode, so replacement characters are emitted in the current chunk instead of being held until later input or finish. Keep valid incomplete prefixes buffered across chunk boundaries. Keep TextDecoderStream from holding continuation bytes after an invalid lead byte at a chunk boundary. Add LibTextCodec and TextDecoderStream coverage for invalid tails, valid split sequences, EOF partials, surrogate sequences, and malformed continuation tails.
2026-05-18 07:24:35 -03:00
EXPECT_EQ(MUST(streaming_decoder_for_overlong_three_byte_sequence.to_utf8(bytes({ 0xe0, 0x80 }))), "\xef\xbf\xbd\xef\xbf\xbd"sv);
EXPECT_EQ(MUST(streaming_decoder_for_overlong_three_byte_sequence.finish()), ""sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder_for_out_of_range_four_byte_sequence = TextCodec::StreamingDecoder { "UTF-8"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Stop buffering invalid UTF-8 tails Treat trailing UTF-8 prefixes with an invalid second byte as complete input for streaming decode, so replacement characters are emitted in the current chunk instead of being held until later input or finish. Keep valid incomplete prefixes buffered across chunk boundaries. Keep TextDecoderStream from holding continuation bytes after an invalid lead byte at a chunk boundary. Add LibTextCodec and TextDecoderStream coverage for invalid tails, valid split sequences, EOF partials, surrogate sequences, and malformed continuation tails.
2026-05-18 07:24:35 -03:00
EXPECT_EQ(MUST(streaming_decoder_for_out_of_range_four_byte_sequence.to_utf8(bytes({ 0xf4, 0x90 }))), "\xef\xbf\xbd\xef\xbf\xbd"sv);
EXPECT_EQ(MUST(streaming_decoder_for_out_of_range_four_byte_sequence.finish()), ""sv);
}
TEST_CASE(test_streaming_decoder_utf8_valid_second_byte_tail)
{
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder = TextCodec::StreamingDecoder { "UTF-8"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Stop buffering invalid UTF-8 tails Treat trailing UTF-8 prefixes with an invalid second byte as complete input for streaming decode, so replacement characters are emitted in the current chunk instead of being held until later input or finish. Keep valid incomplete prefixes buffered across chunk boundaries. Keep TextDecoderStream from holding continuation bytes after an invalid lead byte at a chunk boundary. Add LibTextCodec and TextDecoderStream coverage for invalid tails, valid split sequences, EOF partials, surrogate sequences, and malformed continuation tails.
2026-05-18 07:24:35 -03:00
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0xe0, 0xa0 }))), ""sv);
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0x80 }))), "\xe0\xa0\x80"sv);
EXPECT_EQ(MUST(streaming_decoder.finish()), ""sv);
}
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
TEST_CASE(test_streaming_decoder_utf16_odd_byte)
{
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder = TextCodec::StreamingDecoder { "UTF-16LE"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0x41, 0x00, 0x42 }))), "A"sv);
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0x00 }))), "B"sv);
EXPECT_EQ(MUST(streaming_decoder.finish()), ""sv);
}
TEST_CASE(test_streaming_decoder_utf16_surrogate_pair_split)
{
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder = TextCodec::StreamingDecoder { "UTF-16BE"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0x00, 0x41, 0xd8, 0x3d }))), "A"sv);
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0xde, 0x00 }))), "😀"sv);
EXPECT_EQ(MUST(streaming_decoder.finish()), ""sv);
}
TEST_CASE(test_streaming_decoder_gb18030_four_byte_tail)
{
auto& decoder = decoder_for("gb18030"sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder = TextCodec::StreamingDecoder { "gb18030"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
auto gb18030_sequence = Vector<u8> { 0x81, 0x30, 0x81, 0x30 };
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 'a', 0x81, 0x30, 0x81 }))), "a"sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0x30 }))), MUST(decoder.to_utf8(StringView(bytes(gb18030_sequence)), TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)));
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
EXPECT_EQ(MUST(streaming_decoder.finish()), ""sv);
}
TEST_CASE(test_streaming_decoder_big5_overlapping_trail)
{
auto& decoder = decoder_for("Big5"sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder = TextCodec::StreamingDecoder { "Big5"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
auto big5_sequence = Vector<u8> { 0xa4, 0xa4 };
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes(big5_sequence))), MUST(decoder.to_utf8(StringView(bytes(big5_sequence)), TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)));
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
EXPECT_EQ(MUST(streaming_decoder.finish()), ""sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder_for_split_input = TextCodec::StreamingDecoder { "Big5"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
EXPECT_EQ(MUST(streaming_decoder_for_split_input.to_utf8(bytes({ 0xa4 }))), ""sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(streaming_decoder_for_split_input.to_utf8(bytes({ 0xa4 }))), MUST(decoder.to_utf8(StringView(bytes(big5_sequence)), TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)));
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
EXPECT_EQ(MUST(streaming_decoder_for_split_input.finish()), ""sv);
}
TEST_CASE(test_streaming_decoder_euc_jp_three_byte_tail)
{
auto& decoder = decoder_for("EUC-JP"sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder = TextCodec::StreamingDecoder { "EUC-JP"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
auto euc_jp_sequence = Vector<u8> { 0x8f, 0xa2, 0xaf };
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0x8f, 0xa2 }))), ""sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0xaf }))), MUST(decoder.to_utf8(StringView(bytes(euc_jp_sequence)), TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)));
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
EXPECT_EQ(MUST(streaming_decoder.finish()), ""sv);
}
TEST_CASE(test_streaming_decoder_shift_jis_tail)
{
auto& decoder = decoder_for("Shift_JIS"sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
auto streaming_decoder = TextCodec::StreamingDecoder { "Shift_JIS"sv, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement };
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
auto shift_jis_sequence = Vector<u8> { 0x82, 0xa0 };
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0x82 }))), ""sv);
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(streaming_decoder.to_utf8(bytes({ 0xa0 }))), MUST(decoder.to_utf8(StringView(bytes(shift_jis_sequence)), TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)));
LibTextCodec: Add streaming decoder Introduce a StreamingDecoder wrapper that lets callers feed bytes to a Decoder one chunk at a time. It buffers any incomplete trailing byte sequence at the end of a chunk and prepends it to the next chunk, so a multi-byte code point split across a chunk boundary is decoded correctly once the next chunk arrives. To support that, add an incomplete_tail_length() virtual on Decoder returning the number of trailing bytes that form an incomplete sequence per the Encoding Standard's decoder handler byte ranges, with overrides for UTF-8, UTF-16BE, UTF-16LE, GB18030, Big5, EUC-JP, ISO-2022-JP, Shift_JIS, and EUC-KR. The default implementation returns 0, which keeps single-byte legacy decoders correct. This is the foundation for the upcoming incremental HTML parser, which needs to decode network response bodies as they arrive.
2026-04-27 15:00:25 -03:00
EXPECT_EQ(MUST(streaming_decoder.finish()), ""sv);
}
LibTextCodec: Make utf-16be and utf-16le codecs actually work There were two problems: 1. They didn't handle surrogates 2. They used signed chars, leading to eg 0x00e4 being treated as 0xffe4 Also add a basic test that catches both issues. There's some code duplication with Utf16CodePointIterator::operator*(), but let's get things working first.
2023-01-22 14:59:00 -03:00
TEST_CASE(test_utf16le_decode)
{
Tests/LibTextCodec: Do not refer to concrete encoders/decoders Which allows for their removal from the header file once the implementation is ported to rust.
2026-06-19 18:47:01 -03:00
auto& decoder = decoder_for("UTF-16LE"sv);
LibTextCodec: Make utf-16be and utf-16le codecs actually work There were two problems: 1. They didn't handle surrogates 2. They used signed chars, leading to eg 0x00e4 being treated as 0xffe4 Also add a basic test that catches both issues. There's some code duplication with Utf16CodePointIterator::operator*(), but let's get things working first.
2023-01-22 14:59:00 -03:00
// This is the output of `python3 -c "print('säk😀'.encode('utf-16le'))"`.
auto test_string = "s\x00\xe4\x00k\x00=\xd8\x00\xde"sv;
LibTextCodec: Remove decoder validation API All users instead rely on the 'Fatal' option being passed through.
2026-06-22 16:58:47 -03:00
auto utf8 = MUST(decoder.to_utf8(test_string, TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Fatal));
LibTextCodec: Use AK facilities to validate and convert UTF-16 to UTF-8 This allows LibTextCodec to make use of simdutf, and also reduces the number of places with manual UTF-16 implementations.
2024-07-18 13:07:37 -03:00
EXPECT_EQ(utf8, "säk😀"sv);
LibTextCodec: Make utf-16be and utf-16le codecs actually work There were two problems: 1. They didn't handle surrogates 2. They used signed chars, leading to eg 0x00e4 being treated as 0xffe4 Also add a basic test that catches both issues. There's some code duplication with Utf16CodePointIterator::operator*(), but let's get things working first.
2023-01-22 14:59:00 -03:00
}
LibTextCodec: Allow processing decoded UTF-16 code points Add Decoder::process_code_points() so callers can stream decoded code points without first materializing a UTF-8 string. Implement the UTF-16 variants by walking code units directly and emitting replacement code points for malformed surrogate pairs.
2026-05-17 14:05:53 -03:00
TEST_CASE(test_utf16le_process_code_points)
{
Tests/LibTextCodec: Do not refer to concrete encoders/decoders Which allows for their removal from the header file once the implementation is ported to rust.
2026-06-19 18:47:01 -03:00
auto& decoder = decoder_for("UTF-16LE"sv);
LibTextCodec: Allow processing decoded UTF-16 code points Add Decoder::process_code_points() so callers can stream decoded code points without first materializing a UTF-8 string. Implement the UTF-16 variants by walking code units directly and emitting replacement code points for malformed surrogate pairs.
2026-05-17 14:05:53 -03:00
EXPECT_EQ(process_code_points(decoder, StringView(bytes({ 0xff, 0xfe, 'A', 0x00, 0x3d, 0xd8, 0x00, 0xde }))), (Vector<u32> { 0x41, 0x1F600 }));
EXPECT_EQ(process_code_points(decoder, StringView(bytes({ 0x3d, 0xd8, 'A', 0x00, 0x00, 0xde }))), (Vector<u32> { 0xfffd, 0x41, 0xfffd }));
LibTextCodec: Preserve malformed decoder replacements Reject UTF-8 second bytes outside the Encoding Standard's per-lead-byte bounds before consuming the rest of each sequence. This keeps surrogate and out-of-range sequences from collapsing multiple malformed bytes into one replacement character. Also report an odd trailing UTF-16 byte as U+FFFD through the streaming code point path and route UTF-16 to_utf8() through the same logic. This keeps lazy and eager script decoding aligned for bytecode cache source hashes. Cover the malformed UTF-8 and UTF-16 cases in LibTextCodec, TextDecoder, and bytecode-cache source decoding tests.
2026-05-18 07:53:22 -03:00
EXPECT_EQ(process_code_points(decoder, StringView(bytes({ 'A', 0x00, 0xff }))), (Vector<u32> { 0x41, 0xfffd }));
LibWeb+LibTextCodec: Wire decoder options through TextDecoder Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders, and thread them through TextDecoder and TextDecoderStream. This lets Web-facing decoder APIs preserve BOMs when requested and use fatal error handling without post-processing decoded output. NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name used by encoding_rs and was previously an implementation detail. The new name matches what is used by the encoding standard as it is now also used in LibWeb.
2026-06-22 17:07:40 -03:00
EXPECT_EQ(MUST(decoder.to_utf8(StringView(bytes({ 'A', 0x00, 0xff })), TextCodec::IgnoreBOM::No, TextCodec::ErrorMode::Replacement)), "A\xef\xbf\xbd"sv);
LibTextCodec: Allow processing decoded UTF-16 code points Add Decoder::process_code_points() so callers can stream decoded code points without first materializing a UTF-8 string. Implement the UTF-16 variants by walking code units directly and emitting replacement code points for malformed surrogate pairs.
2026-05-17 14:05:53 -03:00
}
Reference in a new issue Copy permalink