dmaax/ladybird
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ladybird/Libraries/LibJS/SourceCode.cpp

424 lines
15 KiB
C++
Raw Normal View History

LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
/*
Meta: Update my e-mail address everywhere
2024-10-04 08:19:50 -03:00
* Copyright (c) 2022-2023, Andreas Kling <andreas@ladybird.org>
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
*
* SPDX-License-Identifier: BSD-2-Clause
*/
LibJS: Avoid repeated lazy source decoding Function.prototype.toString() can ask for the same cached function source text repeatedly after a script was materialized from the bytecode cache. In that path SourceCode still owns only the encoded source bytes, so every request decoded the requested source range again. Large ASCII bundles made that path expensive enough to stall Speedometer 2.1's Ember debug test. Keep the byte-backed SourceCode representation lazy, but let ASCII UTF-8 source ranges slice the source bytes directly. For other ASCII byte-backed encodings, first ask the decoder to prove that the source bytes map to the same UTF-16 code units at the same positions. Cache extracted function source text on shared function data after the first request so repeated toString() calls do not keep going back to SourceCode. Add bytecode cache coverage for toString() on lazy UTF-8 source bytes, keep malformed UTF-8 and UTF-16 edge cases on the decoder path, and cover PDFDocEncoding bytes that are not identity-mapped.
2026-05-19 08:59:58 -03:00
#include <AK/AllOf.h>
LibJS: Make Error stack trace generation faster with a line break cache Generating Error objects got a lot slower after the introduction of SourceCode in b0b022507b2f73be2f4cef17deb8ece91dae6822. This was noticeable with `test-js` which generates a lot of errors, so walking the source code over and over to compute (line, column) was eating a ton of time. This patch makes repeated lookups a lot faster by building a cache of line break offsets in the source code. The cache is built on first offset lookup, so we only pay for this in code that actually throws. On my machine, this takes `test-js` runtime from 6.7 sec to 4.3 sec.
2022-11-24 10:16:56 -03:00
#include <AK/BinarySearch.h>
LibJS: Avoid repeated lazy source decoding Function.prototype.toString() can ask for the same cached function source text repeatedly after a script was materialized from the bytecode cache. In that path SourceCode still owns only the encoded source bytes, so every request decoded the requested source range again. Large ASCII bundles made that path expensive enough to stall Speedometer 2.1's Ember debug test. Keep the byte-backed SourceCode representation lazy, but let ASCII UTF-8 source ranges slice the source bytes directly. For other ASCII byte-backed encodings, first ask the decoder to prove that the source bytes map to the same UTF-16 code units at the same positions. Cache extracted function source text on shared function data after the first request so repeated toString() calls do not keep going back to SourceCode. Add bytecode cache coverage for toString() on lazy UTF-8 source bytes, keep malformed UTF-8 and UTF-16 edge cases on the decoder path, and cover PDFDocEncoding bytes that are not identity-mapped.
2026-05-19 08:59:58 -03:00
#include <AK/CharacterTypes.h>
LibJS: Lazily map UTF-8 source offsets Avoid repeatedly decoding an entire byte-backed UTF-8 source when a cached function asks for its source text and the script contains a non-ASCII code point before the requested range. Build a small map from UTF-16 code unit offsets to byte offsets on demand for valid UTF-8 sources, then slice the source bytes directly for subsequent extractions. Invalid or non-UTF-8 sources keep using the existing decoder fallback. Add coverage for lazy range extraction after non-ASCII UTF-8 and for bytecode-cache materialization of function source text from such a script.
2026-05-19 10:56:55 -03:00
#include <AK/Utf8View.h>
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
#include <LibJS/SourceCode.h>
#include <LibJS/SourceRange.h>
#include <LibJS/Token.h>
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
#include <LibTextCodec/Decoder.h>
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
namespace JS {
LibJS: Avoid repeated lazy source decoding Function.prototype.toString() can ask for the same cached function source text repeatedly after a script was materialized from the bytecode cache. In that path SourceCode still owns only the encoded source bytes, so every request decoded the requested source range again. Large ASCII bundles made that path expensive enough to stall Speedometer 2.1's Ember debug test. Keep the byte-backed SourceCode representation lazy, but let ASCII UTF-8 source ranges slice the source bytes directly. For other ASCII byte-backed encodings, first ask the decoder to prove that the source bytes map to the same UTF-16 code units at the same positions. Cache extracted function source text on shared function data after the first request so repeated toString() calls do not keep going back to SourceCode. Add bytecode cache coverage for toString() on lazy UTF-8 source bytes, keep malformed UTF-8 and UTF-16 edge cases on the decoder path, and cover PDFDocEncoding bytes that are not identity-mapped.
2026-05-19 08:59:58 -03:00
static bool ascii_source_bytes_decode_to_same_code_units(StringView standardized_encoding, ReadonlyBytes bytes)
{
auto decoder = TextCodec::decoder_for_exact_name(standardized_encoding);
if (!decoder.has_value())
return false;
size_t byte_offset = 0;
bool bytes_are_identity_mapped = true;
auto result = decoder->process_code_points(StringView { bytes }, [&](auto code_point) -> ErrorOr<void> {
if (byte_offset >= bytes.size()) {
bytes_are_identity_mapped = false;
return {};
}
auto byte = bytes[byte_offset++];
if (code_point != byte)
bytes_are_identity_mapped = false;
return {};
});
result.release_value_but_fixme_should_propagate_errors();
return bytes_are_identity_mapped && byte_offset == bytes.size();
}
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 08:18:45 -03:00
NonnullRefPtr<SourceCode const> SourceCode::create(String filename, Utf16String code)
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
{
return adopt_ref(*new SourceCode(move(filename), move(code)));
}
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
NonnullRefPtr<SourceCode const> SourceCode::create(String filename, size_t length_in_code_units, String source_encoding, Core::ImmutableBytes source_bytes)
{
return adopt_ref(*new SourceCode(move(filename), length_in_code_units, move(source_encoding), move(source_bytes)));
}
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 08:18:45 -03:00
SourceCode::SourceCode(String filename, Utf16String code)
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
: m_filename(move(filename))
, m_code(move(code))
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
, m_code_view(m_code->utf16_view())
LibJS: Cache length-in-code-units in SourceCode This avoids some bit twiddling whenever accessing the length in code units in the lexer.
2025-11-08 18:57:01 -03:00
, m_length_in_code_units(m_code_view.length_in_code_units())
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
{
}
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
SourceCode::SourceCode(String filename, size_t length_in_code_units, String source_encoding, Core::ImmutableBytes source_bytes)
: m_filename(move(filename))
, m_source_encoding(move(source_encoding))
, m_source_bytes(move(source_bytes))
, m_length_in_code_units(length_in_code_units)
{
}
void SourceCode::ensure_code() const
{
if (m_code.has_value())
return;
LibJS: Remove unused UTF-8 SourceCode storage Keep SourceCode in one of two representations: materialized UTF-16 source text, or lazy immutable source bytes with an encoding. The UTF-8 String representation had no remaining callers, and made source text range extraction carry an extra code path.
2026-05-17 15:09:14 -03:00
m_code = decode_source_range(0, m_length_in_code_units);
m_source_bytes = {};
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
m_code_view = m_code->utf16_view();
}
Utf16String const& SourceCode::code() const
{
ensure_code();
return *m_code;
}
Utf16View const& SourceCode::code_view() const
{
ensure_code();
return m_code_view;
}
LibJS: Cache ASCII-to-UTF-16 source conversion for Rust compilation The Rust FFI requires UTF-16 source data, so ASCII-stored source code must be widened to UTF-16. Previously, this conversion was done into a temporary buffer on every call to compile_function, meaning the entire source file was converted for each lazily-compiled function. For large modules with many functions, this caused heavy spinning. Move the conversion into SourceCode::utf16_data() which lazily converts and caches the result once per source file. Subsequent compilations of functions from the same file reuse the cached data.
2026-02-24 18:07:27 -03:00
u16 const* SourceCode::utf16_data() const
{
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
ensure_code();
LibJS: Cache ASCII-to-UTF-16 source conversion for Rust compilation The Rust FFI requires UTF-16 source data, so ASCII-stored source code must be widened to UTF-16. Previously, this conversion was done into a temporary buffer on every call to compile_function, meaning the entire source file was converted for each lazily-compiled function. For large modules with many functions, this caused heavy spinning. Move the conversion into SourceCode::utf16_data() which lazily converts and caches the result once per source file. Subsequent compilations of functions from the same file reuse the cached data.
2026-02-24 18:07:27 -03:00
if (!m_code_view.has_ascii_storage())
return reinterpret_cast<u16 const*>(m_code_view.utf16_span().data());
if (m_utf16_data_cache.is_empty() && m_length_in_code_units > 0) {
auto ascii = m_code_view.ascii_span();
m_utf16_data_cache.ensure_capacity(m_length_in_code_units);
for (size_t i = 0; i < m_length_in_code_units; ++i)
m_utf16_data_cache.unchecked_append(static_cast<u16>(ascii[i]));
}
return m_utf16_data_cache.data();
}
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
Utf16String SourceCode::source_text_from_offsets(size_t start_offset, size_t length) const
{
if (length == 0)
return {};
LibJS: Lazily map UTF-8 source offsets Avoid repeatedly decoding an entire byte-backed UTF-8 source when a cached function asks for its source text and the script contains a non-ASCII code point before the requested range. Build a small map from UTF-16 code unit offsets to byte offsets on demand for valid UTF-8 sources, then slice the source bytes directly for subsequent extractions. Invalid or non-UTF-8 sources keep using the existing decoder fallback. Add coverage for lazy range extraction after non-ASCII UTF-8 and for bytecode-cache materialization of function source text from such a script.
2026-05-19 10:56:55 -03:00
VERIFY(start_offset <= NumericLimits<size_t>::max() - length);
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
if (m_code.has_value())
return Utf16String::from_utf16(m_code->utf16_view().substring_view(start_offset, length));
LibJS: Avoid repeated lazy source decoding Function.prototype.toString() can ask for the same cached function source text repeatedly after a script was materialized from the bytecode cache. In that path SourceCode still owns only the encoded source bytes, so every request decoded the requested source range again. Large ASCII bundles made that path expensive enough to stall Speedometer 2.1's Ember debug test. Keep the byte-backed SourceCode representation lazy, but let ASCII UTF-8 source ranges slice the source bytes directly. For other ASCII byte-backed encodings, first ask the decoder to prove that the source bytes map to the same UTF-16 code units at the same positions. Cache extracted function source text on shared function data after the first request so repeated toString() calls do not keep going back to SourceCode. Add bytecode cache coverage for toString() on lazy UTF-8 source bytes, keep malformed UTF-8 and UTF-16 edge cases on the decoder path, and cover PDFDocEncoding bytes that are not identity-mapped.
2026-05-19 08:59:58 -03:00
if (m_source_bytes.is_valid()) {
if (source_bytes_can_be_sliced_by_code_unit_offsets()) {
auto bytes = m_source_bytes.bytes();
VERIFY(m_length_in_code_units == bytes.size());
auto source_text_bytes = bytes.slice(start_offset, length);
if (all_of(source_text_bytes, AK::is_ascii))
return Utf16String::from_ascii_without_validation(source_text_bytes);
return Utf16String::from_utf8(StringView { source_text_bytes });
}
LibJS: Lazily map UTF-8 source offsets Avoid repeatedly decoding an entire byte-backed UTF-8 source when a cached function asks for its source text and the script contains a non-ASCII code point before the requested range. Build a small map from UTF-16 code unit offsets to byte offsets on demand for valid UTF-8 sources, then slice the source bytes directly for subsequent extractions. Invalid or non-UTF-8 sources keep using the existing decoder fallback. Add coverage for lazy range extraction after non-ASCII UTF-8 and for bytecode-cache materialization of function source text from such a script.
2026-05-19 10:56:55 -03:00
if (auto source_text = source_text_from_utf8_source_bytes(start_offset, length); source_text.has_value())
return source_text.release_value();
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
return decode_source_range(start_offset, length);
LibJS: Avoid repeated lazy source decoding Function.prototype.toString() can ask for the same cached function source text repeatedly after a script was materialized from the bytecode cache. In that path SourceCode still owns only the encoded source bytes, so every request decoded the requested source range again. Large ASCII bundles made that path expensive enough to stall Speedometer 2.1's Ember debug test. Keep the byte-backed SourceCode representation lazy, but let ASCII UTF-8 source ranges slice the source bytes directly. For other ASCII byte-backed encodings, first ask the decoder to prove that the source bytes map to the same UTF-16 code units at the same positions. Cache extracted function source text on shared function data after the first request so repeated toString() calls do not keep going back to SourceCode. Add bytecode cache coverage for toString() on lazy UTF-8 source bytes, keep malformed UTF-8 and UTF-16 edge cases on the decoder path, and cover PDFDocEncoding bytes that are not identity-mapped.
2026-05-19 08:59:58 -03:00
}
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
LibJS: Remove unused UTF-8 SourceCode storage Keep SourceCode in one of two representations: materialized UTF-16 source text, or lazy immutable source bytes with an encoding. The UTF-8 String representation had no remaining callers, and made source text range extraction carry an extra code path.
2026-05-17 15:09:14 -03:00
ensure_code();
return Utf16String::from_utf16(m_code->utf16_view().substring_view(start_offset, length));
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
}
LibJS: Avoid repeated lazy source decoding Function.prototype.toString() can ask for the same cached function source text repeatedly after a script was materialized from the bytecode cache. In that path SourceCode still owns only the encoded source bytes, so every request decoded the requested source range again. Large ASCII bundles made that path expensive enough to stall Speedometer 2.1's Ember debug test. Keep the byte-backed SourceCode representation lazy, but let ASCII UTF-8 source ranges slice the source bytes directly. For other ASCII byte-backed encodings, first ask the decoder to prove that the source bytes map to the same UTF-16 code units at the same positions. Cache extracted function source text on shared function data after the first request so repeated toString() calls do not keep going back to SourceCode. Add bytecode cache coverage for toString() on lazy UTF-8 source bytes, keep malformed UTF-8 and UTF-16 edge cases on the decoder path, and cover PDFDocEncoding bytes that are not identity-mapped.
2026-05-19 08:59:58 -03:00
bool SourceCode::source_bytes_can_be_sliced_by_code_unit_offsets() const
{
if (!m_source_bytes_can_be_sliced_by_code_unit_offsets.has_value()) {
auto standardized_encoding = TextCodec::get_standardized_encoding(m_source_encoding);
if (!standardized_encoding.has_value()) {
m_source_bytes_can_be_sliced_by_code_unit_offsets = false;
return *m_source_bytes_can_be_sliced_by_code_unit_offsets;
}
auto bytes = m_source_bytes.bytes();
if (m_length_in_code_units != bytes.size()) {
m_source_bytes_can_be_sliced_by_code_unit_offsets = false;
return *m_source_bytes_can_be_sliced_by_code_unit_offsets;
}
auto source_bytes_are_ascii = all_of(bytes, AK::is_ascii);
if (standardized_encoding->equals_ignoring_ascii_case("UTF-8"sv)) {
m_source_bytes_can_be_sliced_by_code_unit_offsets = source_bytes_are_ascii;
return *m_source_bytes_can_be_sliced_by_code_unit_offsets;
}
if (!source_bytes_are_ascii) {
m_source_bytes_can_be_sliced_by_code_unit_offsets = false;
return *m_source_bytes_can_be_sliced_by_code_unit_offsets;
}
m_source_bytes_can_be_sliced_by_code_unit_offsets = ascii_source_bytes_decode_to_same_code_units(*standardized_encoding, bytes);
}
return *m_source_bytes_can_be_sliced_by_code_unit_offsets;
}
LibJS: Lazily map UTF-8 source offsets Avoid repeatedly decoding an entire byte-backed UTF-8 source when a cached function asks for its source text and the script contains a non-ASCII code point before the requested range. Build a small map from UTF-16 code unit offsets to byte offsets on demand for valid UTF-8 sources, then slice the source bytes directly for subsequent extractions. Invalid or non-UTF-8 sources keep using the existing decoder fallback. Add coverage for lazy range extraction after non-ASCII UTF-8 and for bytecode-cache materialization of function source text from such a script.
2026-05-19 10:56:55 -03:00
Optional<Utf16String> SourceCode::source_text_from_utf8_source_bytes(size_t start_offset, size_t length) const
{
auto start_byte_offset = byte_offset_for_utf8_code_unit_offset(start_offset);
if (!start_byte_offset.has_value())
return {};
auto end_byte_offset = byte_offset_for_utf8_code_unit_offset(start_offset + length);
if (!end_byte_offset.has_value())
return {};
VERIFY(*start_byte_offset <= *end_byte_offset);
auto source_text_bytes = m_source_bytes.bytes().slice(*start_byte_offset, *end_byte_offset - *start_byte_offset);
if (all_of(source_text_bytes, AK::is_ascii))
return Utf16String::from_ascii_without_validation(source_text_bytes);
return Utf16String::from_utf8(StringView { source_text_bytes });
}
bool SourceCode::ensure_utf8_source_byte_spans() const
{
if (m_tried_to_build_utf8_source_byte_spans)
return m_can_use_utf8_source_byte_spans;
m_tried_to_build_utf8_source_byte_spans = true;
auto standardized_encoding = TextCodec::get_standardized_encoding(m_source_encoding);
if (!standardized_encoding.has_value() || !standardized_encoding->equals_ignoring_ascii_case("UTF-8"sv))
return false;
auto bytes = m_source_bytes.bytes();
StringView input { bytes };
if (bytes.size() >= 3 && bytes[0] == 0xEF && bytes[1] == 0xBB && bytes[2] == 0xBF) {
input = input.substring_view(3);
m_utf8_source_byte_span_initial_byte_offset = 3;
} else if (bytes.size() >= 2 && ((bytes[0] == 0xFE && bytes[1] == 0xFF) || (bytes[0] == 0xFF && bytes[1] == 0xFE))) {
return false;
}
auto utf8_view = Utf8View { input };
if (!utf8_view.validate(AllowLonelySurrogates::No))
return false;
size_t code_unit_offset = 0;
for (auto it = utf8_view.begin(); it != utf8_view.end(); ++it) {
auto code_point = *it;
size_t code_unit_length = code_point <= 0xffff ? 1 : 2;
auto byte_length = it.underlying_code_point_length_in_bytes();
if (byte_length != code_unit_length) {
m_utf8_source_byte_spans.append({
.code_unit_offset = code_unit_offset,
.code_unit_length = code_unit_length,
.byte_offset = m_utf8_source_byte_span_initial_byte_offset + utf8_view.byte_offset_of(it),
.byte_length = byte_length,
});
}
code_unit_offset += code_unit_length;
}
if (code_unit_offset != m_length_in_code_units) {
m_utf8_source_byte_spans.clear();
return false;
}
m_can_use_utf8_source_byte_spans = true;
return true;
}
Optional<size_t> SourceCode::byte_offset_for_utf8_code_unit_offset(size_t code_unit_offset) const
{
if (code_unit_offset > m_length_in_code_units)
return {};
if (!ensure_utf8_source_byte_spans())
return {};
size_t low = 0;
size_t high = m_utf8_source_byte_spans.size();
while (low < high) {
auto middle = low + (high - low) / 2;
auto const& span = m_utf8_source_byte_spans[middle];
auto span_end = span.code_unit_offset + span.code_unit_length;
if (span_end <= code_unit_offset)
low = middle + 1;
else
high = middle;
}
if (low < m_utf8_source_byte_spans.size()) {
auto const& span = m_utf8_source_byte_spans[low];
if (code_unit_offset >= span.code_unit_offset && code_unit_offset < span.code_unit_offset + span.code_unit_length) {
if (code_unit_offset == span.code_unit_offset)
return span.byte_offset;
return {};
}
}
size_t byte_delta = m_utf8_source_byte_span_initial_byte_offset;
if (low > 0) {
auto const& previous_span = m_utf8_source_byte_spans[low - 1];
auto previous_span_end_byte_offset = previous_span.byte_offset + previous_span.byte_length;
auto previous_span_end_code_unit_offset = previous_span.code_unit_offset + previous_span.code_unit_length;
VERIFY(previous_span_end_byte_offset >= previous_span_end_code_unit_offset);
byte_delta = previous_span_end_byte_offset - previous_span_end_code_unit_offset;
}
return code_unit_offset + byte_delta;
}
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
Utf16String SourceCode::decode_source_range(size_t start_offset, size_t length) const
{
if (length == 0)
return {};
VERIFY(start_offset <= NumericLimits<size_t>::max() - length);
auto end_offset = start_offset + length;
StringView input { m_source_bytes.bytes() };
auto decoder = TextCodec::decoder_for(m_source_encoding);
VERIFY(decoder.has_value());
TextCodec::Decoder* actual_decoder = &decoder.value();
auto unicode_decoder = TextCodec::bom_sniff_to_decoder(input);
if (unicode_decoder.has_value()) {
auto input_bytes = input.bytes();
auto byte_order_mark_size = input_bytes.size() >= 3 && input_bytes[0] == 0xEF && input_bytes[1] == 0xBB && input_bytes[2] == 0xBF ? 3 : 2;
actual_decoder = &unicode_decoder.value();
input = input.substring_view(byte_order_mark_size);
}
StringBuilder builder(StringBuilder::Mode::UTF16, length);
size_t current_offset = 0;
auto result = actual_decoder->process_code_points(input, [&](auto code_point) -> ErrorOr<void> {
char16_t code_units[2];
size_t code_point_length_in_code_units = 0;
(void)AK::UnicodeUtils::code_point_to_utf16(code_point, [&](auto code_unit) {
code_units[code_point_length_in_code_units++] = code_unit;
});
for (size_t i = 0; i < code_point_length_in_code_units; ++i) {
auto code_unit_offset = current_offset + i;
if (code_unit_offset >= start_offset && code_unit_offset < end_offset)
TRY(builder.try_append_code_unit(code_units[i]));
}
current_offset += code_point_length_in_code_units;
return {};
});
result.release_value_but_fixme_should_propagate_errors();
return builder.to_utf16_string();
}
LibJS: Make line-and-column resolution fast for large minified JS Instead of caching start-of-line offsets, we now cache byte offsets at regular intervals. This fixes an issue where we had terrible performance on large minified JS, since that often means one very, VERY long line (with no line endings to cache). My machine was spending ~35ms per stack frame when throwing errors on some heavy minified websites, and after this patch, we now spend <1ms per stack frame.
2023-09-12 08:03:56 -03:00
void SourceCode::fill_position_cache() const
LibJS: Make Error stack trace generation faster with a line break cache Generating Error objects got a lot slower after the introduction of SourceCode in b0b022507b2f73be2f4cef17deb8ece91dae6822. This was noticeable with `test-js` which generates a lot of errors, so walking the source code over and over to compute (line, column) was eating a ton of time. This patch makes repeated lookups a lot faster by building a cache of line break offsets in the source code. The cache is built on first offset lookup, so we only pay for this in code that actually throws. On my machine, this takes `test-js` runtime from 6.7 sec to 4.3 sec.
2022-11-24 10:16:56 -03:00
{
Everywhere: Fix typos - act III
2025-04-09 14:02:30 -03:00
constexpr size_t predicted_minimum_cached_positions = 8;
LibJS: Cache source code positions more often The source code position cache was moved from a line based approach to a "chunk"-based approach to improve performance on large, minified JavaScript files with few lines, but this has had an adverse effect on _multi-line_ source files. Reintroduce some of the old behaviour by caching lines again, with some added sanity limits to avoid caching empty/overly small lines. Source code positions in files with few lines will still be cached less often, since minified JavaScript files can be assumed to be unusually large, and since stack traces for minified JavaScript are less useful as well. On WPT tests with large JavaScript dependencies like `css/css-masking/animations/clip-interpolation.html` this reduces the amount of time spent in `SourceCode::range_from_offsets` by as much as 99.98%, for the small small price of 80KB extra memory usage.
2024-12-01 18:21:18 -03:00
constexpr size_t minimum_distance_between_cached_positions = 32;
constexpr size_t maximum_distance_between_cached_positions = 8192;
LibJS: Make Error stack trace generation faster with a line break cache Generating Error objects got a lot slower after the introduction of SourceCode in b0b022507b2f73be2f4cef17deb8ece91dae6822. This was noticeable with `test-js` which generates a lot of errors, so walking the source code over and over to compute (line, column) was eating a ton of time. This patch makes repeated lookups a lot faster by building a cache of line break offsets in the source code. The cache is built on first offset lookup, so we only pay for this in code that actually throws. On my machine, this takes `test-js` runtime from 6.7 sec to 4.3 sec.
2022-11-24 10:16:56 -03:00
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
auto const& code = this->code();
if (code.is_empty())
LibJS: Make Error stack trace generation faster with a line break cache Generating Error objects got a lot slower after the introduction of SourceCode in b0b022507b2f73be2f4cef17deb8ece91dae6822. This was noticeable with `test-js` which generates a lot of errors, so walking the source code over and over to compute (line, column) was eating a ton of time. This patch makes repeated lookups a lot faster by building a cache of line break offsets in the source code. The cache is built on first offset lookup, so we only pay for this in code that actually throws. On my machine, this takes `test-js` runtime from 6.7 sec to 4.3 sec.
2022-11-24 10:16:56 -03:00
return;
LibJS: Cache source code positions more often The source code position cache was moved from a line based approach to a "chunk"-based approach to improve performance on large, minified JavaScript files with few lines, but this has had an adverse effect on _multi-line_ source files. Reintroduce some of the old behaviour by caching lines again, with some added sanity limits to avoid caching empty/overly small lines. Source code positions in files with few lines will still be cached less often, since minified JavaScript files can be assumed to be unusually large, and since stack traces for minified JavaScript are less useful as well. On WPT tests with large JavaScript dependencies like `css/css-masking/animations/clip-interpolation.html` this reduces the amount of time spent in `SourceCode::range_from_offsets` by as much as 99.98%, for the small small price of 80KB extra memory usage.
2024-12-01 18:21:18 -03:00
u32 previous_code_point = 0;
LibJS: Store full realized SourceRange with each AST node We were spending way too much time converting unrealized source ranges into line/column pairs on real web content. This improves JS parsing speed on x.com by 1.13x
2025-12-27 16:58:36 -03:00
u32 line = 1;
u32 column = 1;
u32 offset_of_last_starting_point = 0;
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 08:18:45 -03:00
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
m_cached_positions.ensure_capacity(predicted_minimum_cached_positions + (code.length_in_code_units() / maximum_distance_between_cached_positions));
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
m_cached_positions.append({ .position = { .line = 1, .column = 1 }, .offset = 0 });
LibJS: Make line-and-column resolution fast for large minified JS Instead of caching start-of-line offsets, we now cache byte offsets at regular intervals. This fixes an issue where we had terrible performance on large minified JS, since that often means one very, VERY long line (with no line endings to cache). My machine was spending ~35ms per stack frame when throwing errors on some heavy minified websites, and after this patch, we now spend <1ms per stack frame.
2023-09-12 08:03:56 -03:00
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
auto view = code.utf16_view();
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 08:18:45 -03:00
LibJS: Make Error stack trace generation faster with a line break cache Generating Error objects got a lot slower after the introduction of SourceCode in b0b022507b2f73be2f4cef17deb8ece91dae6822. This was noticeable with `test-js` which generates a lot of errors, so walking the source code over and over to compute (line, column) was eating a ton of time. This patch makes repeated lookups a lot faster by building a cache of line break offsets in the source code. The cache is built on first offset lookup, so we only pay for this in code that actually throws. On my machine, this takes `test-js` runtime from 6.7 sec to 4.3 sec.
2022-11-24 10:16:56 -03:00
for (auto it = view.begin(); it != view.end(); ++it) {
u32 code_point = *it;
LibJS: Cache source code positions more often The source code position cache was moved from a line based approach to a "chunk"-based approach to improve performance on large, minified JavaScript files with few lines, but this has had an adverse effect on _multi-line_ source files. Reintroduce some of the old behaviour by caching lines again, with some added sanity limits to avoid caching empty/overly small lines. Source code positions in files with few lines will still be cached less often, since minified JavaScript files can be assumed to be unusually large, and since stack traces for minified JavaScript are less useful as well. On WPT tests with large JavaScript dependencies like `css/css-masking/animations/clip-interpolation.html` this reduces the amount of time spent in `SourceCode::range_from_offsets` by as much as 99.98%, for the small small price of 80KB extra memory usage.
2024-12-01 18:21:18 -03:00
bool is_line_terminator = code_point == '\r' || (code_point == '\n' && previous_code_point != '\r') || code_point == LINE_SEPARATOR || code_point == PARAGRAPH_SEPARATOR;
LibJS: Make Error stack trace generation faster with a line break cache Generating Error objects got a lot slower after the introduction of SourceCode in b0b022507b2f73be2f4cef17deb8ece91dae6822. This was noticeable with `test-js` which generates a lot of errors, so walking the source code over and over to compute (line, column) was eating a ton of time. This patch makes repeated lookups a lot faster by building a cache of line break offsets in the source code. The cache is built on first offset lookup, so we only pay for this in code that actually throws. On my machine, this takes `test-js` runtime from 6.7 sec to 4.3 sec.
2022-11-24 10:16:56 -03:00
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 08:18:45 -03:00
auto offset = view.iterator_offset(it);
LibJS: Store full realized SourceRange with each AST node We were spending way too much time converting unrealized source ranges into line/column pairs on real web content. This improves JS parsing speed on x.com by 1.13x
2025-12-27 16:58:36 -03:00
VERIFY(offset <= NumericLimits<u32>::max());
LibJS: Cache source code positions more often The source code position cache was moved from a line based approach to a "chunk"-based approach to improve performance on large, minified JavaScript files with few lines, but this has had an adverse effect on _multi-line_ source files. Reintroduce some of the old behaviour by caching lines again, with some added sanity limits to avoid caching empty/overly small lines. Source code positions in files with few lines will still be cached less often, since minified JavaScript files can be assumed to be unusually large, and since stack traces for minified JavaScript are less useful as well. On WPT tests with large JavaScript dependencies like `css/css-masking/animations/clip-interpolation.html` this reduces the amount of time spent in `SourceCode::range_from_offsets` by as much as 99.98%, for the small small price of 80KB extra memory usage.
2024-12-01 18:21:18 -03:00
bool is_nonempty_line = is_line_terminator && previous_code_point != '\n' && previous_code_point != LINE_SEPARATOR && previous_code_point != PARAGRAPH_SEPARATOR && (code_point == '\n' || previous_code_point != '\r');
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 08:18:45 -03:00
auto distance_between_cached_position = offset - offset_of_last_starting_point;
LibJS: Cache source code positions more often The source code position cache was moved from a line based approach to a "chunk"-based approach to improve performance on large, minified JavaScript files with few lines, but this has had an adverse effect on _multi-line_ source files. Reintroduce some of the old behaviour by caching lines again, with some added sanity limits to avoid caching empty/overly small lines. Source code positions in files with few lines will still be cached less often, since minified JavaScript files can be assumed to be unusually large, and since stack traces for minified JavaScript are less useful as well. On WPT tests with large JavaScript dependencies like `css/css-masking/animations/clip-interpolation.html` this reduces the amount of time spent in `SourceCode::range_from_offsets` by as much as 99.98%, for the small small price of 80KB extra memory usage.
2024-12-01 18:21:18 -03:00
if ((distance_between_cached_position >= minimum_distance_between_cached_positions && is_nonempty_line) || distance_between_cached_position >= maximum_distance_between_cached_positions) {
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
m_cached_positions.append({ .position = { .line = line, .column = column }, .offset = static_cast<u32>(offset) });
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 08:18:45 -03:00
offset_of_last_starting_point = offset;
LibJS: Make line-and-column resolution fast for large minified JS Instead of caching start-of-line offsets, we now cache byte offsets at regular intervals. This fixes an issue where we had terrible performance on large minified JS, since that often means one very, VERY long line (with no line endings to cache). My machine was spending ~35ms per stack frame when throwing errors on some heavy minified websites, and after this patch, we now spend <1ms per stack frame.
2023-09-12 08:03:56 -03:00
}
if (is_line_terminator) {
line += 1;
column = 1;
} else {
column += 1;
}
LibJS: Cache source code positions more often The source code position cache was moved from a line based approach to a "chunk"-based approach to improve performance on large, minified JavaScript files with few lines, but this has had an adverse effect on _multi-line_ source files. Reintroduce some of the old behaviour by caching lines again, with some added sanity limits to avoid caching empty/overly small lines. Source code positions in files with few lines will still be cached less often, since minified JavaScript files can be assumed to be unusually large, and since stack traces for minified JavaScript are less useful as well. On WPT tests with large JavaScript dependencies like `css/css-masking/animations/clip-interpolation.html` this reduces the amount of time spent in `SourceCode::range_from_offsets` by as much as 99.98%, for the small small price of 80KB extra memory usage.
2024-12-01 18:21:18 -03:00
previous_code_point = code_point;
LibJS: Make Error stack trace generation faster with a line break cache Generating Error objects got a lot slower after the introduction of SourceCode in b0b022507b2f73be2f4cef17deb8ece91dae6822. This was noticeable with `test-js` which generates a lot of errors, so walking the source code over and over to compute (line, column) was eating a ton of time. This patch makes repeated lookups a lot faster by building a cache of line break offsets in the source code. The cache is built on first offset lookup, so we only pay for this in code that actually throws. On my machine, this takes `test-js` runtime from 6.7 sec to 4.3 sec.
2022-11-24 10:16:56 -03:00
}
}
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
SourceRange SourceCode::range_from_offsets(u32 start_offset, [[maybe_unused]] u32 end_offset) const
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
{
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
// If the underlying code is an empty string, the range is 1,1 no matter what.
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
auto const& code = this->code();
if (code.is_empty())
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
return { *this, { .line = 1, .column = 1 } };
LibJS: Make Error stack trace generation faster with a line break cache Generating Error objects got a lot slower after the introduction of SourceCode in b0b022507b2f73be2f4cef17deb8ece91dae6822. This was noticeable with `test-js` which generates a lot of errors, so walking the source code over and over to compute (line, column) was eating a ton of time. This patch makes repeated lookups a lot faster by building a cache of line break offsets in the source code. The cache is built on first offset lookup, so we only pay for this in code that actually throws. On my machine, this takes `test-js` runtime from 6.7 sec to 4.3 sec.
2022-11-24 10:16:56 -03:00
LibJS: Make line-and-column resolution fast for large minified JS Instead of caching start-of-line offsets, we now cache byte offsets at regular intervals. This fixes an issue where we had terrible performance on large minified JS, since that often means one very, VERY long line (with no line endings to cache). My machine was spending ~35ms per stack frame when throwing errors on some heavy minified websites, and after this patch, we now spend <1ms per stack frame.
2023-09-12 08:03:56 -03:00
if (m_cached_positions.is_empty())
fill_position_cache();
LibJS: Make Error stack trace generation faster with a line break cache Generating Error objects got a lot slower after the introduction of SourceCode in b0b022507b2f73be2f4cef17deb8ece91dae6822. This was noticeable with `test-js` which generates a lot of errors, so walking the source code over and over to compute (line, column) was eating a ton of time. This patch makes repeated lookups a lot faster by building a cache of line break offsets in the source code. The cache is built on first offset lookup, so we only pay for this in code that actually throws. On my machine, this takes `test-js` runtime from 6.7 sec to 4.3 sec.
2022-11-24 10:16:56 -03:00
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
CachedPosition current { .position = { .line = 1, .column = 1 }, .offset = 0 };
LibJS: Make Error stack trace generation faster with a line break cache Generating Error objects got a lot slower after the introduction of SourceCode in b0b022507b2f73be2f4cef17deb8ece91dae6822. This was noticeable with `test-js` which generates a lot of errors, so walking the source code over and over to compute (line, column) was eating a ton of time. This patch makes repeated lookups a lot faster by building a cache of line break offsets in the source code. The cache is built on first offset lookup, so we only pay for this in code that actually throws. On my machine, this takes `test-js` runtime from 6.7 sec to 4.3 sec.
2022-11-24 10:16:56 -03:00
LibJS: Make line-and-column resolution fast for large minified JS Instead of caching start-of-line offsets, we now cache byte offsets at regular intervals. This fixes an issue where we had terrible performance on large minified JS, since that often means one very, VERY long line (with no line endings to cache). My machine was spending ~35ms per stack frame when throwing errors on some heavy minified websites, and after this patch, we now spend <1ms per stack frame.
2023-09-12 08:03:56 -03:00
if (!m_cached_positions.is_empty()) {
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
CachedPosition const dummy;
LibJS: Make line-and-column resolution fast for large minified JS Instead of caching start-of-line offsets, we now cache byte offsets at regular intervals. This fixes an issue where we had terrible performance on large minified JS, since that often means one very, VERY long line (with no line endings to cache). My machine was spending ~35ms per stack frame when throwing errors on some heavy minified websites, and after this patch, we now spend <1ms per stack frame.
2023-09-12 08:03:56 -03:00
size_t nearest_index = 0;
binary_search(m_cached_positions, dummy, &nearest_index,
[&](auto&, auto& starting_point) {
return start_offset - starting_point.offset;
});
current = m_cached_positions[nearest_index];
LibJS: Make Error stack trace generation faster with a line break cache Generating Error objects got a lot slower after the introduction of SourceCode in b0b022507b2f73be2f4cef17deb8ece91dae6822. This was noticeable with `test-js` which generates a lot of errors, so walking the source code over and over to compute (line, column) was eating a ton of time. This patch makes repeated lookups a lot faster by building a cache of line break offsets in the source code. The cache is built on first offset lookup, so we only pay for this in code that actually throws. On my machine, this takes `test-js` runtime from 6.7 sec to 4.3 sec.
2022-11-24 10:16:56 -03:00
}
LibJS: Unbreak test262-parser-tests after source range caching changes Turns out we have to be a little more lenient when dealing with the slightly hostile inputs in tests262-parser-tests. This fix papers over some problematic situations by returning a fallback range if we can't figure out exactly where an error occurred. I've added a FIXME about returning nicer values.
2022-11-24 14:01:52 -03:00
Optional<Position> start;
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
LibJS: Cache source code positions more often The source code position cache was moved from a line based approach to a "chunk"-based approach to improve performance on large, minified JavaScript files with few lines, but this has had an adverse effect on _multi-line_ source files. Reintroduce some of the old behaviour by caching lines again, with some added sanity limits to avoid caching empty/overly small lines. Source code positions in files with few lines will still be cached less often, since minified JavaScript files can be assumed to be unusually large, and since stack traces for minified JavaScript are less useful as well. On WPT tests with large JavaScript dependencies like `css/css-masking/animations/clip-interpolation.html` this reduces the amount of time spent in `SourceCode::range_from_offsets` by as much as 99.98%, for the small small price of 80KB extra memory usage.
2024-12-01 18:21:18 -03:00
u32 previous_code_point = 0;
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
LibJS+LibWeb: Keep cached script source text lazy Avoid decoding warm-cache script responses into full UTF-16 SourceCode buffers when a bytecode cache sidecar is available. SourceCode now keeps the original immutable source bytes and source encoding, then decodes only when full source text or a Function.toString() range is requested. Compute the bytecode cache source hash while streaming decoded code points from the response bytes, so cache validation does not force an intermediate UTF-8 string. Function and class source text metadata now stores SourceCode ranges instead of views into a materialized buffer.
2026-05-17 14:54:58 -03:00
auto view = code.utf16_view();
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 08:18:45 -03:00
for (auto it = view.iterator_at_code_unit_offset(current.offset); it != view.end(); ++it) {
LibJS: Unbreak test262-parser-tests after source range caching changes Turns out we have to be a little more lenient when dealing with the slightly hostile inputs in tests262-parser-tests. This fix papers over some problematic situations by returning a fallback range if we can't figure out exactly where an error occurred. I've added a FIXME about returning nicer values.
2022-11-24 14:01:52 -03:00
// If we're on or after the start offset, this is the start position.
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 08:18:45 -03:00
if (!start.has_value() && view.iterator_offset(it) >= start_offset) {
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
start = Position {
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
.line = current.position.line,
.column = current.position.column,
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
};
}
u32 code_point = *it;
LibJS: Cache source code positions more often The source code position cache was moved from a line based approach to a "chunk"-based approach to improve performance on large, minified JavaScript files with few lines, but this has had an adverse effect on _multi-line_ source files. Reintroduce some of the old behaviour by caching lines again, with some added sanity limits to avoid caching empty/overly small lines. Source code positions in files with few lines will still be cached less often, since minified JavaScript files can be assumed to be unusually large, and since stack traces for minified JavaScript are less useful as well. On WPT tests with large JavaScript dependencies like `css/css-masking/animations/clip-interpolation.html` this reduces the amount of time spent in `SourceCode::range_from_offsets` by as much as 99.98%, for the small small price of 80KB extra memory usage.
2024-12-01 18:21:18 -03:00
bool const is_line_terminator = code_point == '\r' || (code_point == '\n' && previous_code_point != '\r') || code_point == LINE_SEPARATOR || code_point == PARAGRAPH_SEPARATOR;
previous_code_point = code_point;
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
if (is_line_terminator) {
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
current.position.line += 1;
current.position.column = 1;
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
continue;
}
LibJS: Port the JS lexer and parser to UTF-16 This ports the lexer to UTF-16 and deals with the immediate fallout up to the AST. The AST will be dealt with in upcoming commits. The lexer will still accept UTF-8 strings as input, and will transcode them to UTF-16 for lexing. This doesn't actually incur a new allocation, as we were already converting the input StringView to a ByteString for each lexer. One immediate logical benefit here is that we do not need to know off- hand how many UTF-8 bytes some special code points occupy. They all happen to be a single UTF-16 code unit. So instead of advancing the lexer by 3 positions in some cases, we can just always advance by 1.
2025-08-06 08:18:45 -03:00
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
current.position.column += 1;
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
}
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
// If we didn't find a start position, just return 1,1.
LibJS: Unbreak test262-parser-tests after source range caching changes Turns out we have to be a little more lenient when dealing with the slightly hostile inputs in tests262-parser-tests. This fix papers over some problematic situations by returning a fallback range if we can't figure out exactly where an error occurred. I've added a FIXME about returning nicer values.
2022-11-24 14:01:52 -03:00
// FIXME: This is a hack. Find a way to return the nicest possible values here.
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
if (!start.has_value())
return SourceRange { *this, { .line = 1, .column = 1 } };
LibJS: Unbreak test262-parser-tests after source range caching changes Turns out we have to be a little more lenient when dealing with the slightly hostile inputs in tests262-parser-tests. This fix papers over some problematic situations by returning a fallback range if we can't figure out exactly where an error occurred. I've added a FIXME about returning nicer values.
2022-11-24 14:01:52 -03:00
LibJS: Drop source ranges from bytecode source maps Store source map locations as bytecode offset, line, and column. Runtime consumers only emit the start line and column, so source end positions and source text offsets do not need to be carried through Executable source maps, bytecode cache serialization, or the Rust FFI. Keep SourceCode's internal position cache able to track source text offsets so callers can still translate source offsets to line and column pairs when needed. Hash dump-bytecode IDs from the name, first source position, and bytecode size instead of source slices that need end offsets. Bump the bytecode cache format version for the slimmer serialized source map entry shape.
2026-05-14 03:29:39 -03:00
return SourceRange { *this, *start };
LibJS: Reduce AST memory usage by shrink-wrapping source range info Before this change, each AST node had a 64-byte SourceRange member. This SourceRange had the following layout: filename: StringView (16 bytes) start: Position (24 bytes) end: Position (24 bytes) The Position structs have { line, column, offset }, all members size_t. To reduce memory consumption, AST nodes now only store the following: source_code: NonnullRefPtr<SourceCode> (8 bytes) start_offset: u32 (4 bytes) end_offset: u32 (4 bytes) SourceCode is a new ref-counted data structure that keeps the filename and original parsed source code in a single location, and all AST nodes have a pointer to it. The start_offset and end_offset can be turned into (line, column) when necessary by calling SourceCode::range_from_offsets(). This will walk the source code string and compute line/column numbers on the fly, so it's not necessarily fast, but it should be rare since this information is primarily used for diagnostics and exception stack traces. With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a ~23% reduction in memory usage when loading twitter.com/awesomekling (330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
}
}
Reference in a new issue Copy permalink