LibJS: Reduce AST memory usage by shrink-wrapping source range info
Before this change, each AST node had a 64-byte SourceRange member.
This SourceRange had the following layout:
filename: StringView (16 bytes)
start: Position (24 bytes)
end: Position (24 bytes)
The Position structs have { line, column, offset }, all members size_t.
To reduce memory consumption, AST nodes now only store the following:
source_code: NonnullRefPtr<SourceCode> (8 bytes)
start_offset: u32 (4 bytes)
end_offset: u32 (4 bytes)
SourceCode is a new ref-counted data structure that keeps the filename
and original parsed source code in a single location, and all AST nodes
have a pointer to it.
The start_offset and end_offset can be turned into (line, column) when
necessary by calling SourceCode::range_from_offsets(). This will walk
the source code string and compute line/column numbers on the fly, so
it's not necessarily fast, but it should be rare since this information
is primarily used for diagnostics and exception stack traces.
With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a
~23% reduction in memory usage when loading twitter.com/awesomekling
(330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
|
|
|
/*
|
2024-10-04 08:19:50 -03:00
|
|
|
* Copyright (c) 2022-2023, Andreas Kling <andreas@ladybird.org>
|
LibJS: Reduce AST memory usage by shrink-wrapping source range info
Before this change, each AST node had a 64-byte SourceRange member.
This SourceRange had the following layout:
filename: StringView (16 bytes)
start: Position (24 bytes)
end: Position (24 bytes)
The Position structs have { line, column, offset }, all members size_t.
To reduce memory consumption, AST nodes now only store the following:
source_code: NonnullRefPtr<SourceCode> (8 bytes)
start_offset: u32 (4 bytes)
end_offset: u32 (4 bytes)
SourceCode is a new ref-counted data structure that keeps the filename
and original parsed source code in a single location, and all AST nodes
have a pointer to it.
The start_offset and end_offset can be turned into (line, column) when
necessary by calling SourceCode::range_from_offsets(). This will walk
the source code string and compute line/column numbers on the fly, so
it's not necessarily fast, but it should be rare since this information
is primarily used for diagnostics and exception stack traces.
With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a
~23% reduction in memory usage when loading twitter.com/awesomekling
(330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
|
|
|
*
|
|
|
|
|
* SPDX-License-Identifier: BSD-2-Clause
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#pragma once
|
|
|
|
|
|
2026-05-17 14:54:58 -03:00
|
|
|
#include <AK/Optional.h>
|
2023-01-26 10:33:18 -03:00
|
|
|
#include <AK/String.h>
|
2025-08-06 08:18:45 -03:00
|
|
|
#include <AK/Utf16String.h>
|
2022-11-24 10:16:56 -03:00
|
|
|
#include <AK/Vector.h>
|
2026-05-17 14:54:58 -03:00
|
|
|
#include <LibCore/ImmutableBytes.h>
|
2025-07-19 17:49:30 -03:00
|
|
|
#include <LibJS/Export.h>
|
LibJS: Reduce AST memory usage by shrink-wrapping source range info
Before this change, each AST node had a 64-byte SourceRange member.
This SourceRange had the following layout:
filename: StringView (16 bytes)
start: Position (24 bytes)
end: Position (24 bytes)
The Position structs have { line, column, offset }, all members size_t.
To reduce memory consumption, AST nodes now only store the following:
source_code: NonnullRefPtr<SourceCode> (8 bytes)
start_offset: u32 (4 bytes)
end_offset: u32 (4 bytes)
SourceCode is a new ref-counted data structure that keeps the filename
and original parsed source code in a single location, and all AST nodes
have a pointer to it.
The start_offset and end_offset can be turned into (line, column) when
necessary by calling SourceCode::range_from_offsets(). This will walk
the source code string and compute line/column numbers on the fly, so
it's not necessarily fast, but it should be rare since this information
is primarily used for diagnostics and exception stack traces.
With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a
~23% reduction in memory usage when loading twitter.com/awesomekling
(330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
|
|
|
#include <LibJS/Forward.h>
|
2023-09-12 08:03:56 -03:00
|
|
|
#include <LibJS/Position.h>
|
LibJS: Reduce AST memory usage by shrink-wrapping source range info
Before this change, each AST node had a 64-byte SourceRange member.
This SourceRange had the following layout:
filename: StringView (16 bytes)
start: Position (24 bytes)
end: Position (24 bytes)
The Position structs have { line, column, offset }, all members size_t.
To reduce memory consumption, AST nodes now only store the following:
source_code: NonnullRefPtr<SourceCode> (8 bytes)
start_offset: u32 (4 bytes)
end_offset: u32 (4 bytes)
SourceCode is a new ref-counted data structure that keeps the filename
and original parsed source code in a single location, and all AST nodes
have a pointer to it.
The start_offset and end_offset can be turned into (line, column) when
necessary by calling SourceCode::range_from_offsets(). This will walk
the source code string and compute line/column numbers on the fly, so
it's not necessarily fast, but it should be rare since this information
is primarily used for diagnostics and exception stack traces.
With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a
~23% reduction in memory usage when loading twitter.com/awesomekling
(330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
|
|
|
|
|
|
|
|
namespace JS {
|
|
|
|
|
|
2025-07-19 17:49:30 -03:00
|
|
|
class JS_API SourceCode : public RefCounted<SourceCode> {
|
LibJS: Reduce AST memory usage by shrink-wrapping source range info
Before this change, each AST node had a 64-byte SourceRange member.
This SourceRange had the following layout:
filename: StringView (16 bytes)
start: Position (24 bytes)
end: Position (24 bytes)
The Position structs have { line, column, offset }, all members size_t.
To reduce memory consumption, AST nodes now only store the following:
source_code: NonnullRefPtr<SourceCode> (8 bytes)
start_offset: u32 (4 bytes)
end_offset: u32 (4 bytes)
SourceCode is a new ref-counted data structure that keeps the filename
and original parsed source code in a single location, and all AST nodes
have a pointer to it.
The start_offset and end_offset can be turned into (line, column) when
necessary by calling SourceCode::range_from_offsets(). This will walk
the source code string and compute line/column numbers on the fly, so
it's not necessarily fast, but it should be rare since this information
is primarily used for diagnostics and exception stack traces.
With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a
~23% reduction in memory usage when loading twitter.com/awesomekling
(330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
|
|
|
public:
|
2025-08-06 08:18:45 -03:00
|
|
|
static NonnullRefPtr<SourceCode const> create(String filename, Utf16String code);
|
2026-05-17 14:54:58 -03:00
|
|
|
static NonnullRefPtr<SourceCode const> create(String filename, size_t length_in_code_units, String source_encoding, Core::ImmutableBytes source_bytes);
|
LibJS: Reduce AST memory usage by shrink-wrapping source range info
Before this change, each AST node had a 64-byte SourceRange member.
This SourceRange had the following layout:
filename: StringView (16 bytes)
start: Position (24 bytes)
end: Position (24 bytes)
The Position structs have { line, column, offset }, all members size_t.
To reduce memory consumption, AST nodes now only store the following:
source_code: NonnullRefPtr<SourceCode> (8 bytes)
start_offset: u32 (4 bytes)
end_offset: u32 (4 bytes)
SourceCode is a new ref-counted data structure that keeps the filename
and original parsed source code in a single location, and all AST nodes
have a pointer to it.
The start_offset and end_offset can be turned into (line, column) when
necessary by calling SourceCode::range_from_offsets(). This will walk
the source code string and compute line/column numbers on the fly, so
it's not necessarily fast, but it should be rare since this information
is primarily used for diagnostics and exception stack traces.
With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a
~23% reduction in memory usage when loading twitter.com/awesomekling
(330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
|
|
|
|
2025-08-06 08:18:45 -03:00
|
|
|
String const& filename() const { return m_filename; }
|
2026-05-17 14:54:58 -03:00
|
|
|
Utf16String const& code() const;
|
|
|
|
|
Utf16View const& code_view() const;
|
2025-11-08 18:57:01 -03:00
|
|
|
size_t length_in_code_units() const { return m_length_in_code_units; }
|
LibJS: Reduce AST memory usage by shrink-wrapping source range info
Before this change, each AST node had a 64-byte SourceRange member.
This SourceRange had the following layout:
filename: StringView (16 bytes)
start: Position (24 bytes)
end: Position (24 bytes)
The Position structs have { line, column, offset }, all members size_t.
To reduce memory consumption, AST nodes now only store the following:
source_code: NonnullRefPtr<SourceCode> (8 bytes)
start_offset: u32 (4 bytes)
end_offset: u32 (4 bytes)
SourceCode is a new ref-counted data structure that keeps the filename
and original parsed source code in a single location, and all AST nodes
have a pointer to it.
The start_offset and end_offset can be turned into (line, column) when
necessary by calling SourceCode::range_from_offsets(). This will walk
the source code string and compute line/column numbers on the fly, so
it's not necessarily fast, but it should be rare since this information
is primarily used for diagnostics and exception stack traces.
With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a
~23% reduction in memory usage when loading twitter.com/awesomekling
(330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
|
|
|
|
2026-02-24 18:07:27 -03:00
|
|
|
u16 const* utf16_data() const;
|
2026-05-17 14:54:58 -03:00
|
|
|
Utf16String source_text_from_offsets(size_t start_offset, size_t length) const;
|
2026-02-24 18:07:27 -03:00
|
|
|
|
LibJS: Reduce AST memory usage by shrink-wrapping source range info
Before this change, each AST node had a 64-byte SourceRange member.
This SourceRange had the following layout:
filename: StringView (16 bytes)
start: Position (24 bytes)
end: Position (24 bytes)
The Position structs have { line, column, offset }, all members size_t.
To reduce memory consumption, AST nodes now only store the following:
source_code: NonnullRefPtr<SourceCode> (8 bytes)
start_offset: u32 (4 bytes)
end_offset: u32 (4 bytes)
SourceCode is a new ref-counted data structure that keeps the filename
and original parsed source code in a single location, and all AST nodes
have a pointer to it.
The start_offset and end_offset can be turned into (line, column) when
necessary by calling SourceCode::range_from_offsets(). This will walk
the source code string and compute line/column numbers on the fly, so
it's not necessarily fast, but it should be rare since this information
is primarily used for diagnostics and exception stack traces.
With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a
~23% reduction in memory usage when loading twitter.com/awesomekling
(330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
|
|
|
SourceRange range_from_offsets(u32 start_offset, u32 end_offset) const;
|
|
|
|
|
|
|
|
|
|
private:
|
2025-08-06 08:18:45 -03:00
|
|
|
SourceCode(String filename, Utf16String code);
|
2026-05-17 14:54:58 -03:00
|
|
|
SourceCode(String filename, size_t length_in_code_units, String source_encoding, Core::ImmutableBytes source_bytes);
|
|
|
|
|
void ensure_code() const;
|
|
|
|
|
Utf16String decode_source_range(size_t start_offset, size_t length) const;
|
2026-05-19 08:59:58 -03:00
|
|
|
bool source_bytes_can_be_sliced_by_code_unit_offsets() const;
|
2026-05-19 10:56:55 -03:00
|
|
|
Optional<Utf16String> source_text_from_utf8_source_bytes(size_t start_offset, size_t length) const;
|
|
|
|
|
bool ensure_utf8_source_byte_spans() const;
|
|
|
|
|
Optional<size_t> byte_offset_for_utf8_code_unit_offset(size_t code_unit_offset) const;
|
|
|
|
|
|
|
|
|
|
struct Utf8SourceByteSpan {
|
|
|
|
|
size_t code_unit_offset { 0 };
|
|
|
|
|
size_t code_unit_length { 0 };
|
|
|
|
|
size_t byte_offset { 0 };
|
|
|
|
|
size_t byte_length { 0 };
|
|
|
|
|
};
|
LibJS: Reduce AST memory usage by shrink-wrapping source range info
Before this change, each AST node had a 64-byte SourceRange member.
This SourceRange had the following layout:
filename: StringView (16 bytes)
start: Position (24 bytes)
end: Position (24 bytes)
The Position structs have { line, column, offset }, all members size_t.
To reduce memory consumption, AST nodes now only store the following:
source_code: NonnullRefPtr<SourceCode> (8 bytes)
start_offset: u32 (4 bytes)
end_offset: u32 (4 bytes)
SourceCode is a new ref-counted data structure that keeps the filename
and original parsed source code in a single location, and all AST nodes
have a pointer to it.
The start_offset and end_offset can be turned into (line, column) when
necessary by calling SourceCode::range_from_offsets(). This will walk
the source code string and compute line/column numbers on the fly, so
it's not necessarily fast, but it should be rare since this information
is primarily used for diagnostics and exception stack traces.
With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a
~23% reduction in memory usage when loading twitter.com/awesomekling
(330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
|
|
|
|
2023-01-26 10:33:18 -03:00
|
|
|
String m_filename;
|
2026-05-17 14:54:58 -03:00
|
|
|
Optional<Utf16String> mutable m_code;
|
|
|
|
|
String m_source_encoding;
|
|
|
|
|
Core::ImmutableBytes mutable m_source_bytes;
|
|
|
|
|
Utf16View mutable m_code_view;
|
2025-11-08 18:57:01 -03:00
|
|
|
size_t m_length_in_code_units { 0 };
|
2022-11-24 10:16:56 -03:00
|
|
|
|
2023-09-12 08:03:56 -03:00
|
|
|
// For fast mapping of offsets to line/column numbers, we build a list of
|
|
|
|
|
// starting points (with byte offsets into the source string) and which
|
|
|
|
|
// line:column they map to. This can then be binary-searched.
|
|
|
|
|
void fill_position_cache() const;
|
2026-05-14 03:29:39 -03:00
|
|
|
struct CachedPosition {
|
|
|
|
|
Position position;
|
|
|
|
|
u32 offset { 0 };
|
|
|
|
|
};
|
|
|
|
|
Vector<CachedPosition> mutable m_cached_positions;
|
2026-02-24 18:07:27 -03:00
|
|
|
|
|
|
|
|
// Cached UTF-16 widening of ASCII source data, lazily populated by
|
|
|
|
|
// utf16_data() for use by the Rust compilation pipeline.
|
|
|
|
|
Vector<u16> mutable m_utf16_data_cache;
|
2026-05-19 08:59:58 -03:00
|
|
|
Optional<bool> mutable m_source_bytes_can_be_sliced_by_code_unit_offsets;
|
2026-05-19 10:56:55 -03:00
|
|
|
Vector<Utf8SourceByteSpan> mutable m_utf8_source_byte_spans;
|
|
|
|
|
size_t mutable m_utf8_source_byte_span_initial_byte_offset { 0 };
|
|
|
|
|
bool mutable m_tried_to_build_utf8_source_byte_spans { false };
|
|
|
|
|
bool mutable m_can_use_utf8_source_byte_spans { false };
|
LibJS: Reduce AST memory usage by shrink-wrapping source range info
Before this change, each AST node had a 64-byte SourceRange member.
This SourceRange had the following layout:
filename: StringView (16 bytes)
start: Position (24 bytes)
end: Position (24 bytes)
The Position structs have { line, column, offset }, all members size_t.
To reduce memory consumption, AST nodes now only store the following:
source_code: NonnullRefPtr<SourceCode> (8 bytes)
start_offset: u32 (4 bytes)
end_offset: u32 (4 bytes)
SourceCode is a new ref-counted data structure that keeps the filename
and original parsed source code in a single location, and all AST nodes
have a pointer to it.
The start_offset and end_offset can be turned into (line, column) when
necessary by calling SourceCode::range_from_offsets(). This will walk
the source code string and compute line/column numbers on the fly, so
it's not necessarily fast, but it should be rare since this information
is primarily used for diagnostics and exception stack traces.
With this, ASTNode shrinks from 80 bytes to 32 bytes. This gives us a
~23% reduction in memory usage when loading twitter.com/awesomekling
(330 MiB before, 253 MiB after!) :^)
2022-11-21 13:37:38 -03:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
}
|