Allow having separate GC heaps and implement coordinated marking between
them; this is useful for keeping wasm and js GC heaps separated with a
clear boundary.
Enable -Wexit-time-destructors for all in-tree library targets and
update process-lifetime library statics so they no longer register
exit-time destructors. Long-lived caches, lookup tables, singleton
registries, and generated constants now use NeverDestroyed or leaked
references where the data is intended to live until process exit.
Update LibWeb, LibLine, and the binding generators so regenerated
sources follow the same rule instead of reintroducing destructed
statics.
Mirrors `WeakHashSet` for map shapes. Cell-typed key and/or value
slots are stored as `Weak<T>` so entries vanish when their referent
is collected; non-cell slots are stored directly.
Make Heap::allocate() reject GC-cell allocation while a collection is
active. This catches finalizers or other collection-time hooks that try
to allocate before they can re-enter collection indirectly.
Allow root containers to gather GC::Ref<T> and GC::Ptr<T> by treating
their stored pointer as a Cell pointer.
This avoids requiring T to be complete when rooting containers such as
RootVector<GC::Ref<T>>.
Similar to GC::Root<T>, make GC::RootVector<T> constructible without
explicitly passing a Heap.
This is implemented by having RootVectorBase use GC::Heap::the() for
heap-free construction.
Collection was purely allocation-driven: a GC only ran once
allocation since the last collection passed a threshold of 7/4 of
the live set (floored at 8 MiB). A page that allocated garbage but
never reached that threshold held onto it indefinitely once it went
idle, so we never handed memory back to the system promptly.
Run a 4-second repeating timer while the mutator is allocating; on
each tick IdleCollectionPolicy picks one of three actions:
- Park the timer when nothing has been allocated since the last
collection. The next allocation re-arms it, so a fully idle heap
costs nothing.
- Collect when this tick's allocation rate fell below 1/4 of the
peak rate seen this episode (the mutator left an active phase),
provided at least threshold/16 of garbage has piled up, so we
don't mark the whole live heap to reclaim a trivial amount.
- Otherwise let a watchdog collect after 15 ticks (60 seconds), so
garbage cannot sit indefinitely on a heap that allocates too
steadily to show a rate drop, or too slowly to clear the gate.
The GC heap is never completely silent in practice, since event-loop
housekeeping keeps queuing small objects like HTML tasks; that is
why the trigger watches for a relative rate drop rather than for
zero allocation.
The per-tick decision lives in IdleCollectionPolicy, separate from
the timer plumbing, with a unit test covering the rate-drop trigger,
the minimum-garbage gate, the watchdog, and parking when idle.
Heap destruction uses CollectEverything to synchronously free every live
cell. Empty blocks from that pass do not need deferred madvise work.
Registering them with the global decommit worker can start a detached
thread during process teardown. LeakSanitizer can then hang while doing
its final thread scan with log_threads enabled.
Let full sweep deallocation opt out of deferred decommit registration.
Normal incremental sweeping stays on the background worker path. The
sanitizer LibGC container and visitor tests now exit with verbose LSan
thread logging enabled.
Kick the BlockAllocator decommit worker when incremental sweep finishes,
matching the synchronous sweep path. Empty HeapBlock slots freed during
incremental sweep otherwise remain in the freshly freed pool and avoid
madvise() unless a later synchronous sweep wakes the worker.
This requires the Variant to contain at least one visitable type.
For example, requiring them all to be visitable wouldn't allow types
such as `Variant<Empty, GC::Ref<Document>>`.
A finalized GC::Function should no longer be callable. Clear the
underlying AK::Function during finalization so stale references fail as
empty callbacks instead of calling through finalized capture storage.
The phase breakdown was authored when sweep_dead_cells was the
single STW sweep phase, with sweep_callbacks and weak-container
work nested under it. After incremental sweep landed, that nesting
no longer matches reality: sweep_callbacks runs at STW every
collection, the weak-container prune is its own STW step, and
sweep_dead_cells only runs for CollectEverything.
Promote prune_weak_containers and sweep_callbacks to top-level
phases so they show up correctly in the report, and gate the
sweep_dead_cells subsection on a non-zero time so normal
collections no longer print a wall of zero rows.
The prune-weak-containers loop was previously untimed, leaving an
unaccounted gap in the per-GC totals. Wire it through the existing
ScopedPhaseTimer mechanism. The PhaseTimings field for it is
renamed from sweep_weak_containers_us to prune_weak_containers_us
to disambiguate from sweep_weak_blocks_us, which times a different
piece of work.
Record incremental sweep batch timing while LIBGC_LOG_LEVEL enables GC
reporting. Print the batch summary once the incremental sweep fully
finishes, so normal collection reports include the delayed sweep work
instead of leaving the sweep section empty for incremental collections.
Now that Heap maintains a persistent set of live heap blocks, use it
in the marking and conservative root scanning phases instead of
rebuilding a local copy on every collection cycle.
Maintain a HashTable<HeapBlock*> of live heap blocks in the Heap,
updated on block creation and destruction.
Weak containers (WeakMap, WeakSet, WeakRef, FinalizationRegistry)
now check block liveness before accessing cell memory in their
remove_dead_cells() methods. This prevents use-after-free when
blocks have been freed during incremental sweeping.
Move weak container cleanup (remove_dead_cells) out of both
sweep_dead_cells() and start_incremental_sweep() to the place
where it is actually safe to inspect cell state: collect_garbage().
Previously, remove_dead_cells could access cells that had already
been swept and poisoned by ASAN, causing use-after-poison crashes
when a new GC triggered while an incremental sweep was in progress.
Instead of sweeping all heap blocks in one go after marking, sweep
incrementally, one block at a time, interleaved with program execution.
This significantly reduces worst-case GC pause times by spreading
sweep work across multiple smaller time slices.
Sweep is driven by two complementary mechanisms:
1. Timer-based sweeping: A 16ms repeating timer drives background
sweep work, processing blocks for up to 5ms per timer fire.
2. Allocation-directed sweeping: Each allocator sweeps its own
pending blocks before creating new ones, ensuring forward
progress even without timer events.
Each allocator maintains its own list of blocks pending sweep,
and allocators with pending work are tracked in a separate list
for efficient timer-driven sweeping.
Key implementation details:
- Newly allocated cells during sweep are marked immediately to
prevent premature collection.
- Mark bits are cleared incrementally as each block is swept,
rather than in a separate pass over the entire heap.
- Finalization and weak reference processing remain stop-the-world
since they must complete atomically before any sweeping occurs.
Use GC::Timer instead of Core::Timer for the cursor blink timer so
that capturing [this] in the callback properly protects the Document
from garbage collection.
Move HeapTimer out of LibWeb and into LibGC as GC::Timer, inheriting
from GC::Cell instead of JS::Cell. Add a finalize() override that
stops the timer, ensuring it is cleaned up during GC finalization.
Reading LIBGC_LOG_LEVEL once at startup picks the verbosity for the
collection reports:
0/default: silent.
1: per-GC report with totals and detailed per-phase timings.
2+: everything in level 1, plus the full block allocator dump.
The existing collect_garbage(..., print_report=true) entry point still
works and behaves like a per-call floor of level 1, so the DevTools
"collect-garbage" inspector request keeps emitting a report regardless
of the env var. The allocator dump is now off by default at level 1
since most reports do not need it.
Right-aligned the percentage column in the breakdown so the numbers
line up cleanly.
The per-collection report now includes microsecond timings (with
percentage of total) for each phase and major subphase:
gather_roots
must-survive scan / embedder roots / explicit roots
conservative roots
register scan / stack scan / conservative-vector / cell lookup
mark_live_cells
initial visit / BFS marking / clear uprooted
finalize_unmarked_cells
sweep_weak_blocks
sweep_dead_cells
block iteration / weak containers / sweep callbacks
block reclassify / update threshold
Timings are recorded via a small RAII helper into a file-scope struct,
keeping all the plumbing inside Heap.cpp so the public Heap.h surface
stays untouched. Sweep stats now travel back to collect_garbage() the
same way, which lets the report move out of sweep_dead_cells() into a
single print_gc_report() helper run after every phase has completed.
Switch the per-GC report to a precise timer and microsecond output, and
format all byte counts via human_readable_size so they read naturally
(e.g. "12.3 MiB" instead of "12923847 bytes").
Use a direct anonymous mapping for POSIX BlockAllocator chunks and trim
any temporary padding needed to make the live 2 MiB chunk HeapBlock-
aligned.
The GC only needs each 16 KiB HeapBlock slot aligned so from_cell() can
recover the block base by masking low bits. Request that alignment from
mach_vm_map() as well, rather than aligning whole chunks to 2 MiB.
Some JS::Object subclasses need the GC-cell macro plumbing while still
computing their class_name() dynamically.
Add a GC_CELL helper that provides the Base alias and heap friendship
without defining class_name(), then expose it through LibJS. Teach the
LibJS GC Clang plugin to treat this helper as a JS_OBJECT-style macro so
the existing validation still checks the class name and base class
arguments.
This commit splits out synchronization primitives from LibThreading into
LibSync. This is because LibThreading depends on LibCore, while LibCore
needs the synchronization primitives from LibThreading. This worked
while they were header only, but when I tried to add an implementation
file it ran into the circular dependency. To abstract away the pthread
implementation using cpp files is necessary so the synchronization
primitives were moved to a separate library.
MADV_FREE is lazy: the kernel only reclaims pages once it actually
needs them. As a result, blocks freed during a transient burst of
allocation continue to count toward our RSS for an arbitrarily long
time after a busy page goes idle.
Switch to MADV_DONTNEED on Linux so freed blocks drop out of RSS
immediately. macOS keeps the existing FREE_REUSABLE/FREE_REUSE paired
protocol (which integrates with its RSS accounting and gives the same
"eager release" behavior).
On cloudflare.com, some big GCs now drop ~350 MB instantly instead of
accumulating into a long-lived MADV_FREE backlog.
The slot lists used to be drained in a random order to make heap
layout less predictable, but on top of the per-CellAllocator type-
isolation we already enforce, the security delta is negligible.
LIFO via take_last() gives us better cache locality (we hand out
the most recently freed slot, which is more likely to be warm) and
saves a get_random_uniform call on every allocation.
deallocate_block() used to call MADV_FREE_REUSABLE / MADV_FREE /
MADV_DONTNEED inline on every freed block. With sweep typically
freeing many blocks per GC, the cumulative syscall cost shows up
as real GC pause time.
Move the work onto a single global "decommit worker" thread:
- deallocate_block now just poisons the slot and pushes it onto a
per-allocator m_freshly_freed queue. No syscalls.
- allocate_block prefers m_freshly_freed over m_blocks, so a slot
that's recycled before the worker sees it skips the
REUSABLE/REUSE pair entirely. This is the main payoff.
- Heap::sweep_dead_cells kicks the worker at the end of sweep.
The worker sleeps 50 ms after each kick to give the JS thread
breathing room, then drains each registered allocator's
m_freshly_freed, madvises slots in batches of 64 with
sched_yield between batches, and splices them onto m_blocks.
- Per-allocator refcount + condvar lets ~BlockAllocator wait
until the worker has dropped its reference before our storage
goes away. (Chunks themselves remain leaked: type-isolated VM
is permanent, so we never tear them down.)
Previously every 16 KiB HeapBlock was its own posix_memalign /
mach_vm_map / VirtualAlloc, which churned VMAs and made the kernel's
vm_area_struct list balloon for any non-trivial heap.
Carve slots out of 2 MiB chunks instead. The kernel now sees one
mmap per 128 blocks. Chunks are owned exclusively by a single
BlockAllocator and are never released back to the OS or shared
across allocators -- that's how we keep the heap's VM permanently
type-isolated, where a virtual address used for a cell of type T
is never reused for any other type. We don't bother tracking chunk
bases for teardown: the destructor leaks them by design.
Per-block memory return is preserved: deallocate_block still calls
MADV_FREE_REUSABLE / MADV_FREE / MADV_DONTNEED / DiscardVirtualMemory
so the kernel can reclaim physical pages under pressure.
Add a Cell hook for externally owned memory, and retally live external
bytes while sweeping after a collection.
Use the combined live cell and external byte count when sizing the next
GC threshold. External allocation notifications also participate in the
allocation-since-GC trigger.
Establish post-collection heap thresholds with a 1.75x growth factor
over live byte count, with an 8 MiB minimum.
These constants were chosen based on a benchmark sweep of the
Speedometer browser benchmarks (and a wider set of JS workloads) — see
the PR description for the data behind the choice.
Keep the constants in Heap.cpp instead of Heap.h so future tweaks don't
trigger 1000+ file rebuilds.
In sanitizer builds, we need to convert the fake ASan stack pointers to
the real one in order to perform a conservative scan. We were blindly
scanning these stack frames regardless of whether they belong to the
_active_ stack range, i.e. the current function's frame and everything
above it. It's very likely that stale pointers exist below the stack
pointer, and we now take care to exclude that range.
Fixes a flake in the LibJS builtins/WeakRef/WeakRef.prototype.deref.js
test.
The #pragma once was placed after the #include directives instead of
immediately after the copyright comment, inconsistent with every other
header file
The Weak<T>::operator=(U const&) template incorrectly compares the
internal pointer directly against the `other` reference. If this
template is instantiated, it causes a compilation error because a Ptr<T>
cannot be compared directly to a U const&
