Store parser errors, source range filenames, source code filenames,
module source, and Rust parser errors as UTF-16 where they flow back
into JavaScript-visible strings. Keep byte-oriented source buffers
byte-backed.
Remove temporary PrimitiveString, ByteString, and UTF-8 detours from
JSON, RegExp, module debug logging, print formatting, and tests.
Store JavaScript bytecode side data in the WebContent HTTP memory
cache and replay it when serving cached responses. Also update an
already-complete memory-cache entry when asynchronous bytecode cache
generation finishes, so the first source-only response does not keep
shadowing the disk-cache sidecar during same-process navigations.
Keep the HTTP memory-cache backfill keyed with the request headers that
populated the memory-cache entry, so Vary responses still receive their
generated bytecode sidecar.
Add LibHTTP coverage for round-tripping bytecode side data through a
memory-cache entry, attaching it after the response body has already
been cached, and matching Vary headers during updates. Add LibWeb
coverage for preserving the memory-cache request headers when cloning
responses.
Represent WebIDL C++ types with a single CppType model that tracks
nullability, optional presence, and contained storage.
GC-like values now use GC::Ref/GC::Ptr directly, while containers choose
"plain", "Root", or "Conservative" container types depending on what
they contain. For example, sequence<Element> becomes a RootVector of
GC::Ref values, while sequence<SomeDictionary> becomes a
ConservativeVector only when the dictionary contains GC-like values.
This moves the generated bindings away from wrapping GC values in
GC::Root by default.
This has broad fallout as the types passed to interfaces for GC
objects changes almost fully across the board.
Keep consumed response body bytes in Core::ImmutableBytes instead of
requiring a ByteBuffer. This lets responses that already arrived as
file-backed immutable data keep that representation through body
consumption, while streamed responses can still adopt their
accumulated ByteBuffer without another copy.
Update the body consumers that only inspect bytes to read from
immutable byte views. Font loading still copies at its existing
ownership boundary, where the off-thread preparation path takes a
ByteBuffer.
Map JavaScript bytecode cache sidecars from the HTTP disk cache instead
of copying them into anonymous shared buffers while handing response
headers to WebContent. Store the mapped data as ImmutableBytes on the
fetch response so script fetching can decode directly from the mapped
sidecar bytes.
Add LibHTTP coverage for retrieving associated cache data as a mappable
file, alongside the existing byte-buffer retrieval API.
The lichess.org lobby stylesheet sometimes gets render-blocked,
resulting in a blank page.
The main fetch would queue work and then spin_until wait for a pending
preload result on the main thread. This re-entered the event loop and
stranded the queued PendingResponse callback behind a pile of nested
event processing. So the preload response existed, but the stylesheet
never made it through the normal handoff path.
I chose to follow the PreloadEntry.cpp model to address this and track
at most one pending preload PendingResponse for each FetchParams,
resolving it directly from the preload callback when the response
arrives. This removes the spin_until, keeps the handoff asynchronous,
and lets the blocked stylesheet finish loading normally.
If the current single-consumer preload path ever changes, this logic
must be widened to handle it.
Attach cached JavaScript bytecode sidecars to HTTP response headers so
WebContent can materialize classic and module scripts directly from a
decoded cache blob on cache hits.
Carry the disk cache vary key with the sidecar and reuse it when storing
fresh bytecode, avoiding mismatches against the augmented network
request headers used to create the cache entry.
Keep CORS-filtered module responses intact for status, MIME, and script
creation checks. Read bytecode sidecar data only from the internal
response, and treat decode or materialization failure as a cache miss
that falls back to normal source compilation.
Previously, each chunk was delivered through a pull promise. Every pull
allocated around seven GC objects (the pull promise, its capability, and
the reaction handlers from react_to_promise) and none of them did
anything useful. The pull algorithm is now a no-op, and
FetchedDataReceiver enqueues bytes into the controller as they arrive.
If we've already fired off an error, calling stop_fetch() should
make no difference, other than stopping the Requests::Request.
Eventually, we'll probably want abort() and terminate() to
eventually stop the Requests::Request in an unobservable way.
This allows the Request to be cleaned up when it becomes inactive,
which in turn allows the GC to clean up the FetchController which is
indirectly captured by a root in the Request callbacks.
To avoid some churn, Body::source() returns the SourceTypeInternal
object, as it's not exposed to JS. The constructor is also adjusted so
that Body::clone() doesn't have to convert to a SourceType and then
immediately back again.
Change the parameters types of the functions `coarsen_time` and
`coarsened_shared_current_time` from `bool` to
`CanUseCrossOriginIsolatedAPIs` for more coherence with the surrounding
code.
By creating a new FetchController here, we prevent the original
FetchController from affecting the actual ongoing request. This is
necessary to allow FetchController::stop_fetch() to stop an ongoing
fetch in the subsequent commit.
The purpose of the copy appears to be only to change the
httpFetchParams' request over to the newly-cloned httpRequest, so all
existing references should refer back to the originals rather than new
instances.
Previously, when loading a document, we would try to sniff the MIME
type by reading from the response body's source. However, for streaming
HTTP responses, the body source is Empty (the data comes through the
stream instead), so we had no bytes to sniff.
This caused pages like hypr.land (which sends no Content-Type header)
to be misidentified as plain text instead of HTML, since the MIME
sniffing algorithm would receive zero bytes and fall back to the
default type.
The fix captures the first bytes of the response body during fetch,
storing them on the Body object. These bytes are the "resource header"
defined by the MIME Sniffing spec - up to 1445 bytes, which is enough
to identify any MIME type the spec can detect.
Since bytes may arrive asynchronously during streaming, we use a
callback mechanism: if bytes aren't ready yet when load_document()
needs them, it registers a callback that fires once enough bytes have
been captured (or the stream ends).
The flow is:
1. FetchedDataReceiver receives network bytes, buffers them
2. When Body is created, buffered bytes are flushed to Body's sniff
buffer, and subsequent bytes are appended as they arrive
3. Before calling load_document(), Navigable waits for sniff bytes
4. load_document() passes the bytes to MimeSniff::Resource::sniff()
We currently will perform some revalidation from both WebContent and
RequestServer. For simplicity's sake, now that the memory cache only
holds fresh responses, let's remove revalidation handling from the
WebContent process. If a memory-cached response is stale, it's fine
to just forward that request to RequestServer. It will then either
be served by disk cache, or revalidated at that point.
Once a cache entry is not fresh, we now remove it from the memory cache.
We will avoid handling revalidation from within WebContent. Instead, we
will just forward the request to RequestServer, where the disk cache
will handle revalidation for itself if needed.
No need to duplicate this in LibWeb.
In doing so, this also fixes an apparent bug for SWR handling in LibWeb.
We were previously deciding if we were in the SWR lifetime with:
stale_while_revalidate > current_age
However, the SWR lifetime is meant to be an additional time on top of
the freshness lifetime:
freshness_lifetime + stale_while_revalidate > current_age
There are a couple of remaining RFC 9111 methods in LibWeb's Fetch, but
these are currently directly tied to the way we store GC-allocated HTTP
response objects. So de-coupling that is left as a future exercise.
The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP
caching) implementation. We currently have one implementation in LibWeb
for our in-memory cache and another in RequestServer for our disk cache.
The implementations both largely revolve around interacting with HTTP
headers. But in LibWeb, we are using Fetch's header infra, and in RS we
are using are home-grown header infra from LibHTTP.
So to give these a common denominator, this patch replaces the LibHTTP
implementation with Fetch's infra. Our existing LibHTTP implementation
was not particularly compliant with any spec, so this at least gives us
a standards-based common implementation.
This migration also required moving a handful of other Fetch AOs over
to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP
folder, so perhaps it makes sense for LibHTTP to be the implementation
of that entire set of facilities.)
An upcoming commit will migrate the contents of Headers.h/cpp to LibHTTP
for use outside of LibWeb. These CORS and MIME helpers depend on other
LibWeb facilities, however, so they cannot be moved.
Fixes a regression from commit:
f675cfe90f
It is not sufficient to only check if the builder is empty, as we will
then drop empty header values (when the first found value is empty).
This is tested in WPT by /cors/origin.htm, but that requires an HTTP
server.
The spec declares these as a byte sequence, which we then implemented as
a ByteBuffer. This has become pretty awkward to deal with, as evidenced
by the plethora of `MUST(ByteBuffer::copy(...))` and `.bytes()` calls
everywhere inside Fetch. We would then treat the bytes as a string
anyways by wrapping them in StringView everywhere.
We now store these as a ByteString. This is more comfortable to deal
with, and we no longer need to continually copy underlying storage (as
ByteString is ref-counted).
This work is largely preparatory for an upcoming HTTP header refactor.
Generally just define things in the order they are declared (will make a
change to use ByteString in this file a bit easier to follow). Also make
a couple of free functions be class methods on Header / HeaderList.
Disallow calling `StringBase::bytes()` on temporaries to avoid returning
`ReadonlyBytes` that outlive the underlying string.
With this change, we catch a real UAF:
`load_result.data = maybe_response.release_value().bytes();`
All other updated call sites were already safe, they just needed to use
an intermediate named variable to satisfy the new lvalue-only
requirement.
HTMLLinkElement is the final user of Resource/ResourceClient (used for
preloads and icons). This ports these link types to use fetch according
to the spec.
Preloads were particularly goofy because they would be stored in the
ResourceLoader's ad-hoc cache. But this cache was never consulted for
organic loads, thus were never used. There is more work to be done to
use these preloads within fetch, but for now they at least are stored
in fetch's HTTP cache for re-use.
This first pass only applies to the following two cases:
- Public functions returning a view type into an object they own
- Public ctors storing a view type
This catches a grand total of one (1) issue, which is fixed in
the previous commit.
Note that it's not actually executing tasks in parallel, it's still
throwing them on the HTML event loop task queue, each with its own
unique task source.
This makes our fetch implementation a lot more robust when HTTP caching
is enabled, and you can now click links on https://terminal.shop/
without hitting TODO assertions in fetch.
