Store parser errors, source range filenames, source code filenames,
module source, and Rust parser errors as UTF-16 where they flow back
into JavaScript-visible strings. Keep byte-oriented source buffers
byte-backed.
Remove temporary PrimitiveString, ByteString, and UTF-8 detours from
JSON, RegExp, module debug logging, print formatting, and tests.
Store JavaScript bytecode side data in the WebContent HTTP memory
cache and replay it when serving cached responses. Also update an
already-complete memory-cache entry when asynchronous bytecode cache
generation finishes, so the first source-only response does not keep
shadowing the disk-cache sidecar during same-process navigations.
Keep the HTTP memory-cache backfill keyed with the request headers that
populated the memory-cache entry, so Vary responses still receive their
generated bytecode sidecar.
Add LibHTTP coverage for round-tripping bytecode side data through a
memory-cache entry, attaching it after the response body has already
been cached, and matching Vary headers during updates. Add LibWeb
coverage for preserving the memory-cache request headers when cloning
responses.
Enable -Wexit-time-destructors for all in-tree library targets and
update process-lifetime library statics so they no longer register
exit-time destructors. Long-lived caches, lookup tables, singleton
registries, and generated constants now use NeverDestroyed or leaked
references where the data is intended to live until process exit.
Update LibWeb, LibLine, and the binding generators so regenerated
sources follow the same rule instead of reintroducing destructed
statics.
The preload list is static, immutable data compiled into LibHTTP.
ResourceLoader consults it in-process at the fetch layer before
falling back to the dynamic, per-profile store in the browser
process, so a preloaded host is upgraded without a synchronous IPC.
HSTSStore stays the dynamic store only; the preload list cannot be
unset by a max-age=0 response because it is consulted first, before
the store is ever queried.
When an HTTPS response carries a Strict-Transport-Security header, the
received policy is now respected. Subsequent HTTP requests to a known
HSTS host are upgraded to HTTPS before the fetch algorithm makes
further decisions such as CORS and mixed content.
Fixes tpexpress.co.uk, where an XHR redirects HTTPS -> HTTP -> HTTPS,
relying on a HSTS policy received on the document response to avoid the
CORS failure.
Represent WebIDL C++ types with a single CppType model that tracks
nullability, optional presence, and contained storage.
GC-like values now use GC::Ref/GC::Ptr directly, while containers choose
"plain", "Root", or "Conservative" container types depending on what
they contain. For example, sequence<Element> becomes a RootVector of
GC::Ref values, while sequence<SomeDictionary> becomes a
ConservativeVector only when the dictionary contains GC-like values.
This moves the generated bindings away from wrapping GC values in
GC::Root by default.
This has broad fallout as the types passed to interfaces for GC
objects changes almost fully across the board.
Carry the initiating URL, fetch destination, initiator type, and request
mode from Fetch into LoadRequest and pass them to ContentBlocker. The
existing substring matcher still matches only the request URL, but the
blocker API now receives the context needed by source-aware backends.
Preserve that metadata for CORS preflights and connection hints, and add
coverage for resource-type mapping and blob source URL normalization.
Similar to GC::Root<T>, make GC::RootVector<T> constructible without
explicitly passing a Heap.
This is implemented by having RootVectorBase use GC::Heap::the() for
heap-free construction.
Keep consumed response body bytes in Core::ImmutableBytes instead of
requiring a ByteBuffer. This lets responses that already arrived as
file-backed immutable data keep that representation through body
consumption, while streamed responses can still adopt their
accumulated ByteBuffer without another copy.
Update the body consumers that only inspect bytes to read from
immutable byte views. Font loading still copies at its existing
ownership boundary, where the off-thread preparation path takes a
ByteBuffer.
Allow parser dump utilities to run without installing a ResourceLoader.
Short-circuit resource fetch entry points used by links, stylesheets,
and shared resource requests when no loader is present.
Keep those failures local to headless utility use so dump-html-tree can
parse real pages without a full browser page loader or request service.
Map JavaScript bytecode cache sidecars from the HTTP disk cache instead
of copying them into anonymous shared buffers while handing response
headers to WebContent. Store the mapped data as ImmutableBytes on the
fetch response so script fetching can decode directly from the mapped
sidecar bytes.
Add LibHTTP coverage for retrieving associated cache data as a mappable
file, alongside the existing byte-buffer retrieval API.
Let CacheEntryWriter expose an explicit body-file handoff after a
successful flush. RequestServer sends that file to WebContent before the
request finishes, so Fetch can replace its retained memory-cache buffer
with a file-backed ImmutableBytes body.
Keep the plain flush path fd-free for existing callers, and add coverage
for mapping the body file returned by the writer. The focused disk and
memory cache web tests continue to pass.
Carry response chunks through LibRequests and ResourceLoader as a
ResponseData wrapper. A disk cache hit can retain its mapped storage,
while ordinary streamed chunks still use borrowed bytes.
Store completed memory cache entries as Core::ImmutableBytes. This lets
mapped disk-cache responses stay file-backed while retained by the HTTP
memory cache, instead of forcing another ByteBuffer copy.
When a link element starts a replacement fetch, the old response body
can already have queued its consume-body task. stop_fetch() clears the
fetch algorithms for future work, so that late task re-read an empty
callback and tripped AK::Function's wrapper assertion.
Keep fetch_response_handover() tied to the algorithms object it
selected when creating the consume-body steps. Then make
HTMLLinkElement apply the HTML stale-resource check explicitly with a
fetch generation, so results from superseded link fetches return before
processing the resource.
Add a text test that replaces a stylesheet data URL and waits for the
replacement stylesheet's load event. This covers stale body callbacks
and verifies that superseded stylesheet results do not apply.
The lichess.org lobby stylesheet sometimes gets render-blocked,
resulting in a blank page.
The main fetch would queue work and then spin_until wait for a pending
preload result on the main thread. This re-entered the event loop and
stranded the queued PendingResponse callback behind a pile of nested
event processing. So the preload response existed, but the stylesheet
never made it through the normal handoff path.
I chose to follow the PreloadEntry.cpp model to address this and track
at most one pending preload PendingResponse for each FetchParams,
resolving it directly from the preload callback when the response
arrives. This removes the spin_until, keeps the handoff asynchronous,
and lets the blocked stylesheet finish loading normally.
If the current single-consumer preload path ever changes, this logic
must be widened to handle it.
Attach cached JavaScript bytecode sidecars to HTTP response headers so
WebContent can materialize classic and module scripts directly from a
decoded cache blob on cache hits.
Carry the disk cache vary key with the sidecar and reuse it when storing
fresh bytecode, avoiding mismatches against the augmented network
request headers used to create the cache entry.
Keep CORS-filtered module responses intact for status, MIME, and script
creation checks. Read bytecode sidecar data only from the internal
response, and treat decode or materialization failure as a cache miss
that falls back to normal source compilation.
Previously, each chunk was delivered through a pull promise. Every pull
allocated around seven GC objects (the pull promise, its capability, and
the reaction handlers from react_to_promise) and none of them did
anything useful. The pull algorithm is now a no-op, and
FetchedDataReceiver enqueues bytes into the controller as they arrive.
When the HTML parser blocks on a synchronous external script, run a
separate tokenizer over the unparsed input and issue speculative fetches
for the resources it finds (script src, link rel=stylesheet|preload, img
src), with <base href> tracking and template/foreign-content skipping.
Also fills in the previously-stubbed "consume a preloaded resource"
algorithm and the document's "map of preloaded resources", so that
<link rel="preload"> followed by a matching consumer deduplicates to
a single fetch.
This matches the behavior of other engines. Some CDNs that do content
negotiation will fall back to non alpha-preserving formats if these
values are not present.
When receiving a 401 response, we previously entered the credential
prompt path for any request with a WWW-Authenticate header. This caused
spurious "Username/password prompt is not implemented" warnings and
unnecessary request retries for auth schemes like PrivateToken
(RFC 9577) that don't involve user credential prompts.
Only enter the 401 handling path when the WWW-Authenticate header
contains a scheme we can handle with a username/password prompt
(Basic or Digest). This is consistent with how Chromium, Firefox,
and WebKit all silently skip unrecognized auth schemes.
Fonts require CORS mode per spec, but file:// origins are opaque in
our implementation, so the standard same-origin check in main_fetch
always fails for file-to-file font loads. This caused @font-face rules
referencing local .woff2 files to fail with a NetworkError.
Fix this by treating file:// font loads from file:// pages as
same-origin, routing them through scheme_fetch with basic response
tainting instead of rejecting them for not being HTTP(S).
This matches the behavior of Chromium, WebKit, and Firefox, all of
which allow local font loading from local pages.
This allows the Request to be cleaned up when it becomes inactive,
which in turn allows the GC to clean up the FetchController which is
indirectly captured by a root in the Request callbacks.
To avoid some churn, Body::source() returns the SourceTypeInternal
object, as it's not exposed to JS. The constructor is also adjusted so
that Body::clone() doesn't have to convert to a SourceType and then
immediately back again.
The AD-HOC guard for HTTP 401 retry checks whether the request
contains a WWW-Authenticate header, but WWW-Authenticate is a
response header sent by the server to indicate which
authentication schemes it accepts. The check uses
request->header_list() where response->header_list() is
intended. Since requests never carry this header, the guard
never matches and the entire 401 retry path is dead code.
Change the parameters types of the functions `coarsen_time` and
`coarsened_shared_current_time` from `bool` to
`CanUseCrossOriginIsolatedAPIs` for more coherence with the surrounding
code.
We currently attach HTTP cookie headers from LibWeb within Fetch. This
has the downside that the cookie IPC, and the infrastructure around it,
are all synchronous. This blocks the WebContent process entirely while
the cookie is being retrieved, for every request on a page.
We now attach cookie headers from RequestServer. The state machine in
RequestServer::Request allows us to easily do this work asynchronously.
We can also skip this work entirely when the response is served from
disk cache.
Note that we will continue to parse cookies in the WebContent process.
If something goes awry during parsing. we limit the damage to that
process, instead of the UI or RequestServer.
Also note that WebSocket requests still have cookie headers attached
attached from LibWeb. This will be handled in a future patch.
In the future, we may want to introduce a memory cache for cookies in
RequestServer to avoid IPC altogether as able.
This patch introduces a cookie cache in the WebContent process to reduce
blocking IPC calls when JS accesses document.cookie. The UI process now
maintains a cookie version counter per-domain in shared memory. When JS
reads document.cookie, we check whether we have a valid cached cookie by
comparing the current shared version to the last used version. If they
match, the cached cookie is returned without IPC.
This optimization is based on Chromium's shared versioning, in which it
was observed that 87% of document.cookie accesses were redundant. See:
https://blog.chromium.org/2024/06/introducing-shared-memory-versioning-to.html
Note that this cache only supports document.cookie, not HTTP Cookie
headers. HTTP cookies are attached to requests with varying URLs and
paths. The cookies that match the document URL might not match the
request URL, which we wouldn't know from WebContent. So attaching the
cached document cookie would be incorrect.
On https://twinings.co.uk, we see approximately 600 document.cookie
requests while the page loads. This patch reduces the time spent in
the document.cookie getter from ~45ms to 2-3ms.
By creating a new FetchController here, we prevent the original
FetchController from affecting the actual ongoing request. This is
necessary to allow FetchController::stop_fetch() to stop an ongoing
fetch in the subsequent commit.
The purpose of the copy appears to be only to change the
httpFetchParams' request over to the newly-cloned httpRequest, so all
existing references should refer back to the originals rather than new
instances.
Previously, when loading a document, we would try to sniff the MIME
type by reading from the response body's source. However, for streaming
HTTP responses, the body source is Empty (the data comes through the
stream instead), so we had no bytes to sniff.
This caused pages like hypr.land (which sends no Content-Type header)
to be misidentified as plain text instead of HTML, since the MIME
sniffing algorithm would receive zero bytes and fall back to the
default type.
The fix captures the first bytes of the response body during fetch,
storing them on the Body object. These bytes are the "resource header"
defined by the MIME Sniffing spec - up to 1445 bytes, which is enough
to identify any MIME type the spec can detect.
Since bytes may arrive asynchronously during streaming, we use a
callback mechanism: if bytes aren't ready yet when load_document()
needs them, it registers a callback that fires once enough bytes have
been captured (or the stream ends).
The flow is:
1. FetchedDataReceiver receives network bytes, buffers them
2. When Body is created, buffered bytes are flushed to Body's sniff
buffer, and subsequent bytes are appended as they arrive
3. Before calling load_document(), Navigable waits for sniff bytes
4. load_document() passes the bytes to MimeSniff::Resource::sniff()
We now partition the HTTP disk cache based on the Vary response header.
If a cached response contains a Vary header, we look for each of the
header names in the outgoing HTTP request. The outgoing request must
match every header value in the original request for the cache entry
to be used; otherwise, a new request will be issued, and a separate
cache entry will be created.
Note that we must now defer creating the disk cache file itself until we
have received the response headers. The Vary key is computed from these
headers, and affects the partitioned disk cache file name.
There are further optimizations we can make here. If we have a Vary
mismatch, we could find the best candidate cached response and issue a
conditional HTTP request. The content server may then respond with an
HTTP 304 if the mismatched request headers are actually okay. But for
now, if we have a Vary mismatch, we issue an unconditional request as
a purely correctness-oriented patch.
If the cache mode is no-store, we must not interact with the cache at
all.
If the cache mode is reload, we must not use any cached response.
If the cache-mode is only-if-cached or force-cache, we are permitted
to respond with stale cache responses.
Note that we currently cannot test only-if-cached in test-web. Setting
this mode also requires setting the cors mode to same-origin, but our
http-test-server infra requires setting the cors mode to cors.
Returning null here results in the fetch cache mode becoming hard-set to
no-store. This means the HTTP cache cannot be consulted nor updated. A
future commit will make our disk cache respect this flag, as this is a
valid client-provided value. So instead of setting this flag when the
memory cache is disabled, let's move the check to where the cache later
becomes consulted/updated.
We currently will perform some revalidation from both WebContent and
RequestServer. For simplicity's sake, now that the memory cache only
holds fresh responses, let's remove revalidation handling from the
WebContent process. If a memory-cached response is stale, it's fine
to just forward that request to RequestServer. It will then either
be served by disk cache, or revalidated at that point.
Once a cache entry is not fresh, we now remove it from the memory cache.
We will avoid handling revalidation from within WebContent. Instead, we
will just forward the request to RequestServer, where the disk cache
will handle revalidation for itself if needed.
When fully reading a response body fails, the transform stream's flush
algorithm (which calls processResponseEndOfBody) never runs since flush
only executes on successful stream completion.
This left report_timing_steps null, causing a crash when
process_response_consume_body tried to call report_timing.
The fetch spec doesn't explicitly handle this case. We work around it
by extracting the report timing setup (steps 1-3 of
processResponseEndOfBody) into a helper that we call from both the
success path (via flush) and the error path (via processBodyError).
This is an ad-hoc fix that deviates slightly from the spec, but it
ensures failed fetches still produce useful timing data and don't crash.
Propagate the request initiator type (e.g., "xmlhttprequest", "fetch",
"script", "stylesheet") from LibWeb through the IPC layer to DevTools.
This enables Firefox DevTools to correctly identify XHR/fetch requests
and display appropriate cause types in the Network panel's "Initiator"
column.
We currently do not handle responses for range requests at all in our
HTTP caches. This means if we issue a request for a range of bytes=1-10,
that response will be served to a subsequent request for a range of
bytes=10-20. This is obviously invalid - so until we handle these
requests, just don't cache them for now.
