dmaax/ladybird
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ladybird/Libraries/LibWeb/Fetch/Fetching/FetchedDataReceiver.cpp

182 lines
7.2 KiB
C++
Raw Permalink Normal View History

LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
/*
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 12:23:56 -03:00
* Copyright (c) 2024-2026, Tim Flynn <trflynn89@ladybird.org>
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
* Copyright (c) 2025, Aliaksandr Kalenik <kalenik.aliaksandr@gmail.com>
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
*
* SPDX-License-Identifier: BSD-2-Clause
*/
LibGC+Everywhere: Factor out a LibGC from LibJS Resulting in a massive rename across almost everywhere! Alongside the namespace change, we now have the following names: * JS::NonnullGCPtr -> GC::Ref * JS::GCPtr -> GC::Ptr * JS::HeapFunction -> GC::Function * JS::CellImpl -> GC::Cell * JS::Handle -> GC::Root
2024-11-14 12:01:23 -03:00
#include <LibGC/Function.h>
LibHTTP+LibWeb: Store the in-memory HTTP cache without JS realms The in-memory HTTP Fetch cache currently keeps the realm which created each cache entry alive indefinitely. This patch migrates this cache to LibHTTP, to ensure it is completely unaware of any JS objects. Now that we are not interacting with Fetch response objects, we can no longer use Streams infrastructure to pipe the response body into the Fetch response. Fetch also ultimately creates the cache response once the HTTP response headers have arrived. So the LibHTTP cache will hold entries in a pending list until we have received the entire response body. Then it is moved to a completed list and may be used thereafter.
2025-12-21 10:58:55 -03:00
#include <LibHTTP/Cache/MemoryCache.h>
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
#include <LibWeb/Bindings/ExceptionOrUtils.h>
#include <LibWeb/Fetch/Fetching/FetchedDataReceiver.h>
#include <LibWeb/Fetch/Infrastructure/FetchParams.h>
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 08:25:01 -03:00
#include <LibWeb/Fetch/Infrastructure/HTTP/Bodies.h>
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 12:23:56 -03:00
#include <LibWeb/Fetch/Infrastructure/HTTP/Responses.h>
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
#include <LibWeb/HTML/Scripting/TemporaryExecutionContext.h>
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
#include <LibWeb/Platform/EventLoopPlugin.h>
#include <LibWeb/Streams/ReadableByteStreamController.h>
LibWeb: Move ReadableStream AOs into their own file The main streams AO file has gotten very large, and is a bit difficult to navigate. In an effort to improve DX, this migrates ReadableStream AOs to their own file. And the helper classes used for the tee and pipe- to operations are also in their own files.
2025-04-17 16:47:53 -03:00
#include <LibWeb/Streams/ReadableStream.h>
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
#include <LibWeb/Streams/ReadableStreamOperations.h>
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
namespace Web::Fetch::Fetching {
LibGC+Everywhere: Factor out a LibGC from LibJS Resulting in a massive rename across almost everywhere! Alongside the namespace change, we now have the following names: * JS::NonnullGCPtr -> GC::Ref * JS::GCPtr -> GC::Ptr * JS::HeapFunction -> GC::Function * JS::CellImpl -> GC::Cell * JS::Handle -> GC::Root
2024-11-14 12:01:23 -03:00
GC_DEFINE_ALLOCATOR(FetchedDataReceiver);
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
LibHTTP+LibWeb: Store the in-memory HTTP cache without JS realms The in-memory HTTP Fetch cache currently keeps the realm which created each cache entry alive indefinitely. This patch migrates this cache to LibHTTP, to ensure it is completely unaware of any JS objects. Now that we are not interacting with Fetch response objects, we can no longer use Streams infrastructure to pipe the response body into the Fetch response. Fetch also ultimately creates the cache response once the HTTP response headers have arrived. So the LibHTTP cache will hold entries in a pending list until we have received the entire response body. Then it is moved to a completed list and may be used thereafter.
2025-12-21 10:58:55 -03:00
FetchedDataReceiver::FetchedDataReceiver(GC::Ref<Infrastructure::FetchParams const> fetch_params, GC::Ref<Streams::ReadableStream> stream, RefPtr<HTTP::MemoryCache> http_cache)
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
: m_fetch_params(fetch_params)
, m_stream(stream)
LibHTTP+LibWeb: Store the in-memory HTTP cache without JS realms The in-memory HTTP Fetch cache currently keeps the realm which created each cache entry alive indefinitely. This patch migrates this cache to LibHTTP, to ensure it is completely unaware of any JS objects. Now that we are not interacting with Fetch response objects, we can no longer use Streams infrastructure to pipe the response body into the Fetch response. Fetch also ultimately creates the cache response once the HTTP response headers have arrived. So the LibHTTP cache will hold entries in a pending list until we have received the entire response body. Then it is moved to a completed list and may be used thereafter.
2025-12-21 10:58:55 -03:00
, m_http_cache(move(http_cache))
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
{
}
FetchedDataReceiver::~FetchedDataReceiver() = default;
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 08:25:01 -03:00
void FetchedDataReceiver::set_body(GC::Ref<Fetch::Infrastructure::Body> body)
{
m_body = body;
// Flush any bytes that were buffered before the body was set
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
if (!m_pre_body_sniff_buffer.is_empty()) {
m_body->append_sniff_bytes(m_pre_body_sniff_buffer);
m_pre_body_sniff_buffer.clear();
}
LibWeb: Fix race in MIME sniff bytes when response completes early When a streaming HTTP response completes before set_body() is called on the FetchedDataReceiver, the sniff bytes would never be marked as complete, causing navigation to hang intermittently. The sequence that triggers this: 1. handle_network_bytes(data, Ongoing) runs, but m_body is null, so bytes only go into the FetchedDataReceiver's own m_buffer. 2. handle_network_bytes({}, Complete) runs, but m_body is still null, so the m_body->set_sniff_bytes_complete() call is skipped. 3. set_body(body) is called, which flushes m_buffer into the body via append_sniff_bytes(), but never marks them as complete. 4. populate_session_history_entry_document() calls sniff_bytes_if_available() on the body. Since the source is Empty (streaming) and m_sniff_bytes_complete is false, it returns no value. 5. The async path registers a callback via wait_for_sniff_bytes(), but since the stream already completed, no more data arrives, and the callback never fires. Navigation hangs. Fix this by checking the lifecycle state in set_body(). If we have already moved past the Receiving state, the Complete was already processed and we need to mark sniff bytes as complete now.
2026-03-08 04:29:33 -03:00
// If the stream already completed before the body was set,
LibWeb: Preserve file-backed HTTP cache bodies Carry response chunks through LibRequests and ResourceLoader as a ResponseData wrapper. A disk cache hit can retain its mapped storage, while ordinary streamed chunks still use borrowed bytes. Store completed memory cache entries as Core::ImmutableBytes. This lets mapped disk-cache responses stay file-backed while retained by the HTTP memory cache, instead of forcing another ByteBuffer copy.
2026-05-15 09:49:12 -03:00
// we missed the set_sniff_bytes_complete() call in handle_network_data.
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
if (m_network_complete)
LibWeb: Fix race in MIME sniff bytes when response completes early When a streaming HTTP response completes before set_body() is called on the FetchedDataReceiver, the sniff bytes would never be marked as complete, causing navigation to hang intermittently. The sequence that triggers this: 1. handle_network_bytes(data, Ongoing) runs, but m_body is null, so bytes only go into the FetchedDataReceiver's own m_buffer. 2. handle_network_bytes({}, Complete) runs, but m_body is still null, so the m_body->set_sniff_bytes_complete() call is skipped. 3. set_body(body) is called, which flushes m_buffer into the body via append_sniff_bytes(), but never marks them as complete. 4. populate_session_history_entry_document() calls sniff_bytes_if_available() on the body. Since the source is Empty (streaming) and m_sniff_bytes_complete is false, it returns no value. 5. The async path registers a callback via wait_for_sniff_bytes(), but since the stream already completed, no more data arrives, and the callback never fires. Navigation hangs. Fix this by checking the lifecycle state in set_body(). If we have already moved past the Receiving state, the Complete was already processed and we need to mark sniff bytes as complete now.
2026-03-08 04:29:33 -03:00
m_body->set_sniff_bytes_complete();
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 08:25:01 -03:00
}
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
void FetchedDataReceiver::visit_edges(Visitor& visitor)
{
Base::visit_edges(visitor);
visitor.visit(m_fetch_params);
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 12:23:56 -03:00
visitor.visit(m_response);
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 08:25:01 -03:00
visitor.visit(m_body);
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
visitor.visit(m_stream);
}
// This implements the parallel steps of the pullAlgorithm in HTTP-network-fetch.
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
// https://fetch.spec.whatwg.org/#ref-for-in-parallel⑤
LibWeb: Preserve file-backed HTTP cache bodies Carry response chunks through LibRequests and ResourceLoader as a ResponseData wrapper. A disk cache hit can retain its mapped storage, while ordinary streamed chunks still use borrowed bytes. Store completed memory cache entries as Core::ImmutableBytes. This lets mapped disk-cache responses stay file-backed while retained by the HTTP memory cache, instead of forcing another ByteBuffer copy.
2026-05-15 09:49:12 -03:00
void FetchedDataReceiver::handle_network_data(Requests::ResponseData data, NetworkState state)
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
{
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
if (state == NetworkState::Complete) {
LibWeb: Preserve file-backed HTTP cache bodies Carry response chunks through LibRequests and ResourceLoader as a ResponseData wrapper. A disk cache hit can retain its mapped storage, while ordinary streamed chunks still use borrowed bytes. Store completed memory cache entries as Core::ImmutableBytes. This lets mapped disk-cache responses stay file-backed while retained by the HTTP memory cache, instead of forcing another ByteBuffer copy.
2026-05-15 09:49:12 -03:00
VERIFY(data.bytes().is_empty());
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
m_network_complete = true;
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 08:25:01 -03:00
// Mark sniff bytes as complete when the stream ends
if (m_body)
m_body->set_sniff_bytes_complete();
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
// 2. Otherwise, if the bytes transmission for response’s message body is done normally and stream is readable,
// then close stream, and abort these in-parallel steps.
Platform::EventLoopPlugin::the().deferred_invoke(GC::create_function(heap(), [this]() {
close_stream();
}));
return;
LibWeb: Support MIME type sniffing for streaming HTTP responses Previously, when loading a document, we would try to sniff the MIME type by reading from the response body's source. However, for streaming HTTP responses, the body source is Empty (the data comes through the stream instead), so we had no bytes to sniff. This caused pages like hypr.land (which sends no Content-Type header) to be misidentified as plain text instead of HTML, since the MIME sniffing algorithm would receive zero bytes and fall back to the default type. The fix captures the first bytes of the response body during fetch, storing them on the Body object. These bytes are the "resource header" defined by the MIME Sniffing spec - up to 1445 bytes, which is enough to identify any MIME type the spec can detect. Since bytes may arrive asynchronously during streaming, we use a callback mechanism: if bytes aren't ready yet when load_document() needs them, it registers a callback that fires once enough bytes have been captured (or the stream ends). The flow is: 1. FetchedDataReceiver receives network bytes, buffers them 2. When Body is created, buffered bytes are flushed to Body's sniff buffer, and subsequent bytes are appended as they arrive 3. Before calling load_document(), Navigable waits for sniff bytes 4. load_document() passes the bytes to MimeSniff::Resource::sniff()
2026-01-24 08:25:01 -03:00
}
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 10:33:49 -03:00
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
if (state == NetworkState::Error)
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
return;
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
// 1. If one or more bytes have been transmitted from response’s message body, then:
LibWeb: Preserve file-backed HTTP cache bodies Carry response chunks through LibRequests and ResourceLoader as a ResponseData wrapper. A disk cache hit can retain its mapped storage, while ordinary streamed chunks still use borrowed bytes. Store completed memory cache entries as Core::ImmutableBytes. This lets mapped disk-cache responses stay file-backed while retained by the HTTP memory cache, instead of forcing another ByteBuffer copy.
2026-05-15 09:49:12 -03:00
auto bytes = data.bytes();
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
if (bytes.is_empty())
return;
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
// 1. Let bytes be the transmitted bytes.
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
// FIXME: 2. Let codings be the result of extracting header list values given `Content-Encoding` and response’s header list.
// FIXME: 3. Increase response’s body info’s encoded size by bytes’s length.
// FIXME: 4. Set bytes to the result of handling content codings given codings and bytes.
// FIXME: 5. Increase response’s body info’s decoded size by bytes’s length.
// FIXME: 6. If bytes is failure, then terminate fetchParams’s controller.
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
// Capture bytes for MIME sniffing
if (m_body) {
LibWeb: Preserve immutable consumed body bytes Keep consumed response body bytes in Core::ImmutableBytes instead of requiring a ByteBuffer. This lets responses that already arrived as file-backed immutable data keep that representation through body consumption, while streamed responses can still adopt their accumulated ByteBuffer without another copy. Update the body consumers that only inspect bytes to read from immutable byte views. Font loading still copies at its existing ownership boundary, where the off-thread preparation path takes a ByteBuffer.
2026-05-17 14:53:54 -03:00
if (auto const& immutable_bytes = data.immutable_bytes(); immutable_bytes.has_value() && immutable_bytes->is_file_backed() && m_body->source().has<Empty>() && bytes.size() == immutable_bytes->size())
m_body->set_source(*immutable_bytes, static_cast<u64>(immutable_bytes->size()));
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
m_body->append_sniff_bytes(bytes);
} else if (m_pre_body_sniff_buffer.size() < Infrastructure::MAX_SNIFF_BYTES) {
auto space_remaining = Infrastructure::MAX_SNIFF_BYTES - m_pre_body_sniff_buffer.size();
m_pre_body_sniff_buffer.append(bytes.slice(0, min(bytes.size(), space_remaining)));
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
}
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
RequestServer: Reuse disk-cache files after downloads Let CacheEntryWriter expose an explicit body-file handoff after a successful flush. RequestServer sends that file to WebContent before the request finishes, so Fetch can replace its retained memory-cache buffer with a file-backed ImmutableBytes body. Keep the plain flush path fd-free for existing callers, and add coverage for mapping the body file returned by the writer. The focused disk and memory cache web tests continue to pass.
2026-05-15 09:58:56 -03:00
if (m_http_cache && !m_cache_body_replaces_network_buffer) {
LibWeb: Preserve file-backed HTTP cache bodies Carry response chunks through LibRequests and ResourceLoader as a ResponseData wrapper. A disk cache hit can retain its mapped storage, while ordinary streamed chunks still use borrowed bytes. Store completed memory cache entries as Core::ImmutableBytes. This lets mapped disk-cache responses stay file-backed while retained by the HTTP memory cache, instead of forcing another ByteBuffer copy.
2026-05-15 09:49:12 -03:00
if (auto const& immutable_bytes = data.immutable_bytes(); immutable_bytes.has_value() && immutable_bytes->is_file_backed() && m_cache_buffer.is_empty() && !m_cache_body.has_value()) {
m_cache_body = *immutable_bytes;
} else {
if (m_cache_body.has_value()) {
m_cache_buffer.append(m_cache_body->bytes());
m_cache_body.clear();
}
m_cache_buffer.append(bytes);
}
}
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
// 7. Append bytes to buffer.
enqueue_into_stream(bytes);
// FIXME: 8. If the size of buffer is larger than an upper limit chosen by the user agent, ask the user agent
// to suspend the ongoing fetch.
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
}
RequestServer: Reuse disk-cache files after downloads Let CacheEntryWriter expose an explicit body-file handoff after a successful flush. RequestServer sends that file to WebContent before the request finishes, so Fetch can replace its retained memory-cache buffer with a file-backed ImmutableBytes body. Keep the plain flush path fd-free for existing callers, and add coverage for mapping the body file returned by the writer. The focused disk and memory cache web tests continue to pass.
2026-05-15 09:58:56 -03:00
void FetchedDataReceiver::set_cached_response_body(Core::ImmutableBytes body)
{
if (!m_http_cache)
return;
m_cache_buffer.clear();
m_cache_body = move(body);
m_cache_body_replaces_network_buffer = true;
}
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
// This implements the parallel steps of the pullAlgorithm in HTTP-network-fetch.
// https://fetch.spec.whatwg.org/#ref-for-in-parallel④
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
void FetchedDataReceiver::enqueue_into_stream(ReadonlyBytes bytes)
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
{
// FIXME: 1. If the size of buffer is smaller than a lower limit chosen by the user agent and the ongoing fetch
// is suspended, resume the fetch.
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
if (!m_stream->is_readable())
return;
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
auto& realm = m_stream->realm();
HTML::TemporaryExecutionContext execution_context { realm, HTML::TemporaryExecutionContext::CallbacksEnabled::Yes };
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
// 1. Pull from bytes buffer into stream.
auto byte_buffer = MUST(ByteBuffer::copy(bytes));
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
auto& controller = m_stream->controller()->get<GC::Ref<Streams::ReadableByteStreamController>>();
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
LibWeb: Avoid temporary stream chunks for fetch bytes Teach readable byte stream enqueueing about native byte producers by sharing the post-transfer part of the spec algorithm. This lets fetch hand owned network bytes directly to the byte stream controller instead of first wrapping them in an unobservable Uint8Array and then transferring its buffer back into another ArrayBuffer. Keep the public enqueue(chunk) path spec-shaped and make the native path rejoin after the TransferArrayBuffer step, so observable chunks are still created by the shared reader and queue handling code. This reduces GC churn while playing YouTube videos.
2026-05-15 07:23:23 -03:00
if (auto result = Streams::readable_byte_stream_controller_enqueue_native_bytes(*controller, move(byte_buffer)); result.is_error()) {
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
auto throw_completion = Bindings::exception_to_throw_completion(realm.vm(), result.release_error());
// 2. If stream is errored, then terminate fetchParams’s controller.
Streams::readable_byte_stream_controller_error(*controller, throw_completion.value());
m_fetch_params->controller()->terminate();
}
LibWeb: Use unbuffered network requests for all Fetch requests Previously, unbuffered requests were only available as a special mode for EventSource. With this change, they are enabled by default, which means chunks can be read from the stream as soon as they arrive. This unlocks some interesting possibilities, such as starting to parse HTML documents before the entire response has been received (that, in turn, allows us to initiate subresource fetches earlier or begin executing scripts sooner), or start rendering videos before they are fully downloaded. Co-authored-by: Timothy Flynn <trflynn89@pm.me>
2025-04-17 08:58:24 -03:00
}
void FetchedDataReceiver::close_stream()
{
LibHTTP+LibWeb: Store the in-memory HTTP cache without JS realms The in-memory HTTP Fetch cache currently keeps the realm which created each cache entry alive indefinitely. This patch migrates this cache to LibHTTP, to ensure it is completely unaware of any JS objects. Now that we are not interacting with Fetch response objects, we can no longer use Streams infrastructure to pipe the response body into the Fetch response. Fetch also ultimately creates the cache response once the HTTP response headers have arrived. So the LibHTTP cache will hold entries in a pending list until we have received the entire response body. Then it is moved to a completed list and may be used thereafter.
2025-12-21 10:58:55 -03:00
if (m_http_cache) {
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 12:23:56 -03:00
auto request = m_fetch_params->request();
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
if (m_stream->is_readable() && !m_fetch_params->is_canceled()
&& m_response && request->cache_mode() != HTTP::CacheMode::NoStore) {
LibWeb: Preserve file-backed HTTP cache bodies Carry response chunks through LibRequests and ResourceLoader as a ResponseData wrapper. A disk cache hit can retain its mapped storage, while ordinary streamed chunks still use borrowed bytes. Store completed memory cache entries as Core::ImmutableBytes. This lets mapped disk-cache responses stay file-backed while retained by the HTTP memory cache, instead of forcing another ByteBuffer copy.
2026-05-15 09:49:12 -03:00
auto response_body = m_cache_body.has_value()
? m_cache_body.release_value()
: Core::ImmutableBytes::adopt(move(m_cache_buffer));
m_http_cache->finalize_entry(request->current_url(), request->method(), request->header_list(), m_response->status(), m_response->header_list(), move(response_body));
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
}
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 12:23:56 -03:00
LibHTTP+LibWeb: Store the in-memory HTTP cache without JS realms The in-memory HTTP Fetch cache currently keeps the realm which created each cache entry alive indefinitely. This patch migrates this cache to LibHTTP, to ensure it is completely unaware of any JS objects. Now that we are not interacting with Fetch response objects, we can no longer use Streams infrastructure to pipe the response body into the Fetch response. Fetch also ultimately creates the cache response once the HTTP response headers have arrived. So the LibHTTP cache will hold entries in a pending list until we have received the entire response body. Then it is moved to a completed list and may be used thereafter.
2025-12-21 10:58:55 -03:00
m_http_cache.clear();
LibWeb: Preserve file-backed HTTP cache bodies Carry response chunks through LibRequests and ResourceLoader as a ResponseData wrapper. A disk cache hit can retain its mapped storage, while ordinary streamed chunks still use borrowed bytes. Store completed memory cache entries as Core::ImmutableBytes. This lets mapped disk-cache responses stay file-backed while retained by the HTTP memory cache, instead of forcing another ByteBuffer copy.
2026-05-15 09:49:12 -03:00
m_cache_body.clear();
LibHTTP+LibWeb: Store the in-memory HTTP cache without JS realms The in-memory HTTP Fetch cache currently keeps the realm which created each cache entry alive indefinitely. This patch migrates this cache to LibHTTP, to ensure it is completely unaware of any JS objects. Now that we are not interacting with Fetch response objects, we can no longer use Streams infrastructure to pipe the response body into the Fetch response. Fetch also ultimately creates the cache response once the HTTP response headers have arrived. So the LibHTTP cache will hold entries in a pending list until we have received the entire response body. Then it is moved to a completed list and may be used thereafter.
2025-12-21 10:58:55 -03:00
}
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
if (!m_stream->is_readable())
return;
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 10:33:49 -03:00
LibWeb: Push fetch body chunks directly into the byte stream controller Previously, each chunk was delivered through a pull promise. Every pull allocated around seven GC objects (the pull promise, its capability, and the reaction handlers from react_to_promise) and none of them did anything useful. The pull algorithm is now a no-op, and FetchedDataReceiver enqueues bytes into the controller as they arrive.
2026-04-29 15:08:01 -03:00
HTML::TemporaryExecutionContext execution_context { m_stream->realm(), HTML::TemporaryExecutionContext::CallbacksEnabled::Yes };
m_stream->close();
LibWeb: Accumulate all network bytes in FetchedDataReceiver This will allow us to hand off the bytes to the HTTP memory cache.
2025-12-20 10:33:49 -03:00
}
LibWeb: Support unbuffered fetch requests Supporting unbuffered fetches is actually part of the fetch spec in its HTTP-network-fetch algorithm. We had previously implemented this method in a very ad-hoc manner as a simple wrapper around ResourceLoader. This is still the case, but we now implement a good amount of these steps according to spec, using ResourceLoader's unbuffered API. The response data is forwarded through to the fetch response using streams. This will eventually let us remove the use of ResourceLoader's buffered API, as all responses should just be streamed this way. The streams spec then supplies ways to wait for completion, thus allowing fully buffered responses. However, we have more work to do to make the other parts of our fetch implementation (namely, Body::fully_read) use streams before we can do this.
2024-05-26 09:03:29 -03:00
}
Reference in a new issue Copy permalink