dmaax/ladybird
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ladybird/Services/RequestServer/Request.cpp

1384 lines
55 KiB
C++
Raw Normal View History

RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
/*
* Copyright (c) 2024, Andreas Kling <andreas@ladybird.org>
LibHTTP+LibWeb+RequestServer: Handle the Fetch API's cache mode If the cache mode is no-store, we must not interact with the cache at all. If the cache mode is reload, we must not use any cached response. If the cache-mode is only-if-cached or force-cache, we are permitted to respond with stale cache responses. Note that we currently cannot test only-if-cached in test-web. Setting this mode also requires setting the cors mode to same-origin, but our http-test-server infra requires setting the cors mode to cors.
2026-01-20 19:12:36 -03:00
* Copyright (c) 2025-2026, Tim Flynn <trflynn89@ladybird.org>
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
*
* SPDX-License-Identifier: BSD-2-Clause
*/
RequestServer: Support HTTP response write retries on Windows The Windows RequestPipe implementation uses a non blocking local socket pair, which means the non-fatal "resource is temporarily unavailable" error that can occur in the non-blocking HTTP Response data writes can be retried. This was seen often when loading https://ladybird.org. While the EAGAIN errno is defined on Windows, WSAEWOULDBLOCK is the error code returned in this scenario, so we were not detecting that we could retry and treated the failed write attempt as a proper error. We now detect WSAEWOULDBLOCK and convert it into the errno equivalent EWOULDBLOCK. There is precedent for doing a similar conversion in the Windows PosixSocketHelper::read() implementation. Finally, we retry when we receive either EAGAIN or EWOULDBLOCK error codes on all platforms. While POSIX allows these 2 error codes to have the same value, which they do on Linux according to https://www.man7.org/linux/man-pages/man3/errno.3.html, it is not guarenteed. So we now ensure platforms that return EWOULDBLOCK with a value different than EAGAIN also perform write retries.
2025-11-18 19:17:53 -03:00
#include <AK/GenericShorthands.h>
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
#include <AK/HashMap.h>
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
#include <LibCore/File.h>
#include <LibCore/MimeData.h>
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
#include <LibCore/Notifier.h>
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
#include <LibHTTP/Cache/DiskCache.h>
#include <LibHTTP/Cache/Utilities.h>
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
#include <LibHTTP/Status.h>
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
#include <LibTextCodec/Decoder.h>
#include <RequestServer/CURL.h>
#include <RequestServer/ConnectionFromClient.h>
#include <RequestServer/Request.h>
#include <RequestServer/Resolver.h>
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
#include <RequestServer/ResourceSubstitutionMap.h>
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
namespace RequestServer {
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
extern OwnPtr<ResourceSubstitutionMap> g_resource_substitution_map;
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
static long s_connect_timeout_seconds = 90L;
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
static void log_network_activity(URL::URL const& url, ByteString const& method, void* curl_easy_handle, int curl_result_code, bool is_revalidation, RequestType type)
{
if constexpr (!REQUESTSERVER_WIRE_DEBUG)
return;
if (!curl_easy_handle)
return;
auto get_off = [&](auto option) {
curl_off_t value = 0;
(void)curl_easy_getinfo(curl_easy_handle, option, &value);
return value;
};
auto get_long = [&](auto option) {
long value = 0;
(void)curl_easy_getinfo(curl_easy_handle, option, &value);
return value;
};
long http_status = get_long(CURLINFO_RESPONSE_CODE);
long http_version = get_long(CURLINFO_HTTP_VERSION);
auto queue_us = get_off(CURLINFO_QUEUE_TIME_T);
auto namelookup_us = get_off(CURLINFO_NAMELOOKUP_TIME_T);
auto connect_us = get_off(CURLINFO_CONNECT_TIME_T);
auto appconnect_us = get_off(CURLINFO_APPCONNECT_TIME_T);
auto pretransfer_us = get_off(CURLINFO_PRETRANSFER_TIME_T);
auto starttransfer_us = get_off(CURLINFO_STARTTRANSFER_TIME_T);
auto total_us = get_off(CURLINFO_TOTAL_TIME_T);
auto bytes_downloaded = get_off(CURLINFO_SIZE_DOWNLOAD_T);
auto bytes_uploaded = get_off(CURLINFO_SIZE_UPLOAD_T);
auto download_speed_bps = get_off(CURLINFO_SPEED_DOWNLOAD_T);
// libcurl phase timings are cumulative from t=0, but skipped phases (e.g. DNS/TCP/TLS on a
// reused connection) are reported as 0, breaking the monotonic ordering. Clamp each marker
// to the previous one so skipped phases yield a 0-length delta instead of double-counting.
auto clamp = [](curl_off_t marker, curl_off_t previous) { return marker > previous ? marker : previous; };
auto m_queue = queue_us;
auto m_namelookup = clamp(namelookup_us, m_queue);
auto m_connect = clamp(connect_us, m_namelookup);
auto m_appconnect = clamp(appconnect_us, m_connect);
auto m_pretransfer = clamp(pretransfer_us, m_appconnect);
auto m_starttransfer = clamp(starttransfer_us, m_pretransfer);
auto m_total = clamp(total_us, m_starttransfer);
auto us_to_ms = [](curl_off_t us) { return static_cast<double>(us) / 1000.0; };
auto queue_ms = us_to_ms(m_queue);
auto dns_ms = us_to_ms(m_namelookup - m_queue);
auto tcp_ms = us_to_ms(m_connect - m_namelookup);
auto tls_ms = us_to_ms(m_appconnect - m_connect);
auto request_ms = us_to_ms(m_pretransfer - m_appconnect);
auto wait_ms = us_to_ms(m_starttransfer - m_pretransfer);
auto body_ms = us_to_ms(m_total - m_starttransfer);
auto total_ms = us_to_ms(m_total);
auto kib = [](curl_off_t bytes) { return static_cast<double>(bytes) / 1024.0; };
StringView http_version_str = "HTTP/?"sv;
switch (http_version) {
case CURL_HTTP_VERSION_1_0:
http_version_str = "HTTP/1.0"sv;
break;
case CURL_HTTP_VERSION_1_1:
http_version_str = "HTTP/1.1"sv;
break;
case CURL_HTTP_VERSION_2_0:
http_version_str = "HTTP/2"sv;
break;
case CURL_HTTP_VERSION_3:
http_version_str = "HTTP/3"sv;
break;
default:
break;
}
StringView kind;
if (curl_result_code != CURLE_OK)
kind = "FAIL"sv;
else if (is_revalidation && http_status == 304)
kind = "REVAL-304"sv;
else if (is_revalidation)
kind = "REVAL-FULL"sv;
else
kind = "DOWNLOAD"sv;
StringView background = type == RequestType::BackgroundRevalidation ? " [bg]"sv : ""sv;
if (curl_result_code != CURLE_OK) {
char const* err = curl_easy_strerror(static_cast<CURLcode>(curl_result_code));
dbgln_if(REQUESTSERVER_WIRE_DEBUG, "RequestServer wire: {}{} {} {} -> error: {} (after {:.1} ms, wire {:.1} KiB)",
kind, background, method, url, err, total_ms, kib(bytes_downloaded));
return;
}
auto wire_kibps_during_body = body_ms > 0.0
? kib(bytes_downloaded) / (body_ms / 1000.0)
: 0.0;
dbgln_if(REQUESTSERVER_WIRE_DEBUG, "RequestServer wire: {}{} {} {} {} -> {} | wire {:.1} KiB sent {:.1} KiB | total {:.1} ms = queue {:.1} + dns {:.1} + tcp {:.1} + tls {:.1} + req {:.1} + wait {:.1} + body {:.1} | wire {:.1} KiB/s avg, {:.1} KiB/s during body",
kind, background, method, url, http_version_str, http_status,
kib(bytes_downloaded), kib(bytes_uploaded),
total_ms, queue_ms, dns_ms, tcp_ms, tls_ms, request_ms, wait_ms, body_ms,
kib(download_speed_bps), wire_kibps_during_body);
}
struct WireStats {
// Decoded-side (sampled in on_data_received, after curl decompresses)
Optional<MonotonicTime> first_chunk_at;
Optional<MonotonicTime> last_chunk_at;
u64 chunk_count { 0 };
u64 total_decoded_bytes { 0 };
u64 min_chunk_bytes { NumericLimits<u64>::max() };
u64 max_chunk_bytes { 0 };
AK::Duration sum_inter_chunk_gaps;
AK::Duration max_inter_chunk_gap;
u64 stall_count_100ms { 0 };
AK::Duration max_stall_started_at;
u64 bytes_at_max_stall { 0 };
// Wire-side (sampled in CURLOPT_XFERINFOFUNCTION, before decompression)
Optional<MonotonicTime> first_wire_byte_at;
Optional<MonotonicTime> last_wire_byte_at;
curl_off_t last_wire_dlnow { 0 };
AK::Duration max_wire_gap;
AK::Duration max_wire_stall_started_at;
curl_off_t wire_bytes_at_max_stall { 0 };
u64 wire_stall_count_100ms { 0 };
// Internal-pipeline lifecycle (set by state-machine handlers).
// Lets us see how much wall time we burned in our own code paths
// (cache lookup, our DNS resolver, cookie IPC, curl setup) before
// libcurl ever saw the request.
Optional<MonotonicTime> created_at;
Optional<MonotonicTime> dns_started_at;
Optional<MonotonicTime> dns_completed_at;
Optional<MonotonicTime> cookie_started_at;
Optional<MonotonicTime> cookie_completed_at;
Optional<MonotonicTime> curl_added_at;
Optional<MonotonicTime> complete_observed_at;
// WebContent-side back-pressure on our outgoing pipe. Updated by
// write_queued_bytes_without_blocking when the pipe returns EAGAIN
// (WebContent isn't draining fast enough — typically because its main
// thread is busy parsing, running JS, etc.). `current_window_started`
// is set on entry to a back-pressure window and cleared when we resume
// writing successfully.
Optional<MonotonicTime> current_pressure_window_started;
AK::Duration total_pipe_back_pressure;
u64 pipe_back_pressure_events { 0 };
u64 max_buffered_bytes { 0 };
};
static HashMap<Request const*, WireStats>& wire_stats()
{
static HashMap<Request const*, WireStats> map;
return map;
}
static void record_chunk(Request const* request, size_t bytes)
{
if constexpr (!REQUESTSERVER_WIRE_DEBUG)
return;
auto now = MonotonicTime::now();
auto& stats = wire_stats().ensure(request);
if (stats.chunk_count == 0) {
stats.first_chunk_at = now;
} else {
auto gap = now - *stats.last_chunk_at;
stats.sum_inter_chunk_gaps = stats.sum_inter_chunk_gaps + gap;
if (gap > stats.max_inter_chunk_gap) {
stats.max_inter_chunk_gap = gap;
stats.max_stall_started_at = *stats.last_chunk_at - *stats.first_chunk_at;
stats.bytes_at_max_stall = stats.total_decoded_bytes;
}
if (gap > AK::Duration::from_milliseconds(100))
stats.stall_count_100ms += 1;
}
stats.last_chunk_at = now;
stats.chunk_count += 1;
stats.total_decoded_bytes += bytes;
if (bytes < stats.min_chunk_bytes)
stats.min_chunk_bytes = bytes;
if (bytes > stats.max_chunk_bytes)
stats.max_chunk_bytes = bytes;
}
[[maybe_unused]] static int on_xferinfo(void* user_data, curl_off_t /*dltotal*/, curl_off_t dlnow, curl_off_t /*ultotal*/, curl_off_t /*ulnow*/)
{
if constexpr (!REQUESTSERVER_WIRE_DEBUG)
return 0;
auto* request = static_cast<Request const*>(user_data);
if (dlnow <= 0)
return 0;
auto& stats = wire_stats().ensure(request);
if (dlnow == stats.last_wire_dlnow)
return 0;
auto now = MonotonicTime::now();
if (!stats.first_wire_byte_at.has_value())
stats.first_wire_byte_at = now;
if (stats.last_wire_byte_at.has_value()) {
auto gap = now - *stats.last_wire_byte_at;
if (gap > stats.max_wire_gap) {
stats.max_wire_gap = gap;
stats.max_wire_stall_started_at = *stats.last_wire_byte_at - *stats.first_wire_byte_at;
stats.wire_bytes_at_max_stall = stats.last_wire_dlnow;
}
if (gap > AK::Duration::from_milliseconds(100))
stats.wire_stall_count_100ms += 1;
}
stats.last_wire_byte_at = now;
stats.last_wire_dlnow = dlnow;
return 0;
}
static void log_chunk_stats(Request const* request)
{
if constexpr (!REQUESTSERVER_WIRE_DEBUG)
return;
auto it = wire_stats().find(request);
if (it == wire_stats().end())
return;
auto const& s = it->value;
if (s.chunk_count == 0)
return;
auto kib = [](u64 b) { return static_cast<double>(b) / 1024.0; };
auto kib_off = [](curl_off_t b) { return static_cast<double>(b) / 1024.0; };
auto decoded_span = (s.last_chunk_at.has_value() && s.first_chunk_at.has_value())
? *s.last_chunk_at - *s.first_chunk_at
: AK::Duration {};
auto decoded_span_ms = decoded_span.to_milliseconds();
auto avg_gap_ms = s.chunk_count > 1
? static_cast<double>(s.sum_inter_chunk_gaps.to_microseconds()) / 1000.0 / static_cast<double>(s.chunk_count - 1)
: 0.0;
auto avg_chunk = s.total_decoded_bytes / s.chunk_count;
auto decoded_kibps = decoded_span_ms > 0
? kib(s.total_decoded_bytes) / (static_cast<double>(decoded_span_ms) / 1000.0)
: 0.0;
dbgln_if(REQUESTSERVER_WIRE_DEBUG, "RequestServer wire+: decoded chunks={} bytes={:.1} KiB span={} ms thru={:.1} KiB/s | gap avg={:.1} ms max={} ms stalls(>100ms)={} | worst gap began at +{} ms after {:.1} KiB decoded | chunk bytes avg={} min={} max={}",
s.chunk_count, kib(s.total_decoded_bytes), decoded_span_ms, decoded_kibps,
avg_gap_ms, s.max_inter_chunk_gap.to_milliseconds(), s.stall_count_100ms,
s.max_stall_started_at.to_milliseconds(), kib(s.bytes_at_max_stall),
avg_chunk, s.min_chunk_bytes, s.max_chunk_bytes);
if (s.first_wire_byte_at.has_value() && s.last_wire_byte_at.has_value()) {
auto wire_span = *s.last_wire_byte_at - *s.first_wire_byte_at;
auto wire_span_ms = wire_span.to_milliseconds();
auto wire_kibps = wire_span_ms > 0
? kib_off(s.last_wire_dlnow) / (static_cast<double>(wire_span_ms) / 1000.0)
: 0.0;
auto compression_ratio = s.last_wire_dlnow > 0
? static_cast<double>(s.total_decoded_bytes) / static_cast<double>(s.last_wire_dlnow)
: 0.0;
dbgln_if(REQUESTSERVER_WIRE_DEBUG, "RequestServer wire++: wire bytes={:.1} KiB span={} ms thru={:.1} KiB/s | wire max gap={} ms stalls(>100ms)={} | worst wire gap began at +{} ms after {:.1} KiB on the wire | compression={:.2}x",
kib_off(s.last_wire_dlnow), wire_span_ms, wire_kibps,
s.max_wire_gap.to_milliseconds(), s.wire_stall_count_100ms,
s.max_wire_stall_started_at.to_milliseconds(), kib_off(s.wire_bytes_at_max_stall),
compression_ratio);
}
// wire^: pre-network time spent inside RequestServer (cache + our DNS resolver + cookie IPC
// + curl setup), and the gap between the wire going quiet and us actually emitting this log
// entry. Useful to see if a long "total" was actually our pipeline rather than the network.
if (s.created_at.has_value()) {
auto delta_ms = [](Optional<MonotonicTime> const& a, Optional<MonotonicTime> const& b) -> i64 {
if (!a.has_value() || !b.has_value())
return -1;
return (*b - *a).to_milliseconds();
};
// Cache + Init + WaitForCache: from creation to start of DNS lookup.
auto pre_dns_ms = delta_ms(s.created_at, s.dns_started_at);
// Our DNS resolver (note: distinct from libcurl's `dns` field, which is 0 because we
// pre-resolve and pass via CURLOPT_RESOLVE).
auto our_dns_ms = delta_ms(s.dns_started_at, s.dns_completed_at);
// Cookie IPC round-trip to the UI process.
auto cookie_ms = delta_ms(s.cookie_started_at, s.cookie_completed_at);
// Time from the last completed pre-network step to curl_multi_add_handle.
auto curl_setup_ms = [&]() -> i64 {
Optional<MonotonicTime> last_pre_curl;
if (s.cookie_completed_at.has_value())
last_pre_curl = s.cookie_completed_at;
else if (s.dns_completed_at.has_value())
last_pre_curl = s.dns_completed_at;
else
last_pre_curl = s.created_at;
return delta_ms(last_pre_curl, s.curl_added_at);
}();
auto pre_curl_total_ms = delta_ms(s.created_at, s.curl_added_at);
// Drain delay: time between the last byte arriving and check_active_requests draining
// the completion. Non-zero would mean we noticed completion later than curl did.
Optional<MonotonicTime> last_activity;
if (s.last_wire_byte_at.has_value())
last_activity = s.last_wire_byte_at;
else if (s.last_chunk_at.has_value())
last_activity = s.last_chunk_at;
auto drain_delay_ms = delta_ms(last_activity, s.complete_observed_at);
auto fmt_ms = [](i64 ms) -> ByteString {
return ms < 0 ? ByteString { "-" } : ByteString::formatted("{}", ms);
};
ByteString back_pressure_summary;
if (s.pipe_back_pressure_events > 0) {
back_pressure_summary = ByteString::formatted(
" | pipe back-pressure events={} total={} ms peak-buffered={} bytes",
s.pipe_back_pressure_events,
s.total_pipe_back_pressure.to_milliseconds(),
s.max_buffered_bytes);
}
dbgln_if(REQUESTSERVER_WIRE_DEBUG, "RequestServer wire^: internal pre-curl={} ms = cache+init {} + our-dns {} + cookie {} + curl-setup {} | drain delay {} ms{}",
fmt_ms(pre_curl_total_ms), fmt_ms(pre_dns_ms), fmt_ms(our_dns_ms),
fmt_ms(cookie_ms), fmt_ms(curl_setup_ms), fmt_ms(drain_delay_ms),
back_pressure_summary);
}
}
static void mark_lifecycle_event(Request const* request, Optional<MonotonicTime> WireStats::* field)
{
if constexpr (!REQUESTSERVER_WIRE_DEBUG)
return;
wire_stats().ensure(request).*field = MonotonicTime::now();
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
NonnullOwnPtr<Request> Request::fetch(
LibRequests+RequestServer: Store request IDs as u64 Not super important, but this will match an upcoming ID for stale-while- revalidate requests. It also would techinically be UB if this ID had ever overflowed. Let's make it 64-bit while we are here to also avoid the possibility that it ever will overflow.
2025-12-12 11:56:53 -03:00
u64 request_id,
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
Optional<HTTP::DiskCache&> disk_cache,
LibHTTP+LibWeb+RequestServer: Handle the Fetch API's cache mode If the cache mode is no-store, we must not interact with the cache at all. If the cache mode is reload, we must not use any cached response. If the cache-mode is only-if-cached or force-cache, we are permitted to respond with stale cache responses. Note that we currently cannot test only-if-cached in test-web. Setting this mode also requires setting the cors mode to same-origin, but our http-test-server infra requires setting the cors mode to cors.
2026-01-20 19:12:36 -03:00
HTTP::CacheMode cache_mode,
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
ConnectionFromClient& client,
void* curl_multi,
Resolver& resolver,
URL::URL url,
ByteString method,
LibHTTP+LibWeb+RequestServer: Move Fetch's HTTP header infra to LibHTTP The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP caching) implementation. We currently have one implementation in LibWeb for our in-memory cache and another in RequestServer for our disk cache. The implementations both largely revolve around interacting with HTTP headers. But in LibWeb, we are using Fetch's header infra, and in RS we are using are home-grown header infra from LibHTTP. So to give these a common denominator, this patch replaces the LibHTTP implementation with Fetch's infra. Our existing LibHTTP implementation was not particularly compliant with any spec, so this at least gives us a standards-based common implementation. This migration also required moving a handful of other Fetch AOs over to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP folder, so perhaps it makes sense for LibHTTP to be the implementation of that entire set of facilities.)
2025-11-26 16:13:23 -03:00
NonnullRefPtr<HTTP::HeaderList> request_headers,
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
ByteBuffer request_body,
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
HTTP::Cookie::IncludeCredentials include_credentials,
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
ByteString alt_svc_cache_path,
Core::ProxyData proxy_data)
{
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
auto request = adopt_own(*new Request { request_id, RequestType::Fetch, disk_cache, cache_mode, client, curl_multi, resolver, move(url), move(method), move(request_headers), move(request_body), include_credentials, move(alt_svc_cache_path), proxy_data });
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
request->process();
return request;
}
NonnullOwnPtr<Request> Request::connect(
LibRequests+RequestServer: Store request IDs as u64 Not super important, but this will match an upcoming ID for stale-while- revalidate requests. It also would techinically be UB if this ID had ever overflowed. Let's make it 64-bit while we are here to also avoid the possibility that it ever will overflow.
2025-12-12 11:56:53 -03:00
u64 request_id,
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
ConnectionFromClient& client,
void* curl_multi,
Resolver& resolver,
URL::URL url,
CacheLevel cache_level)
{
auto request = adopt_own(*new Request { request_id, client, curl_multi, resolver, move(url) });
RequestServer: Pre-resolve preconnect handles via our DNS resolver handle_connect_state (used by <link rel=preconnect>) attached easy handles to the multi without setting CURLOPT_RESOLVE, so libcurl spawned its own threaded resolver for them. When the system stub resolver was slow, the thread got stuck and the next ~Request pthread_join'd it on the main thread for many seconds. Route Connect-mode requests through the same DNS + CURLOPT_RESOLVE path that Fetch uses, so libcurl never spawns the thread.
2026-04-26 06:14:46 -03:00
request->m_connect_cache_level = cache_level;
request->transition_to_state(State::DNSLookup);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
return request;
}
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
NonnullOwnPtr<Request> Request::revalidate(
u64 request_id,
Optional<HTTP::DiskCache&> disk_cache,
ConnectionFromClient& client,
void* curl_multi,
Resolver& resolver,
URL::URL url,
ByteString method,
NonnullRefPtr<HTTP::HeaderList> request_headers,
ByteBuffer request_body,
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
HTTP::Cookie::IncludeCredentials include_credentials,
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
ByteString alt_svc_cache_path,
Core::ProxyData proxy_data)
{
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
auto request = adopt_own(*new Request { request_id, RequestType::BackgroundRevalidation, disk_cache, HTTP::CacheMode::Default, client, curl_multi, resolver, move(url), move(method), move(request_headers), move(request_body), include_credentials, move(alt_svc_cache_path), proxy_data });
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
request->process();
return request;
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
Request::Request(
LibRequests+RequestServer: Store request IDs as u64 Not super important, but this will match an upcoming ID for stale-while- revalidate requests. It also would techinically be UB if this ID had ever overflowed. Let's make it 64-bit while we are here to also avoid the possibility that it ever will overflow.
2025-12-12 11:56:53 -03:00
u64 request_id,
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
RequestType type,
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
Optional<HTTP::DiskCache&> disk_cache,
LibHTTP+LibWeb+RequestServer: Handle the Fetch API's cache mode If the cache mode is no-store, we must not interact with the cache at all. If the cache mode is reload, we must not use any cached response. If the cache-mode is only-if-cached or force-cache, we are permitted to respond with stale cache responses. Note that we currently cannot test only-if-cached in test-web. Setting this mode also requires setting the cors mode to same-origin, but our http-test-server infra requires setting the cors mode to cors.
2026-01-20 19:12:36 -03:00
HTTP::CacheMode cache_mode,
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
ConnectionFromClient& client,
void* curl_multi,
Resolver& resolver,
URL::URL url,
ByteString method,
LibHTTP+LibWeb+RequestServer: Move Fetch's HTTP header infra to LibHTTP The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP caching) implementation. We currently have one implementation in LibWeb for our in-memory cache and another in RequestServer for our disk cache. The implementations both largely revolve around interacting with HTTP headers. But in LibWeb, we are using Fetch's header infra, and in RS we are using are home-grown header infra from LibHTTP. So to give these a common denominator, this patch replaces the LibHTTP implementation with Fetch's infra. Our existing LibHTTP implementation was not particularly compliant with any spec, so this at least gives us a standards-based common implementation. This migration also required moving a handful of other Fetch AOs over to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP folder, so perhaps it makes sense for LibHTTP to be the implementation of that entire set of facilities.)
2025-11-26 16:13:23 -03:00
NonnullRefPtr<HTTP::HeaderList> request_headers,
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
ByteBuffer request_body,
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
HTTP::Cookie::IncludeCredentials include_credentials,
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
ByteString alt_svc_cache_path,
Core::ProxyData proxy_data)
: m_request_id(request_id)
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
, m_type(type)
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
, m_disk_cache(disk_cache)
LibHTTP+LibWeb+RequestServer: Handle the Fetch API's cache mode If the cache mode is no-store, we must not interact with the cache at all. If the cache mode is reload, we must not use any cached response. If the cache-mode is only-if-cached or force-cache, we are permitted to respond with stale cache responses. Note that we currently cannot test only-if-cached in test-web. Setting this mode also requires setting the cors mode to same-origin, but our http-test-server infra requires setting the cors mode to cors.
2026-01-20 19:12:36 -03:00
, m_cache_mode(cache_mode)
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
, m_client(client)
, m_curl_multi_handle(curl_multi)
, m_resolver(resolver)
, m_url(move(url))
, m_method(move(method))
, m_request_headers(move(request_headers))
, m_request_body(move(request_body))
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
, m_include_credentials(include_credentials)
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
, m_alt_svc_cache_path(move(alt_svc_cache_path))
, m_proxy_data(proxy_data)
LibHTTP+LibWeb+RequestServer: Move Fetch's HTTP header infra to LibHTTP The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP caching) implementation. We currently have one implementation in LibWeb for our in-memory cache and another in RequestServer for our disk cache. The implementations both largely revolve around interacting with HTTP headers. But in LibWeb, we are using Fetch's header infra, and in RS we are using are home-grown header infra from LibHTTP. So to give these a common denominator, this patch replaces the LibHTTP implementation with Fetch's infra. Our existing LibHTTP implementation was not particularly compliant with any spec, so this at least gives us a standards-based common implementation. This migration also required moving a handful of other Fetch AOs over to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP folder, so perhaps it makes sense for LibHTTP to be the implementation of that entire set of facilities.)
2025-11-26 16:13:23 -03:00
, m_response_headers(HTTP::HeaderList::create())
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
{
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
if constexpr (REQUESTSERVER_WIRE_DEBUG)
wire_stats().ensure(this).created_at = MonotonicTime::now();
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
Request::Request(
LibRequests+RequestServer: Store request IDs as u64 Not super important, but this will match an upcoming ID for stale-while- revalidate requests. It also would techinically be UB if this ID had ever overflowed. Let's make it 64-bit while we are here to also avoid the possibility that it ever will overflow.
2025-12-12 11:56:53 -03:00
u64 request_id,
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
ConnectionFromClient& client,
void* curl_multi,
Resolver& resolver,
URL::URL url)
: m_request_id(request_id)
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
, m_type(RequestType::Connect)
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
, m_client(client)
, m_curl_multi_handle(curl_multi)
, m_resolver(resolver)
, m_url(move(url))
LibHTTP+LibWeb+RequestServer: Move Fetch's HTTP header infra to LibHTTP The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP caching) implementation. We currently have one implementation in LibWeb for our in-memory cache and another in RequestServer for our disk cache. The implementations both largely revolve around interacting with HTTP headers. But in LibWeb, we are using Fetch's header infra, and in RS we are using are home-grown header infra from LibHTTP. So to give these a common denominator, this patch replaces the LibHTTP implementation with Fetch's infra. Our existing LibHTTP implementation was not particularly compliant with any spec, so this at least gives us a standards-based common implementation. This migration also required moving a handful of other Fetch AOs over to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP folder, so perhaps it makes sense for LibHTTP to be the implementation of that entire set of facilities.)
2025-11-26 16:13:23 -03:00
, m_request_headers(HTTP::HeaderList::create())
, m_response_headers(HTTP::HeaderList::create())
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
{
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
if constexpr (REQUESTSERVER_WIRE_DEBUG)
wire_stats().ensure(this).created_at = MonotonicTime::now();
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
Request::~Request()
{
if (!m_response_buffer.is_eof())
dbgln("Warning: Request destroyed with buffered data (it's likely that the client disappeared or the request was cancelled)");
if (m_curl_easy_handle) {
auto result = curl_multi_remove_handle(m_curl_multi_handle, m_curl_easy_handle);
VERIFY(result == CURLM_OK);
curl_easy_cleanup(m_curl_easy_handle);
}
for (auto* string_list : m_curl_string_lists)
curl_slist_free_all(string_list);
RequestServer: Do not flush the disk cache for unsuccessful requests If a request failed, or was stopped, do not attempt to write the cache entry footer to disk. Note that at this point, the cache index will not have been created, thus this entry will not be used in the future. We do still delete any partial file on disk. This serves as a more general fix for the issue addressed in commit 9f2ac14521d54c48b7301b24a3549a0d058a1d6a.
2026-01-23 09:26:37 -03:00
if (m_cache_entry_writer.has_value()) {
if (m_state == State::Complete)
(void)m_cache_entry_writer->flush(m_request_headers, m_response_headers);
else
m_cache_entry_writer->remove_incomplete_entry();
}
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
if constexpr (REQUESTSERVER_WIRE_DEBUG)
wire_stats().remove(this);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
void Request::notify_request_unblocked(Badge<HTTP::DiskCache>)
RequestServer: De-duplicate some disk cache requests We previously had no protection against the same URL being requested multiple times at the same time. For example, if a URL did not have any cache entry and became requested twice, we would open two cache writers concurrently. This would result in both writers piping the response to disk, and we'd have a corrupt cache file. We now hold back requests under certain scenarios until existing cache entries have completed: * If we are opening a cache entry for reading: - If there is an existing reader entry, carry on as normal. We can have multiple readers. - If there is an existing writer entry, defer the request until it is complete. * If we are opening a cache entry for writing: - If there is an existing reader or writer entry, defer the request until it is complete.
2025-10-24 13:54:32 -03:00
{
// FIXME: We may want a timer to limit how long we are waiting for a request before proceeding with a network
// request that skips the disk cache.
transition_to_state(State::Init);
}
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
void Request::notify_retrieved_http_cookie(Badge<ConnectionFromClient>, StringView cookie)
{
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
mark_lifecycle_event(this, &WireStats::cookie_completed_at);
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
if (!cookie.is_empty()) {
auto header = HTTP::Header::isomorphic_encode("Cookie"sv, cookie);
m_request_headers->append(move(header));
}
transition_to_state(State::Fetch);
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
void Request::notify_fetch_complete(Badge<ConnectionFromClient>, int result_code)
{
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
mark_lifecycle_event(this, &WireStats::complete_observed_at);
if (m_type == RequestType::Fetch || m_type == RequestType::BackgroundRevalidation) {
log_network_activity(m_url, m_method, m_curl_easy_handle, result_code, is_revalidation_request(), m_type);
log_chunk_stats(this);
}
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
if (is_revalidation_request()) {
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
if (acquire_status_code() == 304) {
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
if (m_type == RequestType::BackgroundRevalidation && m_disk_cache->mode() == HTTP::DiskCache::Mode::Testing)
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
m_response_headers->set({ HTTP::TEST_CACHE_REVALIDATION_STATUS_HEADER, "fresh"sv });
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
m_cache_entry_reader->revalidation_succeeded(m_response_headers);
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
transition_to_state(m_type == RequestType::Fetch ? State::ReadCache : State::Complete);
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
return;
}
if (revalidation_failed().is_error())
return;
transfer_headers_to_client_if_needed();
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
m_curl_result_code = result_code;
if (m_response_buffer.is_eof())
transition_to_state(State::Complete);
}
void Request::transition_to_state(State state)
{
RequestServer: Add a debug log for request state transitions
2025-12-11 14:43:38 -03:00
dbgln_if(REQUESTSERVER_DEBUG, "Request::Transition[{}]: {} -> {} ({} {})", m_request_id, state_name(m_state), state_name(state), m_method, m_url);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
m_state = state;
process();
}
void Request::process()
{
switch (m_state) {
case State::Init:
handle_initial_state();
break;
case State::ReadCache:
handle_read_cache_state();
break;
RequestServer: De-duplicate some disk cache requests We previously had no protection against the same URL being requested multiple times at the same time. For example, if a URL did not have any cache entry and became requested twice, we would open two cache writers concurrently. This would result in both writers piping the response to disk, and we'd have a corrupt cache file. We now hold back requests under certain scenarios until existing cache entries have completed: * If we are opening a cache entry for reading: - If there is an existing reader entry, carry on as normal. We can have multiple readers. - If there is an existing writer entry, defer the request until it is complete. * If we are opening a cache entry for writing: - If there is an existing reader or writer entry, defer the request until it is complete.
2025-10-24 13:54:32 -03:00
case State::WaitForCache:
// Do nothing; we are waiting for the disk cache to notify us to proceed.
break;
RequestServer: Honor the only-if-cached Cache-Control request directive The only-if-cached directive currently behaves differently in the HTTP Caching RFC compared to the Fetch spec. In the former, we must return an HTTP 504 response if we do not find a cache entry. In the latter, we must return a network error. Note that similar to commit aa1517b72788df64ae8cf4bec6cb9be50f379587, we cannot test the only-if-cached directive. We implement a same-origin restriction aligned with the Fetch API that prevents our test infra from excercising this directive.
2026-01-26 09:44:56 -03:00
case State::FailedCacheOnly:
handle_failed_cache_only_state();
break;
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
case State::ServeSubstitution:
handle_serve_substitution_state();
break;
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
case State::DNSLookup:
handle_dns_lookup_state();
break;
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
case State::RetrieveCookie:
handle_retrieve_cookie_state();
break;
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
case State::Connect:
handle_connect_state();
break;
case State::Fetch:
handle_fetch_state();
break;
case State::Complete:
handle_complete_state();
break;
case State::Error:
handle_error_state();
break;
}
}
void Request::handle_initial_state()
{
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
// Check for resource substitution before anything else.
if (g_resource_substitution_map) {
if (g_resource_substitution_map->lookup(m_url).has_value()) {
transition_to_state(State::ServeSubstitution);
return;
}
}
LibHTTP+LibWeb+RequestServer: Handle the Fetch API's cache mode If the cache mode is no-store, we must not interact with the cache at all. If the cache mode is reload, we must not use any cached response. If the cache-mode is only-if-cached or force-cache, we are permitted to respond with stale cache responses. Note that we currently cannot test only-if-cached in test-web. Setting this mode also requires setting the cors mode to same-origin, but our http-test-server infra requires setting the cors mode to cors.
2026-01-20 19:12:36 -03:00
if (m_cache_mode == HTTP::CacheMode::NoStore) {
m_cache_status = HTTP::CacheRequest::CacheStatus::NotCached;
} else if (m_disk_cache.has_value()) {
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
auto open_mode = m_type == RequestType::BackgroundRevalidation
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
? HTTP::DiskCache::OpenMode::Revalidate
: HTTP::DiskCache::OpenMode::Read;
LibHTTP+LibWeb+RequestServer: Handle the Fetch API's cache mode If the cache mode is no-store, we must not interact with the cache at all. If the cache mode is reload, we must not use any cached response. If the cache-mode is only-if-cached or force-cache, we are permitted to respond with stale cache responses. Note that we currently cannot test only-if-cached in test-web. Setting this mode also requires setting the cors mode to same-origin, but our http-test-server infra requires setting the cors mode to cors.
2026-01-20 19:12:36 -03:00
m_disk_cache->open_entry(*this, m_url, m_method, m_request_headers, m_cache_mode, open_mode)
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
.visit(
[&](Optional<HTTP::CacheEntryReader&> cache_entry_reader) {
m_cache_entry_reader = cache_entry_reader;
if (m_cache_entry_reader.has_value()) {
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
if (m_cache_entry_reader->revalidation_type() == HTTP::CacheEntryReader::RevalidationType::StaleWhileRevalidate)
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
m_client.start_revalidation_request({}, m_method, m_url, m_request_headers, m_request_body, m_include_credentials, m_proxy_data);
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
if (is_revalidation_request())
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
transition_to_state(State::DNSLookup);
else
transition_to_state(State::ReadCache);
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
} else if (m_type == RequestType::BackgroundRevalidation) {
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
// If we were not able to open a cache entry reader for revalidation requests, there's no point
// in issuing a request over the network.
transition_to_state(State::Complete);
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
}
},
[&](HTTP::DiskCache::CacheHasOpenEntry) {
// If an existing entry is open for writing, we must wait for it to complete.
transition_to_state(State::WaitForCache);
});
RequestServer: De-duplicate some disk cache requests We previously had no protection against the same URL being requested multiple times at the same time. For example, if a URL did not have any cache entry and became requested twice, we would open two cache writers concurrently. This would result in both writers piping the response to disk, and we'd have a corrupt cache file. We now hold back requests under certain scenarios until existing cache entries have completed: * If we are opening a cache entry for reading: - If there is an existing reader entry, carry on as normal. We can have multiple readers. - If there is an existing writer entry, defer the request until it is complete. * If we are opening a cache entry for writing: - If there is an existing reader or writer entry, defer the request until it is complete.
2025-10-24 13:54:32 -03:00
if (m_state != State::Init)
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
return;
RequestServer: Create disk cache writers for new requests immediately We previously waited until we received all response headers before we would create the cache entry. We now create one immediately, and handle writing the headers in its own function. This will allow us to know if a cache entry writer already exists for a given cache key, and thus prevent creating a second writer at the same time.
2025-10-24 11:43:47 -03:00
RequestServer: Honor the only-if-cached Cache-Control request directive The only-if-cached directive currently behaves differently in the HTTP Caching RFC compared to the Fetch spec. In the former, we must return an HTTP 504 response if we do not find a cache entry. In the latter, we must return a network error. Note that similar to commit aa1517b72788df64ae8cf4bec6cb9be50f379587, we cannot test the only-if-cached directive. We implement a same-origin restriction aligned with the Fetch API that prevents our test infra from excercising this directive.
2026-01-26 09:44:56 -03:00
if (is_cache_only_request()) {
transition_to_state(State::FailedCacheOnly);
LibHTTP+LibWeb+RequestServer: Handle the Fetch API's cache mode If the cache mode is no-store, we must not interact with the cache at all. If the cache mode is reload, we must not use any cached response. If the cache-mode is only-if-cached or force-cache, we are permitted to respond with stale cache responses. Note that we currently cannot test only-if-cached in test-web. Setting this mode also requires setting the cors mode to same-origin, but our http-test-server infra requires setting the cors mode to cors.
2026-01-20 19:12:36 -03:00
return;
}
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
m_disk_cache->create_entry(*this, m_url, m_method, m_request_headers, m_request_start_time)
.visit(
[&](Optional<HTTP::CacheEntryWriter&> cache_entry_writer) {
m_cache_entry_writer = cache_entry_writer;
if (!m_cache_entry_writer.has_value())
m_cache_status = CacheStatus::NotCached;
},
[&](HTTP::DiskCache::CacheHasOpenEntry) {
// If an existing entry is open for reading or writing, we must wait for it to complete. An entry being
// open for reading is a rare case, but may occur if a cached response expired between the existing
// entry's cache validation and the attempted reader validation when this request was created.
transition_to_state(State::WaitForCache);
});
RequestServer: De-duplicate some disk cache requests We previously had no protection against the same URL being requested multiple times at the same time. For example, if a URL did not have any cache entry and became requested twice, we would open two cache writers concurrently. This would result in both writers piping the response to disk, and we'd have a corrupt cache file. We now hold back requests under certain scenarios until existing cache entries have completed: * If we are opening a cache entry for reading: - If there is an existing reader entry, carry on as normal. We can have multiple readers. - If there is an existing writer entry, defer the request until it is complete. * If we are opening a cache entry for writing: - If there is an existing reader or writer entry, defer the request until it is complete.
2025-10-24 13:54:32 -03:00
if (m_state != State::Init)
return;
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
transition_to_state(State::DNSLookup);
}
void Request::handle_read_cache_state()
{
RequestServer: Store the HTTP status code as an optional Rather than using 0 as a nominal value. This just feels a bit better in an upcoming commit that makes further use of it.
2026-01-22 12:20:26 -03:00
m_status_code = m_cache_entry_reader->status_code();
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
m_reason_phrase = m_cache_entry_reader->reason_phrase();
RequestServer: Rename CacheEntryReader::m_headers to m_response_headers Let's be extra clear what we're talking about here.
2025-10-30 08:24:05 -03:00
m_response_headers = m_cache_entry_reader->response_headers();
LibWebView+RequestServer: Add a simple test mode for the HTTP disk cache This mode allows us to test the HTTP disk cache with two mechanisms: 1. If RequestServer is launched with --http-disk-cache-mode=testing, it will cache requests with a X-Ladybird-Enable-Disk-Cache header. 2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response header indicating how the response was handled by the cache. There is no standard way for a web request to know what happened with respect to the disk cache, so this fills that hole for testing. This mode is not exposed to users.
2025-11-18 12:18:40 -03:00
m_cache_status = CacheStatus::ReadFromCache;
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
if (inform_client_request_started().is_error())
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
return;
LibWebView+RequestServer: Add a simple test mode for the HTTP disk cache This mode allows us to test the HTTP disk cache with two mechanisms: 1. If RequestServer is launched with --http-disk-cache-mode=testing, it will cache requests with a X-Ladybird-Enable-Disk-Cache header. 2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response header indicating how the response was handled by the cache. There is no standard way for a web request to know what happened with respect to the disk cache, so this fills that hole for testing. This mode is not exposed to users.
2025-11-18 12:18:40 -03:00
transfer_headers_to_client_if_needed();
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
LibCore+LibHTTP+RequestServer: Send data via sockets instead of pipes This brings the implementation on Unix in line with Windows, so we can drop a few ifdefs.
2026-01-17 21:21:25 -03:00
m_cache_entry_reader->send_to(
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
m_client_request_pipe->writer_fd(),
RequestServer: Capture async references to Request objects weakly It's possible for the client (WebContent) to stop a request while the request is waiting for an async callback to be invoked. So we cannot assume the request itself will still be alive once these callbacks are finally invoked.
2026-02-25 00:47:10 -03:00
weak_callback(*this, [](auto& self, auto bytes_sent) {
self.m_bytes_transferred_to_client = bytes_sent;
self.m_curl_result_code = CURLE_OK;
self.transition_to_state(State::Complete);
}),
weak_callback(*this, [](auto& self, auto bytes_sent) {
self.m_bytes_transferred_to_client = bytes_sent;
self.m_network_error = Requests::NetworkError::CacheReadFailed;
self.transition_to_state(State::Error);
}));
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
RequestServer: Honor the only-if-cached Cache-Control request directive The only-if-cached directive currently behaves differently in the HTTP Caching RFC compared to the Fetch spec. In the former, we must return an HTTP 504 response if we do not find a cache entry. In the latter, we must return a network error. Note that similar to commit aa1517b72788df64ae8cf4bec6cb9be50f379587, we cannot test the only-if-cached directive. We implement a same-origin restriction aligned with the Fetch API that prevents our test infra from excercising this directive.
2026-01-26 09:44:56 -03:00
void Request::handle_failed_cache_only_state()
{
if (m_cache_mode == HTTP::CacheMode::OnlyIfCached) {
// NB: Contrary to the HTTP Caching RFC, the Fetch API expects a network error. See:
// https://github.com/web-platform-tests/wpt/issues/8032
transition_to_state(State::Error);
return;
}
// https://httpwg.org/specs/rfc9111.html#cache-request-directive.only-if-cached
// The only-if-cached request directive indicates that the client only wishes to obtain a stored response. Caches
// that honor this request directive SHOULD, upon receiving it, respond with either a stored response consistent
// with the other constraints of the request or a 504 (Gateway Timeout) status code.
m_status_code = 504;
m_reason_phrase = MUST(String::from_utf8(HTTP::reason_phrase_for_code(*m_status_code)));
// NB: There is a privacy concern around allowing any origin to determine the cache state of other origins using
// this Cache-Control directive. The Fetch API will prevent { cache: "only-if-cached" } requests that do not
// pass a same-origin test. We mimic this here with the Access-Control-Allow-Origin response header.
m_response_headers->set({ "Access-Control-Allow-Origin"sv, m_url.origin().serialize().to_byte_string() });
if (inform_client_request_started().is_error())
return;
transfer_headers_to_client_if_needed();
m_curl_result_code = CURLE_OK;
transition_to_state(State::Complete);
}
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
void Request::handle_serve_substitution_state()
{
auto substitution = g_resource_substitution_map->lookup(m_url);
VERIFY(substitution.has_value());
dbgln("Request: Substituting '{}' with local file '{}'", m_url.serialize(), substitution->file_path);
auto file = Core::File::open(substitution->file_path, Core::File::OpenMode::Read);
if (file.is_error()) {
dbgln("Request::handle_serve_substitution_state: Failed to open file '{}': {}", substitution->file_path, file.error());
m_network_error = Requests::NetworkError::Unknown;
transition_to_state(State::Error);
return;
}
auto content = file.value()->read_until_eof();
if (content.is_error()) {
dbgln("Request::handle_serve_substitution_state: Failed to read file '{}': {}", substitution->file_path, content.error());
m_network_error = Requests::NetworkError::Unknown;
transition_to_state(State::Error);
return;
}
m_status_code = substitution->status_code;
RequestServer: Store the HTTP status code as an optional Rather than using 0 as a nominal value. This just feels a bit better in an upcoming commit that makes further use of it.
2026-01-22 12:20:26 -03:00
m_reason_phrase = MUST(String::from_utf8(HTTP::reason_phrase_for_code(*m_status_code)));
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
// Determine content type: use override if provided, otherwise guess from filename.
StringView content_type;
if (substitution->content_type.has_value())
content_type = *substitution->content_type;
else
content_type = Core::guess_mime_type_based_on_filename(substitution->file_path);
m_response_headers->append({ "Content-Type"sv, ByteString { content_type } });
m_response_headers->append({ "Content-Length"sv, ByteString::number(content.value().size()) });
RequestServer+LibWeb: Fix resource substitution for cross-origin scripts Substituted responses were missing the Access-Control-Allow-Origin header, causing the CORS check to fail and the script to be treated as a network error. Fix this by adding `Access-Control-Allow-Origin: *` to all substituted responses. Also include the script's src attribute in the error message when a script element's result is null, to make debugging easier.
2026-02-12 06:57:47 -03:00
m_response_headers->append({ "Access-Control-Allow-Origin"sv, "*"sv });
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
if (inform_client_request_started().is_error())
return;
transfer_headers_to_client_if_needed();
auto write_result = m_response_buffer.write_some(content.value());
if (write_result.is_error()) {
dbgln("Request::handle_serve_substitution_state: Failed to write content to response buffer: {}", write_result.error());
m_network_error = Requests::NetworkError::Unknown;
transition_to_state(State::Error);
return;
}
m_curl_result_code = CURLE_OK;
if (write_queued_bytes_without_blocking().is_error())
transition_to_state(State::Error);
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
void Request::handle_dns_lookup_state()
{
auto host = m_url.serialized_host().to_byte_string();
auto const& dns_info = DNSInfo::the();
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
mark_lifecycle_event(this, &WireStats::dns_started_at);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
m_resolver->dns.lookup(host, DNS::Messages::Class::IN, { DNS::Messages::ResourceType::A, DNS::Messages::ResourceType::AAAA }, { .validate_dnssec_locally = dns_info.validate_dnssec_locally })
RequestServer: Capture async references to Request objects weakly It's possible for the client (WebContent) to stop a request while the request is waiting for an async callback to be invoked. So we cannot assume the request itself will still be alive once these callbacks are finally invoked.
2026-02-25 00:47:10 -03:00
->when_rejected(weak_callback(*this, [host](auto& self, auto const& error) {
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
mark_lifecycle_event(&self, &WireStats::dns_completed_at);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
dbgln("Request::handle_dns_lookup_state: DNS lookup failed for '{}': {}", host, error);
RequestServer: Capture async references to Request objects weakly It's possible for the client (WebContent) to stop a request while the request is waiting for an async callback to be invoked. So we cannot assume the request itself will still be alive once these callbacks are finally invoked.
2026-02-25 00:47:10 -03:00
self.m_network_error = Requests::NetworkError::UnableToResolveHost;
self.transition_to_state(State::Error);
}))
.when_resolved(weak_callback(*this, [host](auto& self, NonnullRefPtr<DNS::LookupResult const> dns_result) {
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
mark_lifecycle_event(&self, &WireStats::dns_completed_at);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
if (dns_result->is_empty() || !dns_result->has_cached_addresses()) {
dbgln("Request::handle_dns_lookup_state: DNS lookup failed for '{}'", host);
RequestServer: Capture async references to Request objects weakly It's possible for the client (WebContent) to stop a request while the request is waiting for an async callback to be invoked. So we cannot assume the request itself will still be alive once these callbacks are finally invoked.
2026-02-25 00:47:10 -03:00
self.m_network_error = Requests::NetworkError::UnableToResolveHost;
self.transition_to_state(State::Error);
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
} else if (first_is_one_of(self.m_type, RequestType::Fetch, RequestType::BackgroundRevalidation)) {
RequestServer: Capture async references to Request objects weakly It's possible for the client (WebContent) to stop a request while the request is waiting for an async callback to be invoked. So we cannot assume the request itself will still be alive once these callbacks are finally invoked.
2026-02-25 00:47:10 -03:00
self.m_dns_result = move(dns_result);
self.transition_to_state(State::RetrieveCookie);
RequestServer: Pre-resolve preconnect handles via our DNS resolver handle_connect_state (used by <link rel=preconnect>) attached easy handles to the multi without setting CURLOPT_RESOLVE, so libcurl spawned its own threaded resolver for them. When the system stub resolver was slow, the thread got stuck and the next ~Request pthread_join'd it on the main thread for many seconds. Route Connect-mode requests through the same DNS + CURLOPT_RESOLVE path that Fetch uses, so libcurl never spawns the thread.
2026-04-26 06:14:46 -03:00
} else if (self.m_type == RequestType::Connect && self.m_connect_cache_level == CacheLevel::CreateConnection) {
self.m_dns_result = move(dns_result);
self.transition_to_state(State::Connect);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
} else {
RequestServer: Capture async references to Request objects weakly It's possible for the client (WebContent) to stop a request while the request is waiting for an async callback to be invoked. So we cannot assume the request itself will still be alive once these callbacks are finally invoked.
2026-02-25 00:47:10 -03:00
self.transition_to_state(State::Complete);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
RequestServer: Capture async references to Request objects weakly It's possible for the client (WebContent) to stop a request while the request is waiting for an async callback to be invoked. So we cannot assume the request itself will still be alive once these callbacks are finally invoked.
2026-02-25 00:47:10 -03:00
}));
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
void Request::handle_retrieve_cookie_state()
{
if (m_include_credentials == HTTP::Cookie::IncludeCredentials::No) {
transition_to_state(State::Fetch);
return;
}
if (auto connection = ConnectionFromClient::primary_connection(); connection.has_value()) {
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
mark_lifecycle_event(this, &WireStats::cookie_started_at);
RequestServer: Use correct request map for stale-while-revalidate cookie When we request the HTTP cookie for a SWR request, we were providing the cookie to the standard request corresponding to the SWR request's ID. This had two effects: 1. The SWR request would never finish. 2. If the corresponding standard request happened to be a connect-only request, this would result in a crash as we were expecting it to have gone through the normal fetch process. This was seen on some articles on news.google.com.
2026-04-13 17:53:29 -03:00
connection->async_retrieve_http_cookie(m_client.client_id(), m_request_id, m_type, m_url);
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
} else {
m_network_error = Requests::NetworkError::RequestServerDied;
transition_to_state(State::Error);
}
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
void Request::handle_connect_state()
{
m_curl_easy_handle = curl_easy_init();
if (!m_curl_easy_handle) {
dbgln("Request::handle_connect_state: Failed to initialize curl easy handle");
return;
}
auto set_option = [&](auto option, auto value) {
if (auto result = curl_easy_setopt(m_curl_easy_handle, option, value); result != CURLE_OK)
dbgln("Request::handle_connect_state: Failed to set curl option: {}", curl_easy_strerror(result));
};
set_option(CURLOPT_PRIVATE, this);
RequestServer: Ignore SIGPIPE for the entire service process We don't want to be terminated when we write to a pipe that's closed, so set SIGPIPE to be ignored in main. We already pass MSG_NOSIGNAL when sending over our socket in RequestPipe, so this is a secondary measure. However, cURL assumes by default that SIGPIPE is unhandled, so before any operations that interact with pipes, they set their own handler, interact with the pipe, then restore the original handler. Since we now ignore the signal, we can just tell cURL not to do this extra work.
2026-01-16 20:13:32 -03:00
set_option(CURLOPT_NOSIGNAL, 1L);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
set_option(CURLOPT_URL, m_url.to_byte_string().characters());
set_option(CURLOPT_PORT, m_url.port_or_default());
set_option(CURLOPT_CONNECTTIMEOUT, s_connect_timeout_seconds);
set_option(CURLOPT_CONNECT_ONLY, 1L);
RequestServer: Pre-resolve preconnect handles via our DNS resolver handle_connect_state (used by <link rel=preconnect>) attached easy handles to the multi without setting CURLOPT_RESOLVE, so libcurl spawned its own threaded resolver for them. When the system stub resolver was slow, the thread got stuck and the next ~Request pthread_join'd it on the main thread for many seconds. Route Connect-mode requests through the same DNS + CURLOPT_RESOLVE path that Fetch uses, so libcurl never spawns the thread.
2026-04-26 06:14:46 -03:00
// Pre-populate the multi's hostcache so libcurl skips its threaded resolver entirely.
VERIFY(m_dns_result);
auto formatted_address = build_curl_resolve_list(*m_dns_result, m_url.serialized_host(), m_url.port_or_default());
if (curl_slist* resolve_list = curl_slist_append(nullptr, formatted_address.characters())) {
set_option(CURLOPT_RESOLVE, resolve_list);
m_curl_string_lists.append(resolve_list);
} else {
VERIFY_NOT_REACHED();
}
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
mark_lifecycle_event(this, &WireStats::curl_added_at);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
auto result = curl_multi_add_handle(m_curl_multi_handle, m_curl_easy_handle);
VERIFY(result == CURLM_OK);
}
void Request::handle_fetch_state()
{
dbgln_if(REQUESTSERVER_DEBUG, "RequestServer: DNS lookup successful");
m_curl_easy_handle = curl_easy_init();
if (!m_curl_easy_handle) {
dbgln("Request::handle_start_fetch_state: Failed to initialize curl easy handle");
transition_to_state(State::Error);
return;
}
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
auto is_revalidation_request = this->is_revalidation_request();
RequestServer: Create RequestPipe abstraction for request data transfer The Win32 API equivalent to pipe2() is CreatePipe(), which creates read and write anonymous pipe handles that we can set to non-blocking via SetNamedPipeHandleState(); however, this initial approach caused issues as our Windows infrastructure assumes socket-based handles/fds and that we don't use Windows pipes at all, see Core::System::is_socket() in SystemWindows.cpp. So we use socketpair() to keep our current assumptions true. Given that Windows uses socketpair() and Unix uses pipe2(), this RequestPipe abstraction avoids ifdef soup by hiding the details about how the read/write fds pair is created and how response data is written to the client.
2025-08-24 19:01:37 -03:00
RequestServer: Do not request partial content for failed cache reads This effectively reverts 9b8f6b81084fa704bdebb0e8ec1aeee92be3cb24. I misunderstood what Chrome was doing here - they will issue a range request only for what they call "sparse" cache entries. These entries are basically used to cache partial large file, e.g. a multi-gigabyte video. If they hit a legitimate read error, they will fail the request with a ERR_CACHE_READ_FAILURE status. We will now (again) fail with a network error when a cache read fails.
2025-11-20 17:23:45 -03:00
if (!is_revalidation_request) {
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
if (inform_client_request_started().is_error())
return;
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
auto set_option = [&](auto option, auto value) {
if (auto result = curl_easy_setopt(m_curl_easy_handle, option, value); result != CURLE_OK)
dbgln("Request::handle_start_fetch_state: Failed to set curl option: {}", curl_easy_strerror(result));
};
set_option(CURLOPT_PRIVATE, this);
RequestServer: Ignore SIGPIPE for the entire service process We don't want to be terminated when we write to a pipe that's closed, so set SIGPIPE to be ignored in main. We already pass MSG_NOSIGNAL when sending over our socket in RequestPipe, so this is a secondary measure. However, cURL assumes by default that SIGPIPE is unhandled, so before any operations that interact with pipes, they set their own handler, interact with the pipe, then restore the original handler. Since we now ignore the signal, we can just tell cURL not to do this extra work.
2026-01-16 20:13:32 -03:00
set_option(CURLOPT_NOSIGNAL, 1L);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
if (auto const& path = default_certificate_path(); !path.is_empty())
set_option(CURLOPT_CAINFO, path.characters());
set_option(CURLOPT_ACCEPT_ENCODING, ""); // Empty string lets curl define the accepted encodings.
set_option(CURLOPT_URL, m_url.to_byte_string().characters());
set_option(CURLOPT_PORT, m_url.port_or_default());
set_option(CURLOPT_CONNECTTIMEOUT, s_connect_timeout_seconds);
set_option(CURLOPT_PIPEWAIT, 1L);
set_option(CURLOPT_ALTSVC, m_alt_svc_cache_path.characters());
set_option(CURLOPT_CUSTOMREQUEST, m_method.characters());
set_option(CURLOPT_FOLLOWLOCATION, 0);
RequestServer: Add a `CURL_DEBUG` flag This sets `CURLOPT_VERBOSE` to 1, which enables detailed logging of all cURL connections to stderr.
2025-11-03 07:13:25 -03:00
if constexpr (CURL_DEBUG) {
set_option(CURLOPT_VERBOSE, 1);
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
RequestServer: Instruct curl to use Windows CA cert store This is required for supporting HTTPS requests. Otherwise we fail with CURLE_PEER_FAILED_VERIFICATION.
2025-08-21 22:27:12 -03:00
#if defined(AK_OS_WINDOWS)
// Without explicitly using the OS Native CA cert store on Windows, https requests timeout with CURLE_PEER_FAILED_VERIFICATION
set_option(CURLOPT_SSL_OPTIONS, CURLSSLOPT_NATIVE_CA);
#endif
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
curl_slist* curl_headers = nullptr;
if (m_method.is_one_of("POST"sv, "PUT"sv, "PATCH"sv, "DELETE"sv)) {
set_option(CURLOPT_POSTFIELDSIZE, m_request_body.size());
set_option(CURLOPT_POSTFIELDS, m_request_body.data());
// CURLOPT_POSTFIELDS automatically sets the Content-Type header. Tell curl to remove it by setting a blank
// value if the headers passed in don't contain a content type.
LibHTTP+LibWeb+RequestServer: Move Fetch's HTTP header infra to LibHTTP The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP caching) implementation. We currently have one implementation in LibWeb for our in-memory cache and another in RequestServer for our disk cache. The implementations both largely revolve around interacting with HTTP headers. But in LibWeb, we are using Fetch's header infra, and in RS we are using are home-grown header infra from LibHTTP. So to give these a common denominator, this patch replaces the LibHTTP implementation with Fetch's infra. Our existing LibHTTP implementation was not particularly compliant with any spec, so this at least gives us a standards-based common implementation. This migration also required moving a handful of other Fetch AOs over to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP folder, so perhaps it makes sense for LibHTTP to be the implementation of that entire set of facilities.)
2025-11-26 16:13:23 -03:00
if (!m_request_headers->contains("Content-Type"sv))
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
curl_headers = curl_slist_append(curl_headers, "Content-Type:");
} else if (m_method == "HEAD"sv) {
set_option(CURLOPT_NOBODY, 1L);
}
LibHTTP+LibWeb+RequestServer: Move Fetch's HTTP header infra to LibHTTP The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP caching) implementation. We currently have one implementation in LibWeb for our in-memory cache and another in RequestServer for our disk cache. The implementations both largely revolve around interacting with HTTP headers. But in LibWeb, we are using Fetch's header infra, and in RS we are using are home-grown header infra from LibHTTP. So to give these a common denominator, this patch replaces the LibHTTP implementation with Fetch's infra. Our existing LibHTTP implementation was not particularly compliant with any spec, so this at least gives us a standards-based common implementation. This migration also required moving a handful of other Fetch AOs over to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP folder, so perhaps it makes sense for LibHTTP to be the implementation of that entire set of facilities.)
2025-11-26 16:13:23 -03:00
for (auto const& header : *m_request_headers) {
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
if (header.value.is_empty()) {
// curl will discard the header unless we pass the header name followed by a semicolon (i.e. we need to pass
// "Content-Type;" instead of "Content-Type: ").
//
// See: https://curl.se/libcurl/c/httpcustomheader.html
auto header_string = ByteString::formatted("{};", header.name);
curl_headers = curl_slist_append(curl_headers, header_string.characters());
} else {
auto header_string = ByteString::formatted("{}: {}", header.name, header.value);
curl_headers = curl_slist_append(curl_headers, header_string.characters());
}
}
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
if (is_revalidation_request) {
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
auto revalidation_attributes = HTTP::RevalidationAttributes::create(m_cache_entry_reader->response_headers());
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
VERIFY(revalidation_attributes.etag.has_value() || revalidation_attributes.last_modified.has_value());
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
if (revalidation_attributes.etag.has_value()) {
auto header_string = ByteString::formatted("If-None-Match: {}", *revalidation_attributes.etag);
curl_headers = curl_slist_append(curl_headers, header_string.characters());
}
if (revalidation_attributes.last_modified.has_value()) {
LibHTTP+RequestServer: Send revalidation attributes without parsing The caching RFC is quite strict about the format of date strings. If we received a revalidation attribute with an invalid date string, we would previously fail a runtime assertion. This was because to start a revalidation request, we would simply check for the presence of any revalidation header; but then when we issued the request, we would fail to parse the header, and end up with all attributes being null. We now don't parse the revalidation attributes at all. Whatever we receive in the Last-Modified response header is what we will send in the If-Modified-Since request header, verbatim. For better or worse, this is how other browsers behave. So if the server sends us an invalid date string, it can receive its own date format for revalidation.
2026-02-10 10:00:23 -03:00
auto header_string = ByteString::formatted("If-Modified-Since: {}", *revalidation_attributes.last_modified);
curl_headers = curl_slist_append(curl_headers, header_string.characters());
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
if (curl_headers) {
set_option(CURLOPT_HTTPHEADER, curl_headers);
m_curl_string_lists.append(curl_headers);
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
// FIXME: Set up proxy if applicable
(void)m_proxy_data;
set_option(CURLOPT_HEADERFUNCTION, &on_header_received);
set_option(CURLOPT_HEADERDATA, this);
set_option(CURLOPT_WRITEFUNCTION, &on_data_received);
set_option(CURLOPT_WRITEDATA, this);
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
if constexpr (REQUESTSERVER_WIRE_DEBUG) {
set_option(CURLOPT_NOPROGRESS, 0L);
set_option(CURLOPT_XFERINFOFUNCTION, &on_xferinfo);
set_option(CURLOPT_XFERINFODATA, this);
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
VERIFY(m_dns_result);
auto formatted_address = build_curl_resolve_list(*m_dns_result, m_url.serialized_host(), m_url.port_or_default());
if (curl_slist* resolve_list = curl_slist_append(nullptr, formatted_address.characters())) {
set_option(CURLOPT_RESOLVE, resolve_list);
m_curl_string_lists.append(resolve_list);
} else {
VERIFY_NOT_REACHED();
}
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
mark_lifecycle_event(this, &WireStats::curl_added_at);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
auto result = curl_multi_add_handle(m_curl_multi_handle, m_curl_easy_handle);
VERIFY(result == CURLM_OK);
}
void Request::handle_complete_state()
{
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
if (m_type == RequestType::Fetch) {
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
VERIFY(m_curl_result_code.has_value());
// HTTPS servers might terminate their connection without proper notice of shutdown - i.e. they do not send
// a "close notify" alert. OpenSSL version 3.2 began treating this as an error, which curl translates to
// CURLE_RECV_ERROR in the absence of a Content-Length response header. The Python server used by WPT is one
// such server. We ignore this error if we were actually able to download some response data.
LibHTTP+LibWeb+RequestServer: Move Fetch's HTTP header infra to LibHTTP The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP caching) implementation. We currently have one implementation in LibWeb for our in-memory cache and another in RequestServer for our disk cache. The implementations both largely revolve around interacting with HTTP headers. But in LibWeb, we are using Fetch's header infra, and in RS we are using are home-grown header infra from LibHTTP. So to give these a common denominator, this patch replaces the LibHTTP implementation with Fetch's infra. Our existing LibHTTP implementation was not particularly compliant with any spec, so this at least gives us a standards-based common implementation. This migration also required moving a handful of other Fetch AOs over to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP folder, so perhaps it makes sense for LibHTTP to be the implementation of that entire set of facilities.)
2025-11-26 16:13:23 -03:00
if (m_curl_result_code == CURLE_RECV_ERROR && m_bytes_transferred_to_client != 0 && !m_response_headers->contains("Content-Length"sv))
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
m_curl_result_code = CURLE_OK;
if (m_curl_result_code != CURLE_OK) {
m_network_error = curl_code_to_network_error(*m_curl_result_code);
if (m_network_error == Requests::NetworkError::Unknown) {
char const* curl_error_message = curl_easy_strerror(static_cast<CURLcode>(*m_curl_result_code));
dbgln("Request::handle_complete_state: Unable to map error ({}): \"\033[31;1m{}\033[0m\"", *m_curl_result_code, curl_error_message);
}
LibHTTP+RequestServer: Do not flush partial responses to the cache index If the cURL request completes with anything other than CURLE_OK, we must not keep the cache entry. For example, if the server's connection closes while transferring data, we receive CURLE_PARTIAL_FILE. We don't want this cache entry to be treated as valid in a subsequent request.
2026-01-07 23:49:37 -03:00
RequestServer: Do not flush the disk cache for unsuccessful requests If a request failed, or was stopped, do not attempt to write the cache entry footer to disk. Note that at this point, the cache index will not have been created, thus this entry will not be used in the future. We do still delete any partial file on disk. This serves as a more general fix for the issue addressed in commit 9f2ac14521d54c48b7301b24a3549a0d058a1d6a.
2026-01-23 09:26:37 -03:00
transition_to_state(State::Error);
return;
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
RequestServer: Check for curl errors before sending headers I noticed TLS errors only being dumped to stderr, but not shown on our regular error page. This restores that behavior and adds a regression test.
2026-03-12 12:08:15 -03:00
auto timing_info = acquire_timing_info();
transfer_headers_to_client_if_needed();
LibWeb+RequestServer: Cache bytecode after script handoff Schedule JavaScript bytecode cache generation after downloaded classic scripts and modules have been handed back to the main thread. The cache job reparses and fully compiles on the thread pool, serializes the bytecode blob, and stores it as HTTP cache sidecar data. RequestServer finalizes the disk cache entry before notifying WebContent, so the script fetcher can attach the sidecar immediately.
2026-05-03 14:17:15 -03:00
// Finalize the disk cache entry before notifying WebContent that the request is complete: WebContent may
// immediately fire off a JavaScript bytecode cache store against this entry, and that store needs the cache
// index row to already exist. If we notified first the store would race the index write and be rejected.
if (m_cache_entry_writer.has_value()) {
(void)m_cache_entry_writer->flush(m_request_headers, m_response_headers);
m_cache_entry_writer.clear();
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
m_client.async_request_finished(m_request_id, m_bytes_transferred_to_client, timing_info, m_network_error);
}
LibWeb+RequestServer: Cache bytecode after script handoff Schedule JavaScript bytecode cache generation after downloaded classic scripts and modules have been handed back to the main thread. The cache job reparses and fully compiles on the thread pool, serializes the bytecode blob, and stores it as HTTP cache sidecar data. RequestServer finalizes the disk cache entry before notifying WebContent, so the script fetcher can attach the sidecar immediately.
2026-05-03 14:17:15 -03:00
if (m_cache_entry_writer.has_value()) {
(void)m_cache_entry_writer->flush(m_request_headers, m_response_headers);
m_cache_entry_writer.clear();
}
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
m_client.request_complete({}, *this);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
void Request::handle_error_state()
{
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
if (m_type == RequestType::Fetch) {
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
// FIXME: Implement timing info for failed requests.
m_client.async_request_finished(m_request_id, m_bytes_transferred_to_client, {}, m_network_error.value_or(Requests::NetworkError::Unknown));
}
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
m_client.request_complete({}, *this);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
size_t Request::on_header_received(void* buffer, size_t size, size_t nmemb, void* user_data)
{
auto& request = *static_cast<Request*>(user_data);
auto total_size = size * nmemb;
auto header_line = StringView { static_cast<char const*>(buffer), total_size };
// We need to extract the HTTP reason phrase since it can be a custom value. Fetching infrastructure needs this
// value for setting the status message.
if (!request.m_reason_phrase.has_value() && header_line.starts_with("HTTP/"sv)) {
auto space_index = header_line.find(' ');
if (space_index.has_value())
space_index = header_line.find(' ', *space_index + 1);
if (space_index.has_value()) {
RequestServer: Only trim HTTP whitespace from response headers The Fetch spec defines HTTP whitespace as tab, LF, CR, and space. Previously, trim_whitespace was also stripping vertical tab (U+000B) and form feed (U+000C), which are not HTTP whitespace characters. Switch to HTTP::normalize_header_value which matches the fetch definition. Fixes 4 subtests for WPT test: https://wpt.live/cors/origin.htm
2026-02-26 13:31:17 -03:00
if (auto reason_phrase = HTTP::normalize_header_value(header_line.substring_view(*space_index + 1)); !reason_phrase.is_empty()) {
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
auto decoder = TextCodec::decoder_for_exact_name("ISO-8859-1"sv);
VERIFY(decoder.has_value());
request.m_reason_phrase = MUST(decoder->to_utf8(reason_phrase));
return total_size;
}
}
}
if (auto colon_index = header_line.find(':'); colon_index.has_value()) {
RequestServer: Only trim HTTP whitespace from response headers The Fetch spec defines HTTP whitespace as tab, LF, CR, and space. Previously, trim_whitespace was also stripping vertical tab (U+000B) and form feed (U+000C), which are not HTTP whitespace characters. Switch to HTTP::normalize_header_value which matches the fetch definition. Fixes 4 subtests for WPT test: https://wpt.live/cors/origin.htm
2026-02-26 13:31:17 -03:00
auto name = HTTP::normalize_header_value(header_line.substring_view(0, *colon_index));
auto value = HTTP::normalize_header_value(header_line.substring_view(*colon_index + 1));
LibHTTP+LibWeb+RequestServer: Move Fetch's HTTP header infra to LibHTTP The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP caching) implementation. We currently have one implementation in LibWeb for our in-memory cache and another in RequestServer for our disk cache. The implementations both largely revolve around interacting with HTTP headers. But in LibWeb, we are using Fetch's header infra, and in RS we are using are home-grown header infra from LibHTTP. So to give these a common denominator, this patch replaces the LibHTTP implementation with Fetch's infra. Our existing LibHTTP implementation was not particularly compliant with any spec, so this at least gives us a standards-based common implementation. This migration also required moving a handful of other Fetch AOs over to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP folder, so perhaps it makes sense for LibHTTP to be the implementation of that entire set of facilities.)
2025-11-26 16:13:23 -03:00
request.m_response_headers->append({ name, value });
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
return total_size;
}
size_t Request::on_data_received(void* buffer, size_t size, size_t nmemb, void* user_data)
{
auto& request = *static_cast<Request*>(user_data);
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
if (request.m_type == RequestType::Fetch || request.m_type == RequestType::BackgroundRevalidation)
record_chunk(&request, size * nmemb);
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
if (request.is_revalidation_request()) {
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
// If we arrive here, we did not receive an HTTP 304 response code. We must remove the cache entry and inform
// the client of the new response headers and data.
if (request.revalidation_failed().is_error())
return CURL_WRITEFUNC_ERROR;
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
request.m_disk_cache->create_entry(request, request.m_url, request.m_method, request.m_request_headers, request.m_request_start_time)
.visit(
[&](Optional<HTTP::CacheEntryWriter&> cache_entry_writer) {
request.m_cache_entry_writer = cache_entry_writer;
},
[&](HTTP::DiskCache::CacheHasOpenEntry) {
// This should not be reachable, as cache revalidation holds an exclusive lock on the cache entry.
VERIFY_NOT_REACHED();
});
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
request.transfer_headers_to_client_if_needed();
auto total_size = size * nmemb;
ReadonlyBytes bytes { static_cast<u8 const*>(buffer), total_size };
auto result = [&] -> ErrorOr<void> {
TRY(request.m_response_buffer.write_some(bytes));
return request.write_queued_bytes_without_blocking();
}();
if (result.is_error()) {
dbgln("Request::on_data_received: Aborting request because error occurred whilst writing data to the client: {}", result.error());
return CURL_WRITEFUNC_ERROR;
}
return total_size;
}
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
ErrorOr<void> Request::inform_client_request_started()
{
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
if (m_type == RequestType::BackgroundRevalidation)
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
return {};
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
auto request_pipe = RequestPipe::create();
if (request_pipe.is_error()) {
dbgln("Request::handle_read_from_cache_state: Failed to create pipe: {}", request_pipe.error());
transition_to_state(State::Error);
return request_pipe.release_error();
}
m_client_request_pipe = request_pipe.release_value();
m_client.async_request_started(m_request_id, IPC::File::adopt_fd(m_client_request_pipe->reader_fd()));
return {};
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
void Request::transfer_headers_to_client_if_needed()
{
if (exchange(m_sent_response_headers_to_client, true))
return;
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
// m_status_code may already be set (e.g. from a resource substitution).
RequestServer: Store the HTTP status code as an optional Rather than using 0 as a nominal value. This just feels a bit better in an upcoming commit that makes further use of it.
2026-01-22 12:20:26 -03:00
if (!m_status_code.has_value())
m_status_code = acquire_status_code();
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
RequestServer: Create disk cache writers for new requests immediately We previously waited until we received all response headers before we would create the cache entry. We now create one immediately, and handle writing the headers in its own function. This will allow us to know if a cache entry writer already exists for a given cache key, and thus prevent creating a second writer at the same time.
2025-10-24 11:43:47 -03:00
if (m_cache_entry_writer.has_value()) {
RequestServer: Store the HTTP status code as an optional Rather than using 0 as a nominal value. This just feels a bit better in an upcoming commit that makes further use of it.
2026-01-22 12:20:26 -03:00
if (m_cache_entry_writer->write_status_and_reason(*m_status_code, m_reason_phrase, m_request_headers, m_response_headers).is_error()) {
LibWebView+RequestServer: Add a simple test mode for the HTTP disk cache This mode allows us to test the HTTP disk cache with two mechanisms: 1. If RequestServer is launched with --http-disk-cache-mode=testing, it will cache requests with a X-Ladybird-Enable-Disk-Cache header. 2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response header indicating how the response was handled by the cache. There is no standard way for a web request to know what happened with respect to the disk cache, so this fills that hole for testing. This mode is not exposed to users.
2025-11-18 12:18:40 -03:00
m_cache_status = CacheStatus::NotCached;
RequestServer: Create disk cache writers for new requests immediately We previously waited until we received all response headers before we would create the cache entry. We now create one immediately, and handle writing the headers in its own function. This will allow us to know if a cache entry writer already exists for a given cache key, and thus prevent creating a second writer at the same time.
2025-10-24 11:43:47 -03:00
m_cache_entry_writer.clear();
LibWebView+RequestServer: Add a simple test mode for the HTTP disk cache This mode allows us to test the HTTP disk cache with two mechanisms: 1. If RequestServer is launched with --http-disk-cache-mode=testing, it will cache requests with a X-Ladybird-Enable-Disk-Cache header. 2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response header indicating how the response was handled by the cache. There is no standard way for a web request to know what happened with respect to the disk cache, so this fills that hole for testing. This mode is not exposed to users.
2025-11-18 12:18:40 -03:00
} else {
m_cache_status = CacheStatus::WrittenToCache;
}
RequestServer: Create disk cache writers for new requests immediately We previously waited until we received all response headers before we would create the cache entry. We now create one immediately, and handle writing the headers in its own function. This will allow us to know if a cache entry writer already exists for a given cache key, and thus prevent creating a second writer at the same time.
2025-10-24 11:43:47 -03:00
}
LibWebView+RequestServer: Add a simple test mode for the HTTP disk cache This mode allows us to test the HTTP disk cache with two mechanisms: 1. If RequestServer is launched with --http-disk-cache-mode=testing, it will cache requests with a X-Ladybird-Enable-Disk-Cache header. 2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response header indicating how the response was handled by the cache. There is no standard way for a web request to know what happened with respect to the disk cache, so this fills that hole for testing. This mode is not exposed to users.
2025-11-18 12:18:40 -03:00
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
if (m_type == RequestType::BackgroundRevalidation)
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
return;
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
if (m_disk_cache.has_value() && m_disk_cache->mode() == HTTP::DiskCache::Mode::Testing) {
LibWebView+RequestServer: Add a simple test mode for the HTTP disk cache This mode allows us to test the HTTP disk cache with two mechanisms: 1. If RequestServer is launched with --http-disk-cache-mode=testing, it will cache requests with a X-Ladybird-Enable-Disk-Cache header. 2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response header indicating how the response was handled by the cache. There is no standard way for a web request to know what happened with respect to the disk cache, so this fills that hole for testing. This mode is not exposed to users.
2025-11-18 12:18:40 -03:00
switch (m_cache_status) {
case CacheStatus::Unknown:
break;
case CacheStatus::NotCached:
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
m_response_headers->set({ HTTP::TEST_CACHE_STATUS_HEADER, "not-cached"sv });
LibWebView+RequestServer: Add a simple test mode for the HTTP disk cache This mode allows us to test the HTTP disk cache with two mechanisms: 1. If RequestServer is launched with --http-disk-cache-mode=testing, it will cache requests with a X-Ladybird-Enable-Disk-Cache header. 2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response header indicating how the response was handled by the cache. There is no standard way for a web request to know what happened with respect to the disk cache, so this fills that hole for testing. This mode is not exposed to users.
2025-11-18 12:18:40 -03:00
break;
case CacheStatus::WrittenToCache:
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
m_response_headers->set({ HTTP::TEST_CACHE_STATUS_HEADER, "written-to-cache"sv });
LibWebView+RequestServer: Add a simple test mode for the HTTP disk cache This mode allows us to test the HTTP disk cache with two mechanisms: 1. If RequestServer is launched with --http-disk-cache-mode=testing, it will cache requests with a X-Ladybird-Enable-Disk-Cache header. 2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response header indicating how the response was handled by the cache. There is no standard way for a web request to know what happened with respect to the disk cache, so this fills that hole for testing. This mode is not exposed to users.
2025-11-18 12:18:40 -03:00
break;
case CacheStatus::ReadFromCache:
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
m_response_headers->set({ HTTP::TEST_CACHE_STATUS_HEADER, "read-from-cache"sv });
LibWebView+RequestServer: Add a simple test mode for the HTTP disk cache This mode allows us to test the HTTP disk cache with two mechanisms: 1. If RequestServer is launched with --http-disk-cache-mode=testing, it will cache requests with a X-Ladybird-Enable-Disk-Cache header. 2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response header indicating how the response was handled by the cache. There is no standard way for a web request to know what happened with respect to the disk cache, so this fills that hole for testing. This mode is not exposed to users.
2025-11-18 12:18:40 -03:00
break;
}
}
LibHTTP+LibWeb+RequestServer: Move Fetch's HTTP header infra to LibHTTP The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP caching) implementation. We currently have one implementation in LibWeb for our in-memory cache and another in RequestServer for our disk cache. The implementations both largely revolve around interacting with HTTP headers. But in LibWeb, we are using Fetch's header infra, and in RS we are using are home-grown header infra from LibHTTP. So to give these a common denominator, this patch replaces the LibHTTP implementation with Fetch's infra. Our existing LibHTTP implementation was not particularly compliant with any spec, so this at least gives us a standards-based common implementation. This migration also required moving a handful of other Fetch AOs over to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP folder, so perhaps it makes sense for LibHTTP to be the implementation of that entire set of facilities.)
2025-11-26 16:13:23 -03:00
m_client.async_headers_became_available(m_request_id, m_response_headers->headers(), m_status_code, m_reason_phrase);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
}
ErrorOr<void> Request::write_queued_bytes_without_blocking()
{
RequestServer: Avoid O(N^2) copy when draining response buffer write_queued_bytes_without_blocking() used to allocate a Vector sized to the entire queued response buffer and memcpy every queued byte into it, on every curl on_data_received callback. When the client pipe was slower than the network, the buffer grew, and each arriving chunk triggered a full copy of everything still queued. With enough pending data this added up to tens of gigabytes of memcpy per request and stalled RequestServer for tens of seconds. Drain the stream in a loop using peek_some_contiguous() + send() directly from the underlying chunk, and discard exactly what the socket accepted. No intermediate buffer, no copy. The loop exits on EAGAIN and enables the writer notifier, matching the previous back-pressure behavior.
2026-04-22 04:14:55 -03:00
auto write_bytes_to_disk_cache = [&](ReadonlyBytes bytes) {
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
if (!m_cache_entry_writer.has_value())
return;
RequestServer: Avoid O(N^2) copy when draining response buffer write_queued_bytes_without_blocking() used to allocate a Vector sized to the entire queued response buffer and memcpy every queued byte into it, on every curl on_data_received callback. When the client pipe was slower than the network, the buffer grew, and each arriving chunk triggered a full copy of everything still queued. With enough pending data this added up to tens of gigabytes of memcpy per request and stalled RequestServer for tens of seconds. Drain the stream in a loop using peek_some_contiguous() + send() directly from the underlying chunk, and discard exactly what the socket accepted. No intermediate buffer, no copy. The loop exits on EAGAIN and enables the writer notifier, matching the previous back-pressure behavior.
2026-04-22 04:14:55 -03:00
if (m_cache_entry_writer->write_data(bytes).is_error())
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
m_cache_entry_writer.clear();
};
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
if (m_type == RequestType::BackgroundRevalidation) {
RequestServer: Avoid O(N^2) copy when draining response buffer write_queued_bytes_without_blocking() used to allocate a Vector sized to the entire queued response buffer and memcpy every queued byte into it, on every curl on_data_received callback. When the client pipe was slower than the network, the buffer grew, and each arriving chunk triggered a full copy of everything still queued. With enough pending data this added up to tens of gigabytes of memcpy per request and stalled RequestServer for tens of seconds. Drain the stream in a loop using peek_some_contiguous() + send() directly from the underlying chunk, and discard exactly what the socket accepted. No intermediate buffer, no copy. The loop exits on EAGAIN and enables the writer notifier, matching the previous back-pressure behavior.
2026-04-22 04:14:55 -03:00
while (!m_response_buffer.is_eof()) {
auto bytes = m_response_buffer.peek_some_contiguous();
write_bytes_to_disk_cache(bytes);
MUST(m_response_buffer.discard(bytes.size()));
}
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
RequestServer: Avoid O(N^2) copy when draining response buffer write_queued_bytes_without_blocking() used to allocate a Vector sized to the entire queued response buffer and memcpy every queued byte into it, on every curl on_data_received callback. When the client pipe was slower than the network, the buffer grew, and each arriving chunk triggered a full copy of everything still queued. With enough pending data this added up to tens of gigabytes of memcpy per request and stalled RequestServer for tens of seconds. Drain the stream in a loop using peek_some_contiguous() + send() directly from the underlying chunk, and discard exactly what the socket accepted. No intermediate buffer, no copy. The loop exits on EAGAIN and enables the writer notifier, matching the previous back-pressure behavior.
2026-04-22 04:14:55 -03:00
if (m_curl_result_code.has_value())
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
transition_to_state(State::Complete);
return {};
}
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
if (!m_client_writer_notifier) {
m_client_writer_notifier = Core::Notifier::construct(m_client_request_pipe->writer_fd(), Core::NotificationType::Write);
m_client_writer_notifier->set_enabled(false);
RequestServer: Capture async references to Request objects weakly It's possible for the client (WebContent) to stop a request while the request is waiting for an async callback to be invoked. So we cannot assume the request itself will still be alive once these callbacks are finally invoked.
2026-02-25 00:47:10 -03:00
m_client_writer_notifier->on_activation = weak_callback(*this, [](auto& self) {
if (auto result = self.write_queued_bytes_without_blocking(); result.is_error())
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
dbgln("Warning: Failed to write buffered request data (it's likely the client disappeared): {}", result.error());
RequestServer: Capture async references to Request objects weakly It's possible for the client (WebContent) to stop a request while the request is waiting for an async callback to be invoked. So we cannot assume the request itself will still be alive once these callbacks are finally invoked.
2026-02-25 00:47:10 -03:00
});
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
}
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
if constexpr (REQUESTSERVER_WIRE_DEBUG) {
auto& stats = wire_stats().ensure(this);
if (stats.current_pressure_window_started.has_value()) {
auto window = MonotonicTime::now() - *stats.current_pressure_window_started;
stats.total_pipe_back_pressure = stats.total_pipe_back_pressure + window;
stats.current_pressure_window_started = {};
if (window.to_milliseconds() > 50) {
dbgln("RequestServer wire-pipe-pressure: {} {} unblocked after {} ms (peak buffered={} bytes); WebContent likely behind",
m_method, m_url, window.to_milliseconds(), stats.max_buffered_bytes);
}
}
}
RequestServer: Avoid O(N^2) copy when draining response buffer write_queued_bytes_without_blocking() used to allocate a Vector sized to the entire queued response buffer and memcpy every queued byte into it, on every curl on_data_received callback. When the client pipe was slower than the network, the buffer grew, and each arriving chunk triggered a full copy of everything still queued. With enough pending data this added up to tens of gigabytes of memcpy per request and stalled RequestServer for tens of seconds. Drain the stream in a loop using peek_some_contiguous() + send() directly from the underlying chunk, and discard exactly what the socket accepted. No intermediate buffer, no copy. The loop exits on EAGAIN and enables the writer notifier, matching the previous back-pressure behavior.
2026-04-22 04:14:55 -03:00
while (!m_response_buffer.is_eof()) {
auto bytes = m_response_buffer.peek_some_contiguous();
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
RequestServer: Avoid O(N^2) copy when draining response buffer write_queued_bytes_without_blocking() used to allocate a Vector sized to the entire queued response buffer and memcpy every queued byte into it, on every curl on_data_received callback. When the client pipe was slower than the network, the buffer grew, and each arriving chunk triggered a full copy of everything still queued. With enough pending data this added up to tens of gigabytes of memcpy per request and stalled RequestServer for tens of seconds. Drain the stream in a loop using peek_some_contiguous() + send() directly from the underlying chunk, and discard exactly what the socket accepted. No intermediate buffer, no copy. The loop exits on EAGAIN and enables the writer notifier, matching the previous back-pressure behavior.
2026-04-22 04:14:55 -03:00
auto result = m_client_request_pipe->write(bytes);
if (result.is_error()) {
if (!first_is_one_of(result.error().code(), EAGAIN, EWOULDBLOCK))
return result.release_error();
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
if constexpr (REQUESTSERVER_WIRE_DEBUG) {
auto& stats = wire_stats().ensure(this);
if (!stats.current_pressure_window_started.has_value()) {
stats.current_pressure_window_started = MonotonicTime::now();
stats.pipe_back_pressure_events += 1;
dbgln("RequestServer wire-pipe-pressure: {} {} pipe full, buffering={} bytes",
m_method, m_url, m_response_buffer.used_buffer_size());
}
if (m_response_buffer.used_buffer_size() > stats.max_buffered_bytes)
stats.max_buffered_bytes = m_response_buffer.used_buffer_size();
}
RequestServer: Avoid O(N^2) copy when draining response buffer write_queued_bytes_without_blocking() used to allocate a Vector sized to the entire queued response buffer and memcpy every queued byte into it, on every curl on_data_received callback. When the client pipe was slower than the network, the buffer grew, and each arriving chunk triggered a full copy of everything still queued. With enough pending data this added up to tens of gigabytes of memcpy per request and stalled RequestServer for tens of seconds. Drain the stream in a loop using peek_some_contiguous() + send() directly from the underlying chunk, and discard exactly what the socket accepted. No intermediate buffer, no copy. The loop exits on EAGAIN and enables the writer notifier, matching the previous back-pressure behavior.
2026-04-22 04:14:55 -03:00
m_client_writer_notifier->set_enabled(true);
return {};
}
auto written = result.value();
write_bytes_to_disk_cache(bytes.slice(0, written));
MUST(m_response_buffer.discard(written));
m_bytes_transferred_to_client += written;
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
RequestServer: Avoid O(N^2) copy when draining response buffer write_queued_bytes_without_blocking() used to allocate a Vector sized to the entire queued response buffer and memcpy every queued byte into it, on every curl on_data_received callback. When the client pipe was slower than the network, the buffer grew, and each arriving chunk triggered a full copy of everything still queued. With enough pending data this added up to tens of gigabytes of memcpy per request and stalled RequestServer for tens of seconds. Drain the stream in a loop using peek_some_contiguous() + send() directly from the underlying chunk, and discard exactly what the socket accepted. No intermediate buffer, no copy. The loop exits on EAGAIN and enables the writer notifier, matching the previous back-pressure behavior.
2026-04-22 04:14:55 -03:00
if (written < bytes.size()) {
m_client_writer_notifier->set_enabled(true);
return {};
}
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
RequestServer: Avoid O(N^2) copy when draining response buffer write_queued_bytes_without_blocking() used to allocate a Vector sized to the entire queued response buffer and memcpy every queued byte into it, on every curl on_data_received callback. When the client pipe was slower than the network, the buffer grew, and each arriving chunk triggered a full copy of everything still queued. With enough pending data this added up to tens of gigabytes of memcpy per request and stalled RequestServer for tens of seconds. Drain the stream in a loop using peek_some_contiguous() + send() directly from the underlying chunk, and discard exactly what the socket accepted. No intermediate buffer, no copy. The loop exits on EAGAIN and enables the writer notifier, matching the previous back-pressure behavior.
2026-04-22 04:14:55 -03:00
m_client_writer_notifier->set_enabled(false);
if (m_curl_result_code.has_value())
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
transition_to_state(State::Complete);
return {};
}
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
bool Request::is_revalidation_request() const
{
switch (m_type) {
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
case RequestType::Fetch:
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
return m_cache_entry_reader.has_value() && m_cache_entry_reader->revalidation_type() == HTTP::CacheEntryReader::RevalidationType::MustRevalidate;
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
case RequestType::Connect:
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
return false;
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
case RequestType::BackgroundRevalidation:
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
return m_cache_entry_reader.has_value();
}
VERIFY_NOT_REACHED();
}
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
ErrorOr<void> Request::revalidation_failed()
{
RequestServer: Move Request::Type enum to its own file This will make it a bit easier to transfer over IPC.
2026-04-13 17:43:32 -03:00
if (m_type == RequestType::BackgroundRevalidation && m_disk_cache->mode() == HTTP::DiskCache::Mode::Testing)
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
m_response_headers->set({ HTTP::TEST_CACHE_REVALIDATION_STATUS_HEADER, "expired"sv });
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
m_cache_entry_reader->revalidation_failed();
m_cache_entry_reader.clear();
TRY(inform_client_request_started());
return {};
}
RequestServer: Honor the only-if-cached Cache-Control request directive The only-if-cached directive currently behaves differently in the HTTP Caching RFC compared to the Fetch spec. In the former, we must return an HTTP 504 response if we do not find a cache entry. In the latter, we must return a network error. Note that similar to commit aa1517b72788df64ae8cf4bec6cb9be50f379587, we cannot test the only-if-cached directive. We implement a same-origin restriction aligned with the Fetch API that prevents our test infra from excercising this directive.
2026-01-26 09:44:56 -03:00
bool Request::is_cache_only_request() const
{
if (m_cache_mode == HTTP::CacheMode::OnlyIfCached)
return true;
auto cache_control = m_request_headers->get("Cache-Control"sv);
return cache_control.has_value() && cache_control->contains("only-if-cached"sv, CaseSensitivity::CaseInsensitive);
}
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
u32 Request::acquire_status_code() const
{
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
if (!m_curl_easy_handle)
return 0;
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
long http_status_code = 0;
auto result = curl_easy_getinfo(m_curl_easy_handle, CURLINFO_RESPONSE_CODE, &http_status_code);
VERIFY(result == CURLE_OK);
return static_cast<u32>(http_status_code);
}
Requests::RequestTimingInfo Request::acquire_timing_info() const
{
// curl_easy_perform()
// |
// |--QUEUE
// |--|--NAMELOOKUP
// |--|--|--CONNECT
// |--|--|--|--APPCONNECT
// |--|--|--|--|--PRETRANSFER
// |--|--|--|--|--|--POSTTRANSFER
// |--|--|--|--|--|--|--STARTTRANSFER
// |--|--|--|--|--|--|--|--TOTAL
// |--|--|--|--|--|--|--|--REDIRECT
// FIXME: Implement timing info for cache hits.
if (m_cache_entry_reader.has_value())
return {};
RequestServer: Add --resource-map option for URL-to-file substitution This adds support for intercepting network requests and serving local file content instead. When a URL matches an entry in the substitution map, the local file is served while preserving the original URL's origin for cross-origin checks. Usage: Ladybird --resource-map=/path/to/map.json The JSON file format is: { "substitutions": [ { "url": "https://example.com/script.js", "file": "/path/to/local/script.js", "content_type": "application/javascript", "status_code": 200 } ] } Fields: - url (required): Exact URL to intercept (query string and fragment are stripped before matching) - file (required): Absolute path to local file to serve - content_type (optional): Override Content-Type header (defaults to guessing from filename) - status_code (optional): HTTP status code (defaults to 200) This is incredibly useful for debugging production websites: you can intercept any script, stylesheet, or other resource and replace it with a local copy containing your own debug instrumentation, console.log statements, or experimental fixes - all without modifying the actual site or setting up a local dev server.
2026-01-18 06:13:24 -03:00
// No timing info available for resource substitutions (no curl handle).
if (!m_curl_easy_handle)
return {};
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
auto get_timing_info = [&](auto option) {
curl_off_t time_value = 0;
auto result = curl_easy_getinfo(m_curl_easy_handle, option, &time_value);
VERIFY(result == CURLE_OK);
return time_value;
};
auto queue_time = get_timing_info(CURLINFO_QUEUE_TIME_T);
auto domain_lookup_time = get_timing_info(CURLINFO_NAMELOOKUP_TIME_T);
auto connect_time = get_timing_info(CURLINFO_CONNECT_TIME_T);
auto secure_connect_time = get_timing_info(CURLINFO_APPCONNECT_TIME_T);
auto request_start_time = get_timing_info(CURLINFO_PRETRANSFER_TIME_T);
auto response_start_time = get_timing_info(CURLINFO_STARTTRANSFER_TIME_T);
auto response_end_time = get_timing_info(CURLINFO_TOTAL_TIME_T);
auto encoded_body_size = get_timing_info(CURLINFO_SIZE_DOWNLOAD_T);
long http_version = 0;
auto get_version_result = curl_easy_getinfo(m_curl_easy_handle, CURLINFO_HTTP_VERSION, &http_version);
VERIFY(get_version_result == CURLE_OK);
auto http_version_alpn = Requests::ALPNHttpVersion::None;
switch (http_version) {
case CURL_HTTP_VERSION_1_0:
http_version_alpn = Requests::ALPNHttpVersion::Http1_0;
break;
case CURL_HTTP_VERSION_1_1:
http_version_alpn = Requests::ALPNHttpVersion::Http1_1;
break;
case CURL_HTTP_VERSION_2_0:
http_version_alpn = Requests::ALPNHttpVersion::Http2_TLS;
break;
case CURL_HTTP_VERSION_3:
http_version_alpn = Requests::ALPNHttpVersion::Http3;
break;
default:
http_version_alpn = Requests::ALPNHttpVersion::None;
break;
}
return Requests::RequestTimingInfo {
.domain_lookup_start_microseconds = queue_time,
.domain_lookup_end_microseconds = queue_time + domain_lookup_time,
.connect_start_microseconds = queue_time + domain_lookup_time,
.connect_end_microseconds = queue_time + domain_lookup_time + connect_time + secure_connect_time,
.secure_connect_start_microseconds = queue_time + domain_lookup_time + connect_time,
.request_start_microseconds = queue_time + domain_lookup_time + connect_time + secure_connect_time + request_start_time,
.response_start_microseconds = queue_time + domain_lookup_time + connect_time + secure_connect_time + response_start_time,
.response_end_microseconds = queue_time + domain_lookup_time + connect_time + secure_connect_time + response_end_time,
.encoded_body_size = encoded_body_size,
.http_version_alpn_identifier = http_version_alpn,
};
}
}
Reference in a new issue Copy permalink