dmaax/ladybird
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ladybird/Libraries/LibHTTP/Cache/Utilities.h

87 lines
3.6 KiB
C
Raw Permalink Normal View History

LibRequests+RequestServer: Begin implementing an HTTP disk cache This adds a disk cache for HTTP responses received from the network. For now, we take a rather conservative approach to caching. We don't cache a response until we're 100% sure it is cacheable (there are heuristics we can implement in the future based on the absence of specific headers). The cache is broken into 2 categories of files: 1. An index file. This is a SQL database containing metadata about each cache entry (URL, timestamps, etc.). 2. Cache files. Each cached response is in its own file. The file is an amalgamation of all info needed to reconstruct an HTTP response. This includes the status code, headers, body, etc. A cache entry is created once we receive the headers for a response. The index, however, is not updated at this point. We stream the body into the cache entry as it is received. Once we've successfully cached the entire body, we create an index entry in the database. If any of these steps failed along the way, the cache entry is removed and the index is left untouched. Subsequent requests are checked for cache hits from the index. If a hit is found, we read just enough of the cache entry to inform WebContent of the status code and headers. The body of the response is piped to WC via syscalls, such that the transfer happens entirely in the kernel; no need to allocate the memory for the body in userspace (WC still allocates a buffer to hold the data, of course). If an error occurs while piping the body, we currently error out the request. There is a FIXME to switch to a network request. Cache hits are also validated for freshness before they are used. If a response has expired, we remove it and its index entry, and proceed with a network request.
2025-10-07 20:59:21 -03:00
/*
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 12:23:56 -03:00
* Copyright (c) 2025-2026, Tim Flynn <trflynn89@ladybird.org>
LibRequests+RequestServer: Begin implementing an HTTP disk cache This adds a disk cache for HTTP responses received from the network. For now, we take a rather conservative approach to caching. We don't cache a response until we're 100% sure it is cacheable (there are heuristics we can implement in the future based on the absence of specific headers). The cache is broken into 2 categories of files: 1. An index file. This is a SQL database containing metadata about each cache entry (URL, timestamps, etc.). 2. Cache files. Each cached response is in its own file. The file is an amalgamation of all info needed to reconstruct an HTTP response. This includes the status code, headers, body, etc. A cache entry is created once we receive the headers for a response. The index, however, is not updated at this point. We stream the body into the cache entry as it is received. Once we've successfully cached the entire body, we create an index entry in the database. If any of these steps failed along the way, the cache entry is removed and the index is left untouched. Subsequent requests are checked for cache hits from the index. If a hit is found, we read just enough of the cache entry to inform WebContent of the status code and headers. The body of the response is piped to WC via syscalls, such that the transfer happens entirely in the kernel; no need to allocate the memory for the body in userspace (WC still allocates a buffer to hold the data, of course). If an error occurs while piping the body, we currently error out the request. There is a FIXME to switch to a network request. Cache hits are also validated for freshness before they are used. If a response has expired, we remove it and its index entry, and proceed with a network request.
2025-10-07 20:59:21 -03:00
*
* SPDX-License-Identifier: BSD-2-Clause
*/
#pragma once
LibHTTP: Store associated data with disk cache entries Add disk cache helpers that store and retrieve sidecar payloads with the same cache key and vary key as the HTTP response entry. The first consumer is JavaScript bytecode. Delete sidecars when the owning entry is removed, evicted, or replaced by a fresh response. Count their on-disk size toward the cache budget so bytecode data cannot grow outside eviction accounting. Disk cache tests cover sidecar round-trips, replacement cleanup, and eviction size accounting.
2026-05-03 14:09:18 -03:00
#include <AK/Array.h>
RequestServer: Add a time parameter to the clear cache endpoint This allows removing cache entries last accessed since a provided timestamp.
2025-11-02 18:44:30 -03:00
#include <AK/LexicalPath.h>
LibRequests+RequestServer: Begin implementing an HTTP disk cache This adds a disk cache for HTTP responses received from the network. For now, we take a rather conservative approach to caching. We don't cache a response until we're 100% sure it is cacheable (there are heuristics we can implement in the future based on the absence of specific headers). The cache is broken into 2 categories of files: 1. An index file. This is a SQL database containing metadata about each cache entry (URL, timestamps, etc.). 2. Cache files. Each cached response is in its own file. The file is an amalgamation of all info needed to reconstruct an HTTP response. This includes the status code, headers, body, etc. A cache entry is created once we receive the headers for a response. The index, however, is not updated at this point. We stream the body into the cache entry as it is received. Once we've successfully cached the entire body, we create an index entry in the database. If any of these steps failed along the way, the cache entry is removed and the index is left untouched. Subsequent requests are checked for cache hits from the index. If a hit is found, we read just enough of the cache entry to inform WebContent of the status code and headers. The body of the response is piped to WC via syscalls, such that the transfer happens entirely in the kernel; no need to allocate the memory for the body in userspace (WC still allocates a buffer to hold the data, of course). If an error occurs while piping the body, we currently error out the request. There is a FIXME to switch to a network request. Cache hits are also validated for freshness before they are used. If a response has expired, we remove it and its index entry, and proceed with a network request.
2025-10-07 20:59:21 -03:00
#include <AK/StringView.h>
#include <AK/Time.h>
#include <AK/Types.h>
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
#include <LibHTTP/Forward.h>
LibHTTP+LibWeb+RequestServer: Move Fetch's HTTP header infra to LibHTTP The end goal here is for LibHTTP to be the home of our RFC 9111 (HTTP caching) implementation. We currently have one implementation in LibWeb for our in-memory cache and another in RequestServer for our disk cache. The implementations both largely revolve around interacting with HTTP headers. But in LibWeb, we are using Fetch's header infra, and in RS we are using are home-grown header infra from LibHTTP. So to give these a common denominator, this patch replaces the LibHTTP implementation with Fetch's infra. Our existing LibHTTP implementation was not particularly compliant with any spec, so this at least gives us a standards-based common implementation. This migration also required moving a handful of other Fetch AOs over to LibHTTP. (It turns out these AOs were all from the Fetch/Infra/HTTP folder, so perhaps it makes sense for LibHTTP to be the implementation of that entire set of facilities.)
2025-11-26 16:13:23 -03:00
#include <LibHTTP/HeaderList.h>
LibRequests+RequestServer: Begin implementing an HTTP disk cache This adds a disk cache for HTTP responses received from the network. For now, we take a rather conservative approach to caching. We don't cache a response until we're 100% sure it is cacheable (there are heuristics we can implement in the future based on the absence of specific headers). The cache is broken into 2 categories of files: 1. An index file. This is a SQL database containing metadata about each cache entry (URL, timestamps, etc.). 2. Cache files. Each cached response is in its own file. The file is an amalgamation of all info needed to reconstruct an HTTP response. This includes the status code, headers, body, etc. A cache entry is created once we receive the headers for a response. The index, however, is not updated at this point. We stream the body into the cache entry as it is received. Once we've successfully cached the entire body, we create an index entry in the database. If any of these steps failed along the way, the cache entry is removed and the index is left untouched. Subsequent requests are checked for cache hits from the index. If a hit is found, we read just enough of the cache entry to inform WebContent of the status code and headers. The body of the response is piped to WC via syscalls, such that the transfer happens entirely in the kernel; no need to allocate the memory for the body in userspace (WC still allocates a buffer to hold the data, of course). If an error occurs while piping the body, we currently error out the request. There is a FIXME to switch to a network request. Cache hits are also validated for freshness before they are used. If a response has expired, we remove it and its index entry, and proceed with a network request.
2025-10-07 20:59:21 -03:00
#include <LibURL/Forward.h>
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
namespace HTTP {
LibRequests+RequestServer: Begin implementing an HTTP disk cache This adds a disk cache for HTTP responses received from the network. For now, we take a rather conservative approach to caching. We don't cache a response until we're 100% sure it is cacheable (there are heuristics we can implement in the future based on the absence of specific headers). The cache is broken into 2 categories of files: 1. An index file. This is a SQL database containing metadata about each cache entry (URL, timestamps, etc.). 2. Cache files. Each cached response is in its own file. The file is an amalgamation of all info needed to reconstruct an HTTP response. This includes the status code, headers, body, etc. A cache entry is created once we receive the headers for a response. The index, however, is not updated at this point. We stream the body into the cache entry as it is received. Once we've successfully cached the entire body, we create an index entry in the database. If any of these steps failed along the way, the cache entry is removed and the index is left untouched. Subsequent requests are checked for cache hits from the index. If a hit is found, we read just enough of the cache entry to inform WebContent of the status code and headers. The body of the response is piped to WC via syscalls, such that the transfer happens entirely in the kernel; no need to allocate the memory for the body in userspace (WC still allocates a buffer to hold the data, of course). If an error occurs while piping the body, we currently error out the request. There is a FIXME to switch to a network request. Cache hits are also validated for freshness before they are used. If a response has expired, we remove it and its index entry, and proceed with a network request.
2025-10-07 20:59:21 -03:00
LibWebView+RequestServer: Add a simple test mode for the HTTP disk cache This mode allows us to test the HTTP disk cache with two mechanisms: 1. If RequestServer is launched with --http-disk-cache-mode=testing, it will cache requests with a X-Ladybird-Enable-Disk-Cache header. 2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response header indicating how the response was handled by the cache. There is no standard way for a web request to know what happened with respect to the disk cache, so this fills that hole for testing. This mode is not exposed to users.
2025-11-18 12:18:40 -03:00
constexpr inline auto TEST_CACHE_ENABLED_HEADER = "X-Ladybird-Enable-Disk-Cache"sv;
constexpr inline auto TEST_CACHE_STATUS_HEADER = "X-Ladybird-Disk-Cache-Status"sv;
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
constexpr inline auto TEST_CACHE_REVALIDATION_STATUS_HEADER = "X-Ladybird-Revalidation-Status"sv;
RequestServer: Add a hook to advance a request's clock time for testing For example, we will want to be able to test that a cached object was expired after N seconds. Rather than waiting that time during testing, this adds a testing-only request header to internally advance the clock for a single HTTP request.
2025-11-18 12:36:55 -03:00
constexpr inline auto TEST_CACHE_REQUEST_TIME_OFFSET = "X-Ladybird-Request-Time-Offset"sv;
LibWebView+RequestServer: Add a simple test mode for the HTTP disk cache This mode allows us to test the HTTP disk cache with two mechanisms: 1. If RequestServer is launched with --http-disk-cache-mode=testing, it will cache requests with a X-Ladybird-Enable-Disk-Cache header. 2. In test mode, RS will include a X-Ladybird-Disk-Cache-Status response header indicating how the response was handled by the cache. There is no standard way for a web request to know what happened with respect to the disk cache, so this fills that hole for testing. This mode is not exposed to users.
2025-11-18 12:18:40 -03:00
LibHTTP+LibWebView+RequestServer: Allow users to set disk cache limits This adds a settings box to about:settings to allow users to limit the disk cache size. This will override the default 5 GiB limit. We do not automatically delete cache data if the new limit is suddenly less than the used disk space; this will happen on the next request. This allows multiple changes to the settings in a row without thrashing the cache. In the future, we can add more toggles, such as disabling the disk cache altogether.
2026-02-05 14:25:14 -03:00
constexpr inline u64 DEFAULT_MAXIMUM_DISK_CACHE_SIZE = 5 * GiB;
LibHTTP: Store associated data with disk cache entries Add disk cache helpers that store and retrieve sidecar payloads with the same cache key and vary key as the HTTP response entry. The first consumer is JavaScript bytecode. Delete sidecars when the owning entry is removed, evicted, or replaced by a fresh response. Count their on-disk size toward the cache budget so bytecode data cannot grow outside eviction accounting. Disk cache tests cover sidecar round-trips, replacement cleanup, and eviction size accounting.
2026-05-03 14:09:18 -03:00
enum class CacheEntryAssociatedData {
JavaScriptBytecode,
LibWasm+LibWeb: Serialize and save compiled wasm as a cache blob
2026-05-14 10:32:18 -03:00
WebAssemblyCompiledCode,
};
constexpr inline Array CACHE_ENTRY_ASSOCIATED_DATA_TYPES {
CacheEntryAssociatedData::JavaScriptBytecode,
CacheEntryAssociatedData::WebAssemblyCompiledCode,
LibHTTP: Store associated data with disk cache entries Add disk cache helpers that store and retrieve sidecar payloads with the same cache key and vary key as the HTTP response entry. The first consumer is JavaScript bytecode. Delete sidecars when the owning entry is removed, evicted, or replaced by a fresh response. Count their on-disk size toward the cache budget so bytecode data cannot grow outside eviction accounting. Disk cache tests cover sidecar round-trips, replacement cleanup, and eviction size accounting.
2026-05-03 14:09:18 -03:00
};
LibHTTP+LibWebView+RequestServer: Allow users to set disk cache limits This adds a settings box to about:settings to allow users to limit the disk cache size. This will override the default 5 GiB limit. We do not automatically delete cache data if the new limit is suddenly less than the used disk space; this will happen on the next request. This allows multiple changes to the settings in a row without thrashing the cache. In the future, we can add more toggles, such as disabling the disk cache altogether.
2026-02-05 14:25:14 -03:00
u64 compute_maximum_disk_cache_size(u64 free_bytes, u64 limit_maximum_disk_cache_size = DEFAULT_MAXIMUM_DISK_CACHE_SIZE);
LibHTTP: Impose a limit on singular disk cache entry sizes Let's not attempt to cache entries that are excessively large. We limit the cache data size to be 1/8 of the total disk cache limit, with a cap of 256 MiB.
2026-02-04 21:03:41 -03:00
u64 compute_maximum_disk_cache_entry_size(u64 maximum_disk_cache_size);
LibHTTP: Impose a limit on the total disk cache size Rather than letting our disk cache grow unbounded, let's impose a limit on the estimated total disk cache size. The limits chosen are vaguely inspired by Chromium. We impose a total disk cache limit of 5 GiB. Chromium imposes an overall limit of 1.25 GiB; I've chosen more here because we currently cache uncompressed data from cURL. The limit is further restricted by the amount of available disk space, which we just check once at startup (as does Chromium). We will choose a percentage of the free space available on systems with limited space. Our eviction errs on the side of simplicity. We will remove the least recently accessed entries until the total estimated cache size does not exceed our limit. This could potentially be improved in the future. For example, if the next entry to consider is 40 MiB, and we only need to free 1 MiB of space, we could try evicting slightly more recently used entries. This would prevent evicting more than we need to.
2026-02-04 19:53:00 -03:00
LibRequests+RequestServer: Begin implementing an HTTP disk cache This adds a disk cache for HTTP responses received from the network. For now, we take a rather conservative approach to caching. We don't cache a response until we're 100% sure it is cacheable (there are heuristics we can implement in the future based on the absence of specific headers). The cache is broken into 2 categories of files: 1. An index file. This is a SQL database containing metadata about each cache entry (URL, timestamps, etc.). 2. Cache files. Each cached response is in its own file. The file is an amalgamation of all info needed to reconstruct an HTTP response. This includes the status code, headers, body, etc. A cache entry is created once we receive the headers for a response. The index, however, is not updated at this point. We stream the body into the cache entry as it is received. Once we've successfully cached the entire body, we create an index entry in the database. If any of these steps failed along the way, the cache entry is removed and the index is left untouched. Subsequent requests are checked for cache hits from the index. If a hit is found, we read just enough of the cache entry to inform WebContent of the status code and headers. The body of the response is piped to WC via syscalls, such that the transfer happens entirely in the kernel; no need to allocate the memory for the body in userspace (WC still allocates a buffer to hold the data, of course). If an error occurs while piping the body, we currently error out the request. There is a FIXME to switch to a network request. Cache hits are also validated for freshness before they are used. If a response has expired, we remove it and its index entry, and proceed with a network request.
2025-10-07 20:59:21 -03:00
String serialize_url_for_cache_storage(URL::URL const&);
LibHTTP: Use a memory-backed database for the disk cache in test modes This just lets us create fewer cache directories during WPT. We do still create cache entries on disk, so for WPT, we introduce an extra cache key to prevent conflicts. There is an existing FIXME about this.
2026-02-15 11:35:44 -03:00
u64 create_cache_key(StringView url, StringView method, Optional<String const&> extra_cache_key = {});
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 12:23:56 -03:00
u64 create_vary_key(HeaderList const& request_headers, HeaderList const& response_headers);
LexicalPath path_for_cache_entry(LexicalPath const& cache_directory, u64 cache_key, u64 vary_key);
LibHTTP: Store associated data with disk cache entries Add disk cache helpers that store and retrieve sidecar payloads with the same cache key and vary key as the HTTP response entry. The first consumer is JavaScript bytecode. Delete sidecars when the owning entry is removed, evicted, or replaced by a fresh response. Count their on-disk size toward the cache budget so bytecode data cannot grow outside eviction accounting. Disk cache tests cover sidecar round-trips, replacement cleanup, and eviction size accounting.
2026-05-03 14:09:18 -03:00
LexicalPath path_for_cache_entry_associated_data(LexicalPath const& cache_directory, u64 cache_key, u64 vary_key, CacheEntryAssociatedData);
LibRequests+RequestServer: Begin implementing an HTTP disk cache This adds a disk cache for HTTP responses received from the network. For now, we take a rather conservative approach to caching. We don't cache a response until we're 100% sure it is cacheable (there are heuristics we can implement in the future based on the absence of specific headers). The cache is broken into 2 categories of files: 1. An index file. This is a SQL database containing metadata about each cache entry (URL, timestamps, etc.). 2. Cache files. Each cached response is in its own file. The file is an amalgamation of all info needed to reconstruct an HTTP response. This includes the status code, headers, body, etc. A cache entry is created once we receive the headers for a response. The index, however, is not updated at this point. We stream the body into the cache entry as it is received. Once we've successfully cached the entire body, we create an index entry in the database. If any of these steps failed along the way, the cache entry is removed and the index is left untouched. Subsequent requests are checked for cache hits from the index. If a hit is found, we read just enough of the cache entry to inform WebContent of the status code and headers. The body of the response is piped to WC via syscalls, such that the transfer happens entirely in the kernel; no need to allocate the memory for the body in userspace (WC still allocates a buffer to hold the data, of course). If an error occurs while piping the body, we currently error out the request. There is a FIXME to switch to a network request. Cache hits are also validated for freshness before they are used. If a response has expired, we remove it and its index entry, and proceed with a network request.
2025-10-07 20:59:21 -03:00
RequestServer: Log orphaned cache files when HTTP_DISK_CACHE_DEBUG is on When enabled, this will log all cache entry files which no longer appear in the cache index.
2026-05-11 08:44:32 -03:00
struct CacheEntryData {
u64 cache_key { 0 };
u64 vary_key { 0 };
Optional<CacheEntryAssociatedData> associated_data;
};
Optional<CacheEntryData> cache_entry_data_for_file(LexicalPath const&);
LibHTTP+LibWeb: Do not cache range requests (for now) We currently do not handle responses for range requests at all in our HTTP caches. This means if we issue a request for a range of bytes=1-10, that response will be served to a subsequent request for a range of bytes=10-20. This is obviously invalid - so until we handle these requests, just don't cache them for now.
2026-01-08 00:14:10 -03:00
bool is_cacheable(StringView method, HeaderList const&);
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
bool is_cacheable(u32 status_code, HeaderList const&);
LibRequests+RequestServer: Begin implementing an HTTP disk cache This adds a disk cache for HTTP responses received from the network. For now, we take a rather conservative approach to caching. We don't cache a response until we're 100% sure it is cacheable (there are heuristics we can implement in the future based on the absence of specific headers). The cache is broken into 2 categories of files: 1. An index file. This is a SQL database containing metadata about each cache entry (URL, timestamps, etc.). 2. Cache files. Each cached response is in its own file. The file is an amalgamation of all info needed to reconstruct an HTTP response. This includes the status code, headers, body, etc. A cache entry is created once we receive the headers for a response. The index, however, is not updated at this point. We stream the body into the cache entry as it is received. Once we've successfully cached the entire body, we create an index entry in the database. If any of these steps failed along the way, the cache entry is removed and the index is left untouched. Subsequent requests are checked for cache hits from the index. If a hit is found, we read just enough of the cache entry to inform WebContent of the status code and headers. The body of the response is piped to WC via syscalls, such that the transfer happens entirely in the kernel; no need to allocate the memory for the body in userspace (WC still allocates a buffer to hold the data, of course). If an error occurs while piping the body, we currently error out the request. There is a FIXME to switch to a network request. Cache hits are also validated for freshness before they are used. If a response has expired, we remove it and its index entry, and proceed with a network request.
2025-10-07 20:59:21 -03:00
bool is_header_exempted_from_storage(StringView name);
LibHTTP+LibWeb: Use LibHTTP's cache implementation in LibWeb There are a couple of remaining RFC 9111 methods in LibWeb's Fetch, but these are currently directly tied to the way we store GC-allocated HTTP response objects. So de-coupling that is left as a future exercise.
2025-11-28 12:39:53 -03:00
AK::Duration calculate_freshness_lifetime(u32 status_code, HeaderList const&, AK::Duration current_time_offset_for_testing = {});
AK::Duration calculate_age(HeaderList const&, UnixDateTime request_time, UnixDateTime response_time, AK::Duration current_time_offset_for_testing = {});
LibHTTP+LibWeb: Use LibHTTP to calculate stale-while-revalidate values No need to duplicate this in LibWeb. In doing so, this also fixes an apparent bug for SWR handling in LibWeb. We were previously deciding if we were in the SWR lifetime with: stale_while_revalidate > current_age However, the SWR lifetime is meant to be an additional time on top of the freshness lifetime: freshness_lifetime + stale_while_revalidate > current_age
2025-12-14 11:22:32 -03:00
AK::Duration calculate_stale_while_revalidate_lifetime(HeaderList const&, AK::Duration freshness_lifetime);
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
enum class CacheLifetimeStatus {
Fresh,
Expired,
MustRevalidate,
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
StaleWhileRevalidate,
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
};
LibHTTP: Honor the no-cache Cache-Control request directive
2026-01-26 09:49:53 -03:00
CacheLifetimeStatus cache_lifetime_status(HeaderList const& request_headers, HeaderList const& response_headers, AK::Duration freshness_lifetime, AK::Duration current_age);
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
struct RevalidationAttributes {
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
static RevalidationAttributes create(HeaderList const&);
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
Optional<ByteString> etag;
LibHTTP+RequestServer: Send revalidation attributes without parsing The caching RFC is quite strict about the format of date strings. If we received a revalidation attribute with an invalid date string, we would previously fail a runtime assertion. This was because to start a revalidation request, we would simply check for the presence of any revalidation header; but then when we issued the request, we would fail to parse the header, and end up with all attributes being null. We now don't parse the revalidation attributes at all. Whatever we receive in the Last-Modified response header is what we will send in the If-Modified-Since request header, verbatim. For better or worse, this is how other browsers behave. So if the server sends us an invalid date string, it can receive its own date format for revalidation.
2026-02-10 10:00:23 -03:00
Optional<ByteString> last_modified;
RequestServer: Implement stale cache revalidation When a request becomes stale, we will now issue a revalidation request (if the response indicates it may be revalidated). We do this by issuing a normal fetch request, with If-None-Match and/or If-Modified-Since request headers. If the server replies with an HTTP 304 status, we update the stored response headers to match the 304's headers, and serve the response to the client from the cache. If the server replies with any other code, we remove the cache entry. We will open a new cache entry to cache the new response, if possible.
2025-10-28 18:09:35 -03:00
};
LibHTTP+LibWeb: Use LibHTTP's cache implementation in LibWeb There are a couple of remaining RFC 9111 methods in LibWeb's Fetch, but these are currently directly tied to the way we store GC-allocated HTTP response objects. So de-coupling that is left as a future exercise.
2025-11-28 12:39:53 -03:00
void store_header_and_trailer_fields(HeaderList&, HeaderList const&);
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
void update_header_fields(HeaderList&, HeaderList const&);
LibRequests+RequestServer: Begin implementing an HTTP disk cache This adds a disk cache for HTTP responses received from the network. For now, we take a rather conservative approach to caching. We don't cache a response until we're 100% sure it is cacheable (there are heuristics we can implement in the future based on the absence of specific headers). The cache is broken into 2 categories of files: 1. An index file. This is a SQL database containing metadata about each cache entry (URL, timestamps, etc.). 2. Cache files. Each cached response is in its own file. The file is an amalgamation of all info needed to reconstruct an HTTP response. This includes the status code, headers, body, etc. A cache entry is created once we receive the headers for a response. The index, however, is not updated at this point. We stream the body into the cache entry as it is received. Once we've successfully cached the entire body, we create an index entry in the database. If any of these steps failed along the way, the cache entry is removed and the index is left untouched. Subsequent requests are checked for cache hits from the index. If a hit is found, we read just enough of the cache entry to inform WebContent of the status code and headers. The body of the response is piped to WC via syscalls, such that the transfer happens entirely in the kernel; no need to allocate the memory for the body in userspace (WC still allocates a buffer to hold the data, of course). If an error occurs while piping the body, we currently error out the request. There is a FIXME to switch to a network request. Cache hits are also validated for freshness before they are used. If a response has expired, we remove it and its index entry, and proceed with a network request.
2025-10-07 20:59:21 -03:00
LibHTTP: Implement a strict method to extract Cache-Control directives Our previous implementation was a bit too tolerant of bad header values. For example, extracting a "max-age" from a header value of "abmax-agecd" would have incorrectly parsed successfully. We now find exact (case-insensitive) directive matches. We also handle quoted string values, which may contain important delimeters that we would have previously split on.
2026-01-27 10:10:11 -03:00
bool contains_cache_control_directive(StringView cache_control, StringView directive);
Optional<StringView> extract_cache_control_directive(StringView cache_control, StringView directive);
LibHTTP: Add helper to extract a duration from a Cache-Control directive
2026-01-27 10:19:17 -03:00
Optional<AK::Duration> extract_cache_control_duration_directive(StringView cache_control, StringView directive, Optional<AK::Duration> valueless_fallback = {});
LibHTTP: Implement a strict method to extract Cache-Control directives Our previous implementation was a bit too tolerant of bad header values. For example, extracting a "max-age" from a header value of "abmax-agecd" would have incorrectly parsed successfully. We now find exact (case-insensitive) directive matches. We also handle quoted string values, which may contain important delimeters that we would have previously split on.
2026-01-27 10:10:11 -03:00
LibHTTP+LibWeb: Support the HTTP Vary response header We now partition the HTTP disk cache based on the Vary response header. If a cached response contains a Vary header, we look for each of the header names in the outgoing HTTP request. The outgoing request must match every header value in the original request for the cache entry to be used; otherwise, a new request will be issued, and a separate cache entry will be created. Note that we must now defer creating the disk cache file itself until we have received the response headers. The Vary key is computed from these headers, and affects the partitioned disk cache file name. There are further optimizations we can make here. If we have a Vary mismatch, we could find the best candidate cached response and issue a conditional HTTP request. The content server may then respond with an HTTP 304 if the mismatched request headers are actually okay. But for now, if we have a Vary mismatch, we issue an unconditional request as a purely correctness-oriented patch.
2025-12-28 12:23:56 -03:00
ByteString normalize_request_vary_header_values(StringView header, HeaderList const& request_headers);
LibHTTP+RequestServer: Move the HTTP cache implementation to LibHTTP We currently have two ongoing implementations of RFC 9111, HTTP caching. In order to consolidate these, this patch moves the implementation from RequestServer to LibHTTP for re-use within LibWeb.
2025-11-28 12:04:59 -03:00
AK::Duration compute_current_time_offset_for_testing(Optional<DiskCache&>, HeaderList const& request_headers);
RequestServer: Add a hook to advance a request's clock time for testing For example, we will want to be able to test that a cached object was expired after N seconds. Rather than waiting that time during testing, this adds a testing-only request header to internally advance the clock for a single HTTP request.
2025-11-18 12:36:55 -03:00
LibRequests+RequestServer: Begin implementing an HTTP disk cache This adds a disk cache for HTTP responses received from the network. For now, we take a rather conservative approach to caching. We don't cache a response until we're 100% sure it is cacheable (there are heuristics we can implement in the future based on the absence of specific headers). The cache is broken into 2 categories of files: 1. An index file. This is a SQL database containing metadata about each cache entry (URL, timestamps, etc.). 2. Cache files. Each cached response is in its own file. The file is an amalgamation of all info needed to reconstruct an HTTP response. This includes the status code, headers, body, etc. A cache entry is created once we receive the headers for a response. The index, however, is not updated at this point. We stream the body into the cache entry as it is received. Once we've successfully cached the entire body, we create an index entry in the database. If any of these steps failed along the way, the cache entry is removed and the index is left untouched. Subsequent requests are checked for cache hits from the index. If a hit is found, we read just enough of the cache entry to inform WebContent of the status code and headers. The body of the response is piped to WC via syscalls, such that the transfer happens entirely in the kernel; no need to allocate the memory for the body in userspace (WC still allocates a buffer to hold the data, of course). If an error occurs while piping the body, we currently error out the request. There is a FIXME to switch to a network request. Cache hits are also validated for freshness before they are used. If a response has expired, we remove it and its index entry, and proceed with a network request.
2025-10-07 20:59:21 -03:00
}
Reference in a new issue Copy permalink