dmaax/ladybird
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ladybird/Services/RequestServer/ConnectionFromClient.h

112 lines
5.2 KiB
C
Raw Permalink Normal View History

Services: Rename ProtocolServer to RequestServer The current ProtocolServer was really only used for requests, and with the recent introduction of the WebSocket service, long-lasting connections with another server are not part of it. To better reflect this, this commit renames it to RequestServer. This commit also changes the existing 'protocol' portal to 'request', the existing 'protocol' user and group to 'request', and most mentions of the 'download' aspect of the request to 'request' when relevant, to make everything consistent across the system. Note that LibProtocol still exists as-is, but the more generic Client class and the more specific Download class have both been renamed to a more accurate RequestClient and Request to match the new names. This commit only change names, not behaviors.
2021-04-23 17:45:52 -03:00
/*
Meta: Update my e-mail address everywhere
2024-10-04 08:19:50 -03:00
* Copyright (c) 2018-2020, Andreas Kling <andreas@ladybird.org>
Services: Rename ProtocolServer to RequestServer The current ProtocolServer was really only used for requests, and with the recent introduction of the WebSocket service, long-lasting connections with another server are not part of it. To better reflect this, this commit renames it to RequestServer. This commit also changes the existing 'protocol' portal to 'request', the existing 'protocol' user and group to 'request', and most mentions of the 'download' aspect of the request to 'request' when relevant, to make everything consistent across the system. Note that LibProtocol still exists as-is, but the more generic Client class and the more specific Download class have both been renamed to a more accurate RequestClient and Request to match the new names. This commit only change names, not behaviors.
2021-04-23 17:45:52 -03:00
*
* SPDX-License-Identifier: BSD-2-Clause
*/
#pragma once
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
#include <AK/Badge.h>
Services: Rename ProtocolServer to RequestServer The current ProtocolServer was really only used for requests, and with the recent introduction of the WebSocket service, long-lasting connections with another server are not part of it. To better reflect this, this commit renames it to RequestServer. This commit also changes the existing 'protocol' portal to 'request', the existing 'protocol' user and group to 'request', and most mentions of the 'download' aspect of the request to 'request' when relevant, to make everything consistent across the system. Note that LibProtocol still exists as-is, but the more generic Client class and the more specific Download class have both been renamed to a more accurate RequestClient and Request to match the new names. This commit only change names, not behaviors.
2021-04-23 17:45:52 -03:00
#include <AK/HashMap.h>
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
#include <AK/Optional.h>
#include <AK/Time.h>
RequestServer: Expose cache sidecars over IPC Add RequestServer messages and LibRequests helpers for storing and retrieving HTTP cache sidecar data. Transfer sidecar payloads as anonymous buffers so large JavaScript bytecode blobs do not have to be serialized into IPC message bodies.
2026-05-03 14:12:09 -03:00
#include <LibCore/AnonymousBuffer.h>
LibHTTP+LibWeb+RequestServer: Handle the Fetch API's cache mode If the cache mode is no-store, we must not interact with the cache at all. If the cache mode is reload, we must not use any cached response. If the cache-mode is only-if-cached or force-cache, we are permitted to respond with stale cache responses. Note that we currently cannot test only-if-cached in test-web. Setting this mode also requires setting the cors mode to same-origin, but our http-test-server infra requires setting the cors mode to cors.
2026-01-20 19:12:36 -03:00
#include <LibHTTP/Cache/CacheMode.h>
LibHTTP+LibWebView+RequestServer: Allow users to set disk cache limits This adds a settings box to about:settings to allow users to limit the disk cache size. This will override the default 5 GiB limit. We do not automatically delete cache data if the new limit is suddenly less than the used disk space; this will happen on the next request. This allows multiple changes to the settings in a row without thrashing the cache. In the future, we can add more toggles, such as disabling the disk cache altogether.
2026-02-05 14:25:14 -03:00
#include <LibHTTP/Cache/DiskCacheSettings.h>
RequestServer: Expose cache sidecars over IPC Add RequestServer messages and LibRequests helpers for storing and retrieving HTTP cache sidecar data. Transfer sidecar payloads as anonymous buffers so large JavaScript bytecode blobs do not have to be serialized into IPC message bodies.
2026-05-03 14:12:09 -03:00
#include <LibHTTP/Cache/Utilities.h>
RequestServer: Store fewer objects globally Especially for the disk cache, it always felt a bit sketchy to create these objects on the global stack. We now create them in RequestServer's main() where we can be more certain of destruction order, and pass them to ConnectionFromClient instances.
2026-02-15 11:23:55 -03:00
#include <LibHTTP/Forward.h>
Userland: Rename IPC ClientConnection => ConnectionFromClient This was done with CLion's automatic rename feature and with: find . -name ClientConnection.h | rename 's/ClientConnection\.h/ConnectionFromClient.h/' find . -name ClientConnection.cpp | rename 's/ClientConnection\.cpp/ConnectionFromClient.cpp/'
2022-02-25 07:18:30 -03:00
#include <LibIPC/ConnectionFromClient.h>
RequestServer: Terminate cancelled and failed websockets cleanly Convert DNS failures and connecting-stage close requests into terminal websocket state changes, and remove websocket entries once they close. This keeps clients from getting stuck in CONNECTING and makes early cancelled or failed websocket attempts terminate like the other close paths.
2026-04-17 08:31:36 -03:00
#include <LibRequests/WebSocket.h>
Everywhere: Merge the WebSocket service into RequestServer This keeps the APIs separate as they are wildly different, a future improvement could be to somehow unify the APIs (if possible). Closes #23080.
2024-03-05 21:50:52 -03:00
#include <LibWebSocket/WebSocket.h>
RequestServer: Move Resolver (and related structures) to its own file In an upcoming commit to handle requests as a state machine, we will need access to Resolver from outside of ConnectionFromClient..
2025-10-22 20:26:17 -03:00
#include <RequestServer/Forward.h>
Services: Rename ProtocolServer to RequestServer The current ProtocolServer was really only used for requests, and with the recent introduction of the WebSocket service, long-lasting connections with another server are not part of it. To better reflect this, this commit renames it to RequestServer. This commit also changes the existing 'protocol' portal to 'request', the existing 'protocol' user and group to 'request', and most mentions of the 'download' aspect of the request to 'request' when relevant, to make everything consistent across the system. Note that LibProtocol still exists as-is, but the more generic Client class and the more specific Download class have both been renamed to a more accurate RequestClient and Request to match the new names. This commit only change names, not behaviors.
2021-04-23 17:45:52 -03:00
#include <RequestServer/RequestClientEndpoint.h>
#include <RequestServer/RequestServerEndpoint.h>
namespace RequestServer {
Userland: Rename IPC ClientConnection => ConnectionFromClient This was done with CLion's automatic rename feature and with: find . -name ClientConnection.h | rename 's/ClientConnection\.h/ConnectionFromClient.h/' find . -name ClientConnection.cpp | rename 's/ClientConnection\.cpp/ConnectionFromClient.cpp/'
2022-02-25 07:18:30 -03:00
class ConnectionFromClient final
: public IPC::ConnectionFromClient<RequestClientEndpoint, RequestServerEndpoint> {
C_OBJECT(ConnectionFromClient);
Services: Rename ProtocolServer to RequestServer The current ProtocolServer was really only used for requests, and with the recent introduction of the WebSocket service, long-lasting connections with another server are not part of it. To better reflect this, this commit renames it to RequestServer. This commit also changes the existing 'protocol' portal to 'request', the existing 'protocol' user and group to 'request', and most mentions of the 'download' aspect of the request to 'request' when relevant, to make everything consistent across the system. Note that LibProtocol still exists as-is, but the more generic Client class and the more specific Download class have both been renamed to a more accurate RequestClient and Request to match the new names. This commit only change names, not behaviors.
2021-04-23 17:45:52 -03:00
public:
RequestServer: Store fewer objects globally Especially for the disk cache, it always felt a bit sketchy to create these objects on the global stack. We now create them in RequestServer's main() where we can be more certain of destruction order, and pass them to ConnectionFromClient instances.
2026-02-15 11:23:55 -03:00
enum class IsPrimaryConnection {
No,
Yes,
};
using ConnectionMap = HashMap<int, NonnullRefPtr<ConnectionFromClient>>;
RequestServer+LibHTTP: Cancel requests on client exit That usually happens in "exceptional" states when the client exits unexpectedly (crash, force quit mid-load, etc), leading to the job flush timer firing and attempting to write to a nonexistent object (the client). This commit makes RS simply cancel such jobs; cancelled jobs in this state simply go away instead of sending notifications around.
2024-07-27 19:22:58 -03:00
~ConnectionFromClient() override;
Services: Rename ProtocolServer to RequestServer The current ProtocolServer was really only used for requests, and with the recent introduction of the WebSocket service, long-lasting connections with another server are not part of it. To better reflect this, this commit renames it to RequestServer. This commit also changes the existing 'protocol' portal to 'request', the existing 'protocol' user and group to 'request', and most mentions of the 'download' aspect of the request to 'request' when relevant, to make everything consistent across the system. Note that LibProtocol still exists as-is, but the more generic Client class and the more specific Download class have both been renamed to a more accurate RequestClient and Request to match the new names. This commit only change names, not behaviors.
2021-04-23 17:45:52 -03:00
virtual void die() override;
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
static Optional<ConnectionFromClient&> primary_connection();
void start_revalidation_request(Badge<Request>, ByteString method, URL::URL, NonnullRefPtr<HTTP::HeaderList> request_headers, ByteBuffer request_body, HTTP::Cookie::IncludeCredentials, Core::ProxyData proxy_data);
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
void request_complete(Badge<Request>, Request const&);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
RequestServer: Make the ThreadPool global Previously we made one thread pool per ipc client, which is excessive and can eat up all the available fds very fast. Fixes #760.
2024-07-22 17:52:21 -03:00
private:
RequestServer: Store fewer objects globally Especially for the disk cache, it always felt a bit sketchy to create these objects on the global stack. We now create them in RequestServer's main() where we can be more certain of destruction order, and pass them to ConnectionFromClient instances.
2026-02-15 11:23:55 -03:00
ConnectionFromClient(NonnullOwnPtr<IPC::Transport>, IsPrimaryConnection, ConnectionMap&, Optional<HTTP::DiskCache&>);
RequestServer: Make the ThreadPool global Previously we made one thread pool per ipc client, which is excessive and can eat up all the available fds very fast. Fixes #760.
2024-07-22 17:52:21 -03:00
LibWebView+ImageDecoder+RequestServer+WebContent: Add init_transport
2025-01-03 13:19:46 -03:00
virtual Messages::RequestServer::InitTransportResponse init_transport(int peer_pid) override;
RequestServer: Make the ThreadPool global Previously we made one thread pool per ipc client, which is excessive and can eat up all the available fds very fast. Fixes #760.
2024-07-22 17:52:21 -03:00
virtual Messages::RequestServer::ConnectNewClientResponse connect_new_client() override;
RequestServer: Add an IPC method to reconnect N request clients Similar to the same IPC on ImageDecoder, this just avoids IPC churn.
2025-08-09 14:04:48 -03:00
virtual Messages::RequestServer::ConnectNewClientsResponse connect_new_clients(size_t count) override;
LibHTTP+LibWebView+RequestServer: Allow users to set disk cache limits This adds a settings box to about:settings to allow users to limit the disk cache size. This will override the default 5 GiB limit. We do not automatically delete cache data if the new limit is suddenly less than the used disk space; this will happen on the next request. This allows multiple changes to the settings in a row without thrashing the cache. In the future, we can add more toggles, such as disabling the disk cache altogether.
2026-02-05 14:25:14 -03:00
virtual void set_disk_cache_settings(HTTP::DiskCacheSettings) override;
LibIPC+Everywhere: Always pass ownership of transferred data to clients This has been a longstanding ergonomic issue with our IPC compiler. Non- trivial types were previously passed by const&. So if we wanted to avoid expensive copies, we would have to const_cast and move the data. We now pass ownership of all transferred data to the client subclasses. This allows us to remove const_cast from these methods, and allows us to avoid some trivial expensive copies that we didn't bother to const_cast.
2025-03-08 14:22:39 -03:00
virtual Messages::RequestServer::IsSupportedProtocolResponse is_supported_protocol(ByteString) override;
LibWebView+RequestSever: Wire up a validate-DNSSEC setting option to RS
2025-05-13 07:34:55 -03:00
virtual void set_dns_server(ByteString host_or_address, u16 port, bool use_tls, bool validate_dnssec_locally) override;
LibWebView+RequestServer: Add some UI for DNS settings
2025-04-07 22:56:35 -03:00
virtual void set_use_system_dns() override;
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
virtual void start_request(u64 request_id, ByteString, URL::URL, Vector<HTTP::Header>, ByteBuffer, HTTP::CacheMode, HTTP::Cookie::IncludeCredentials, Core::ProxyData) override;
LibRequests+RequestServer: Store request IDs as u64 Not super important, but this will match an upcoming ID for stale-while- revalidate requests. It also would techinically be UB if this ID had ever overflowed. Let's make it 64-bit while we are here to also avoid the possibility that it ever will overflow.
2025-12-12 11:56:53 -03:00
virtual Messages::RequestServer::StopRequestResponse stop_request(u64 request_id) override;
virtual Messages::RequestServer::SetCertificateResponse set_certificate(u64 request_id, ByteString, ByteString) override;
RequestServer: Do not randomly assign preconnect request IDs Let's avoid the possibility that we randomly generate a request ID that already exists and is in use.
2025-12-11 16:31:12 -03:00
virtual void ensure_connection(u64 request_id, URL::URL url, ::RequestServer::CacheLevel cache_level) override;
RequestServer: Make the ThreadPool global Previously we made one thread pool per ipc client, which is excessive and can eat up all the available fds very fast. Fixes #760.
2024-07-22 17:52:21 -03:00
RequestServer: Use correct request map for stale-while-revalidate cookie When we request the HTTP cookie for a SWR request, we were providing the cookie to the standard request corresponding to the SWR request's ID. This had two effects: 1. The SWR request would never finish. 2. If the corresponding standard request happened to be a connect-only request, this would result in a crash as we were expecting it to have gone through the normal fetch process. This was seen on some articles on news.google.com.
2026-04-13 17:53:29 -03:00
virtual void retrieved_http_cookie(int client_id, u64 request_id, RequestServer::RequestType request_type, String cookie) override;
LibWeb+RequestServer: Attach HTTP cookie headers from RequestServer We currently attach HTTP cookie headers from LibWeb within Fetch. This has the downside that the cookie IPC, and the infrastructure around it, are all synchronous. This blocks the WebContent process entirely while the cookie is being retrieved, for every request on a page. We now attach cookie headers from RequestServer. The state machine in RequestServer::Request allows us to easily do this work asynchronously. We can also skip this work entirely when the response is served from disk cache. Note that we will continue to parse cookies in the WebContent process. If something goes awry during parsing. we limit the damage to that process, instead of the UI or RequestServer. Also note that WebSocket requests still have cookie headers attached attached from LibWeb. This will be handled in a future patch. In the future, we may want to introduce a memory cache for cookies in RequestServer to avoid IPC altogether as able.
2026-02-07 16:06:22 -03:00
LibRequests+RequestServer: Add a method to estimate disk cache size This allows estimating the cache size stored on disk since a provided time stamp, and in total.
2025-11-02 15:10:27 -03:00
virtual void estimate_cache_size_accessed_since(u64 cache_size_estimation_id, UnixDateTime since) override;
RequestServer: Add a time parameter to the clear cache endpoint This allows removing cache entries last accessed since a provided timestamp.
2025-11-02 18:44:30 -03:00
virtual void remove_cache_entries_accessed_since(UnixDateTime since) override;
LibWeb+RequestServer: Send cached bytecode with responses Attach cached JavaScript bytecode sidecars to HTTP response headers so WebContent can materialize classic and module scripts directly from a decoded cache blob on cache hits. Carry the disk cache vary key with the sidecar and reuse it when storing fresh bytecode, avoiding mismatches against the augmented network request headers used to create the cache entry. Keep CORS-filtered module responses intact for status, MIME, and script creation checks. Read bytecode sidecar data only from the internal response, and treat decode or materialization failure as a cache miss that falls back to normal source compilation.
2026-05-03 15:04:35 -03:00
virtual Messages::RequestServer::StoreCacheAssociatedDataResponse store_cache_associated_data(URL::URL, ByteString method, Vector<HTTP::Header> request_headers, Optional<u64> vary_key, HTTP::CacheEntryAssociatedData, Core::AnonymousBuffer) override;
virtual Messages::RequestServer::RetrieveCacheAssociatedDataResponse retrieve_cache_associated_data(URL::URL, ByteString method, Vector<HTTP::Header> request_headers, Optional<u64> vary_key, HTTP::CacheEntryAssociatedData) override;
LibHTTP+RequestServer: Add support for synthetic disk-cache entries This exposes the operation through RequestServer.ipc and RequestClient so the client can create the stub before accessing a content-keyed side-data file.
2026-05-11 10:57:16 -03:00
virtual Messages::RequestServer::CreateSyntheticCacheEntryResponse create_synthetic_cache_entry(URL::URL, ByteString method) override;
LibWebView+RequestServer: Support clearing the HTTP disk cache This is a bit of a blunt hammer, but this hooks an action to clear the HTTP disk cache into the existing Clear Cache action. Upon invocation, it stops all existing cache entries from making further progress, and then deletes the entire cache index and all cache files. In the future, we will of course want more fine-grained control over cache deletion, e.g. via an about:history page.
2025-10-09 15:24:47 -03:00
LibRequests+RequestServer: Store websocket IDs as u64 The bug fixed here is that in RequestServer's ConnectionFromClient, we stored the websocket ID as i32, despite it being i64 everywhere else. Let's make it unsigned while we are here to be consistent with how all other RS-related IDs will be soon.
2025-12-11 13:58:50 -03:00
virtual void websocket_connect(u64 websocket_id, URL::URL, ByteString, Vector<ByteString>, Vector<ByteString>, Vector<HTTP::Header>) override;
virtual void websocket_send(u64 websocket_id, bool, ByteBuffer) override;
LibRequests+RequestServer: Send oversized ws frames via shared memory
2026-05-14 09:39:07 -03:00
virtual void websocket_send_shared(u64 websocket_id, bool, Core::AnonymousBuffer) override;
LibRequests+RequestServer: Store websocket IDs as u64 The bug fixed here is that in RequestServer's ConnectionFromClient, we stored the websocket ID as i32, despite it being i64 everywhere else. Let's make it unsigned while we are here to be consistent with how all other RS-related IDs will be soon.
2025-12-11 13:58:50 -03:00
virtual void websocket_close(u64 websocket_id, u16, ByteString) override;
virtual Messages::RequestServer::WebsocketSetCertificateResponse websocket_set_certificate(u64, ByteString, ByteString) override;
RequestServer: Make the ThreadPool global Previously we made one thread pool per ipc client, which is excessive and can eat up all the available fds very fast. Fixes #760.
2024-07-22 17:52:21 -03:00
RequestServer: Use libcurl for HTTP and HTTPS downloads This replaces the use of our home-grown implementations inherited from SerenityOS.
2024-09-05 08:20:09 -03:00
static int on_socket_callback(void*, int sockfd, int what, void* user_data, void*);
static int on_timeout_callback(void*, long timeout_ms, void* user_data);
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
void check_active_requests();
RequestServer: Terminate cancelled and failed websockets cleanly Convert DNS failures and connecting-stage close requests into terminal websocket state changes, and remove websocket entries once they close. This keeps clients from getting stuck in CONNECTING and makes early cancelled or failed websocket attempts terminate like the other close paths.
2026-04-17 08:31:36 -03:00
void fail_websocket(u64 websocket_id, Requests::WebSocket::Error);
RequestServer: Use libcurl for HTTP and HTTPS downloads This replaces the use of our home-grown implementations inherited from SerenityOS.
2024-09-05 08:20:09 -03:00
LibIPC+LibWeb+LibWebView+Services: Add IPC::TransportHandle Add IPC::TransportHandle as an abstraction for passing IPC transports through .ipc messages. This replaces IPC::File at all sites where a transport (not a generic file) is being transferred between processes. TransportHandle provides from_transport(), clone_from_transport(), and create_transport() methods that encapsulate the fd-to-socket-to-transport conversion in one place. This is preparatory work for Mach port support on macOS -- when that lands, only TransportHandle's internals need to change while all .ipc definitions and call sites remain untouched.
2026-03-11 16:10:08 -03:00
ErrorOr<IPC::TransportHandle> create_client_socket();
RequestServer: Store fewer objects globally Especially for the disk cache, it always felt a bit sketchy to create these objects on the global stack. We now create them in RequestServer's main() where we can be more certain of destruction order, and pass them to ConnectionFromClient instances.
2026-02-15 11:23:55 -03:00
ConnectionMap& m_connections;
Optional<HTTP::DiskCache&> m_disk_cache;
RequestServer: Handle IPC requests on multiple threads concurrently Previously RS handled all the requests in an event loop, leading to issues with connections being started in the middle of other connections being started (and potentially blowing up the stack), ultimately causing requests to be delayed because of other requests. This commit reworks the way we handle these (specifically starting connections) by first serialising the requests, and then performing them in multiple threads concurrently; which yields a significant loading performance and reliability increase.
2024-05-08 15:15:05 -03:00
RequestServer: Use libcurl for HTTP and HTTPS downloads This replaces the use of our home-grown implementations inherited from SerenityOS.
2024-09-05 08:20:09 -03:00
void* m_curl_multi { nullptr };
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
LibRequests+RequestServer: Store request IDs as u64 Not super important, but this will match an upcoming ID for stale-while- revalidate requests. It also would techinically be UB if this ID had ever overflowed. Let's make it 64-bit while we are here to also avoid the possibility that it ever will overflow.
2025-12-12 11:56:53 -03:00
HashMap<u64, NonnullOwnPtr<Request>> m_active_requests;
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
HashMap<u64, NonnullOwnPtr<Request>> m_active_revalidation_requests;
RequestServer: Don't create already-expired WebSockets in DNS callback This prevents a race condition: 1. Try to connect a websocket 2. DNS lookup starts 3. JS causes the websocket to no longer be alive, and it is GCed 4. websocket_close() is called, but it doesn't find a websocket with that websocket_id, so nothing happens 5. DNS lookup completes, and opens the websocket 6. This websocket never gets closed By separately tracking which websockets we are trying to connect, we can record the fact we tried to close it, and then the DNS lookup callback can skip creating the now-unwanted websocket.
2026-03-31 07:39:59 -03:00
HashTable<u64> m_pending_websockets;
LibRequests+RequestServer: Store websocket IDs as u64 The bug fixed here is that in RequestServer's ConnectionFromClient, we stored the websocket ID as i32, despite it being i64 everywhere else. Let's make it unsigned while we are here to be consistent with how all other RS-related IDs will be soon.
2025-12-11 13:58:50 -03:00
HashMap<u64, RefPtr<WebSocket::WebSocket>> m_websockets;
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
RequestServer: Use libcurl for HTTP and HTTPS downloads This replaces the use of our home-grown implementations inherited from SerenityOS.
2024-09-05 08:20:09 -03:00
RefPtr<Core::Timer> m_timer;
HashMap<int, NonnullRefPtr<Core::Notifier>> m_read_notifiers;
HashMap<int, NonnullRefPtr<Core::Notifier>> m_write_notifiers;
RequestServer: Manage request lifetimes as a simple state machine We currently manage request lifetime as both an ActiveRequest structure and a series of lambda callbacks. In an upcoming patch, we will want to "pause" a request to de-duplicate equivalent requests, such that only one request goes over the network and saves its response to the disk cache. To make that easier to reason about, this adds a Request class to manage the lifetime of a request via a state machine. We will now be able to add a "waiting for disk cache" state to stop the request.
2025-10-23 21:44:55 -03:00
LibDNS+LibWeb+Ladybird+RequestServer: Let there be DNS over TLS This commit adds our own DNS resolver, with the aim of implementing DoT (and eventually DoH, maybe even DNSSEC etc.)
2024-11-01 19:53:43 -03:00
NonnullRefPtr<Resolver> m_resolver;
Meta+RequestServer: Enable HTTP/3 for curl This adds an overlay port for curl that adds the features required for HTTP/3. This is not quite compatible with the upstream vcpkg.json, because enabling HTTP/2 makes it use the default SSL backend, which is sectransp for macOS and schannel on Windows. These backends are not compatible with ngtcp2. Additionally, we can not build curl with multiple SSL backends when using ngtcp2. I couldn't find a way to selectively disable/enable dependencies based on what features are enabled, so I made HTTP/2 pick OpenSSL in our overlay port. Upstream vcpkg will likely want to support wolfSSL and GnuTLS backends for ngtcp2, so they'll be additional work to get this into upstream.
2025-07-07 12:03:37 -03:00
ByteString m_alt_svc_cache_path;
LibHTTP+RequestServer: Implement the stale-while-revalidate directive This directive allows our disk cache to serve stale responses for a time indicated by the directive itself, while we revalidate the response in the background. Issuing requests that weren't initiated by a client is a new thing for RequestServer. In this implementation, we associate the request with the client that initiated the request to the stale cache entry. This adds a "background request" mode to the Request object, to prevent us from trying to send any of the revalidation response over IPC.
2025-12-12 10:33:17 -03:00
u64 m_next_revalidation_request_id { 0 };
RequestServer: Add diagnostic wire-activity logging Per-request and per-connection logging that surfaces enough detail to diagnose where time goes when a page load misbehaves. Gated by a new REQUESTSERVER_WIRE_DEBUG cmakedefine. Documentation/RequestServerWireLogging.md describes each label (wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst, wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read them.
2026-04-26 01:56:19 -03:00
Optional<MonotonicTime> m_burst_window_started_at;
u64 m_requests_in_burst_window { 0 };
Services: Rename ProtocolServer to RequestServer The current ProtocolServer was really only used for requests, and with the recent introduction of the WebSocket service, long-lasting connections with another server are not part of it. To better reflect this, this commit renames it to RequestServer. This commit also changes the existing 'protocol' portal to 'request', the existing 'protocol' user and group to 'request', and most mentions of the 'download' aspect of the request to 'request' when relevant, to make everything consistent across the system. Note that LibProtocol still exists as-is, but the more generic Client class and the more specific Download class have both been renamed to a more accurate RequestClient and Request to match the new names. This commit only change names, not behaviors.
2021-04-23 17:45:52 -03:00
};
LibWebSocket+RequestServer: Add a WebSocketImpl using libcurl This implementation can be better improved in the future by ripping out a lot of the manual logic in LibWebSocket and rely on libcurl to parse our message payloads. But for now, this uses the 'raw mode' of curl websockets in connect-only mode to allow for somewhat seamless integration into our event loop.
2025-02-20 08:18:37 -03:00
constexpr inline uintptr_t websocket_private_tag = 0x1;
Services: Rename ProtocolServer to RequestServer The current ProtocolServer was really only used for requests, and with the recent introduction of the WebSocket service, long-lasting connections with another server are not part of it. To better reflect this, this commit renames it to RequestServer. This commit also changes the existing 'protocol' portal to 'request', the existing 'protocol' user and group to 'request', and most mentions of the 'download' aspect of the request to 'request' when relevant, to make everything consistent across the system. Note that LibProtocol still exists as-is, but the more generic Client class and the more specific Download class have both been renamed to a more accurate RequestClient and Request to match the new names. This commit only change names, not behaviors.
2021-04-23 17:45:52 -03:00
}
Reference in a new issue Copy permalink