Add explicit IgnoreBOM and ErrorMode options to LibTextCodec decoders,
and thread them through TextDecoder and TextDecoderStream.
This lets Web-facing decoder APIs preserve BOMs when requested and use
fatal error handling without post-processing decoded output.
NB: RemoveBOM was renamed to IgnoreBOM as "RemoveBOM" is the name
used by encoding_rs and was previously an implementation detail.
The new name matches what is used by the encoding standard as it
is now also used in LibWeb.
Apply helper process sandboxing by default and replace the old
--enable-sandbox switch with --disable-sandbox. Propagate the opt-out
from Ladybird, test-web, and WebDriver to WebContent, WebWorker,
RequestServer, ImageDecoder, and Compositor.
Add Seatbelt-based macOS sandboxing for the browser service processes.
The shared profile builder grants only the filesystem, network, Mach,
and process execution permissions each service needs, with fatal sandbox
violation reporting enabled so denials are visible during development.
Wire sandbox profiles into WebContent, WebWorker, RequestServer,
ImageDecoder, and Compositor. Keep Landlock and Seatbelt APIs visible
only on the platforms that use them. Allow RequestServer resource
substitution files explicitly, preserve read access for read-write cache
paths, and only grant renderer process execution for an existing
Cranelift helper.
Add opt-in Linux sandbox support to RequestServer. Ladybird and test-web
pass --enable-sandbox through when requested, and RequestServer only
installs the sandbox when that flag is present.
Allow reads for resolver and TLS configuration, plus the configured
certificate locations. Create and allow writes to the Ladybird cache
root, so libcurl alt-svc files and HTTP disk-cache files stay inside the
single writable tree. Also allow the systemd-resolved runtime directory
when present, since /etc/resolv.conf can point there.
Extend LibSandbox with owned Landlock paths, an add-if-exists helper,
read/write Landlock access, and reusable seccomp groups for filesystem
writes and network syscalls. Include POSIX file locks and socket byte
count ioctls needed by libc resolver and cache paths. Reuse the new
Landlock helper from Compositor as well, and allow its Mesa shader cache
directory so GPU startup can populate shader cache files after
sandboxing.
Add an explicit initializer for process-lifetime event loops and use it
for browser, helper service, and utility main loops. This preserves weak
event loop references for cross-thread users while making main thread
loop lifetime independent of normal program teardown.
This exposes the operation through RequestServer.ipc and RequestClient
so the client can create the stub before accessing a content-keyed
side-data file.
Map JavaScript bytecode cache sidecars from the HTTP disk cache instead
of copying them into anonymous shared buffers while handing response
headers to WebContent. Store the mapped data as ImmutableBytes on the
fetch response so script fetching can decode directly from the mapped
sidecar bytes.
Add LibHTTP coverage for retrieving associated cache data as a mappable
file, alongside the existing byte-buffer retrieval API.
Let CacheEntryWriter expose an explicit body-file handoff after a
successful flush. RequestServer sends that file to WebContent before the
request finishes, so Fetch can replace its retained memory-cache buffer
with a file-backed ImmutableBytes body.
Keep the plain flush path fd-free for existing callers, and add coverage
for mapping the body file returned by the writer. The focused disk and
memory cache web tests continue to pass.
Expose validated disk cache entry bodies as fd-backed byte ranges. Use
this for cache hits instead of creating a response socket.
Map received body ranges into ImmutableBytes on the client side. Keep
the streamed path for network responses. Adapt buffered callbacks to
receive ImmutableBytes, so cached responses can stay file-backed.
RequestServer, LibRequests, and LibWebView build with the new IPC
message and callback type.
Attach cached JavaScript bytecode sidecars to HTTP response headers so
WebContent can materialize classic and module scripts directly from a
decoded cache blob on cache hits.
Carry the disk cache vary key with the sidecar and reuse it when storing
fresh bytecode, avoiding mismatches against the augmented network
request headers used to create the cache entry.
Keep CORS-filtered module responses intact for status, MIME, and script
creation checks. Read bytecode sidecar data only from the internal
response, and treat decode or materialization failure as a cache miss
that falls back to normal source compilation.
Schedule JavaScript bytecode cache generation after downloaded classic
scripts and modules have been handed back to the main thread.
The cache job reparses and fully compiles on the thread pool,
serializes the bytecode blob, and stores it as HTTP cache sidecar data.
RequestServer finalizes the disk cache entry before notifying
WebContent, so the script fetcher can attach the sidecar immediately.
Add RequestServer messages and LibRequests helpers for storing and
retrieving HTTP cache sidecar data.
Transfer sidecar payloads as anonymous buffers so large JavaScript
bytecode blobs do not have to be serialized into IPC message bodies.
Add the same connect timeout used by regular requests to curl-backed
websockets, buffer unsent websocket bytes, and resume them from a write
notifier instead of retrying curl_easy_send() in a tight loop.
This prevents CPU spins on CURLE_AGAIN and keeps large websocket sends
from hanging when the peer applies backpressure.
Convert DNS failures and connecting-stage close requests into terminal
websocket state changes, and remove websocket entries once they close.
This keeps clients from getting stuck in CONNECTING and makes early
cancelled or failed websocket attempts terminate like the other close
paths.
handle_connect_state (used by <link rel=preconnect>) attached easy
handles to the multi without setting CURLOPT_RESOLVE, so libcurl spawned
its own threaded resolver for them. When the system stub resolver was
slow, the thread got stuck and the next ~Request pthread_join'd it on
the main thread for many seconds.
Route Connect-mode requests through the same DNS + CURLOPT_RESOLVE path
that Fetch uses, so libcurl never spawns the thread.
Per-request and per-connection logging that surfaces enough detail
to diagnose where time goes when a page load misbehaves. Gated by a
new REQUESTSERVER_WIRE_DEBUG cmakedefine.
Documentation/RequestServerWireLogging.md describes each label
(wire/wire+/wire++/wire^, wire-batch, wire-stall, wire-burst,
wire-pipe-pressure, LibDNS wire-dns, UI wire-cookie) and how to read
them.
write_queued_bytes_without_blocking() used to allocate a Vector sized
to the entire queued response buffer and memcpy every queued byte into
it, on every curl on_data_received callback. When the client pipe was
slower than the network, the buffer grew, and each arriving chunk
triggered a full copy of everything still queued. With enough pending
data this added up to tens of gigabytes of memcpy per request and
stalled RequestServer for tens of seconds.
Drain the stream in a loop using peek_some_contiguous() + send()
directly from the underlying chunk, and discard exactly what the
socket accepted. No intermediate buffer, no copy. The loop exits on
EAGAIN and enables the writer notifier, matching the previous
back-pressure behavior.
When curl invokes the socket callback to tell us it only wants one
polling direction (e.g. CURL_POLL_IN without CURL_POLL_OUT, or vice
versa), we previously had no way to disable the other direction's
notifier. Once created, a notifier stayed enabled and kept firing
curl_multi_socket_action() for events curl was no longer interested in.
Merge the read and write branches into a single helper that also
disables the notifier for a direction when its CURL_POLL_* flag is
absent from the mask. This measurably improves performance by avoiding
redundant curl_multi_socket_action() calls on sockets curl has asked
us to stop watching in a given direction.
When we request the HTTP cookie for a SWR request, we were providing the
cookie to the standard request corresponding to the SWR request's ID.
This had two effects:
1. The SWR request would never finish.
2. If the corresponding standard request happened to be a connect-only
request, this would result in a crash as we were expecting it to have
gone through the normal fetch process.
This was seen on some articles on news.google.com.
This prevents a race condition:
1. Try to connect a websocket
2. DNS lookup starts
3. JS causes the websocket to no longer be alive, and it is GCed
4. websocket_close() is called, but it doesn't find a websocket with
that websocket_id, so nothing happens
5. DNS lookup completes, and opens the websocket
6. This websocket never gets closed
By separately tracking which websockets we are trying to connect, we can
record the fact we tried to close it, and then the DNS lookup callback
can skip creating the now-unwanted websocket.
On macOS, use Mach port messaging instead of Unix domain sockets for
all IPC transport. This makes the transport capable of carrying Mach
port rights as message attachments, which is a prerequisite for sending
IOSurface handles over the main IPC channel (currently sent via a
separate out-of-band path). It also avoids the need for the FD
acknowledgement protocol that TransportSocket requires, since Mach port
right transfers are atomic in the kernel.
Three connection establishment patterns:
- Spawned helper processes (WebContent, RequestServer, etc.) use the
existing MachPortServer: the child sends its task port with a reply
port, and the parent responds with a pre-created port pair.
- Socket-bootstrapped connections (WebDriver, BrowserProcess) exchange
Mach port names over the socket, then drop the socket.
- Pre-created pairs for IPC tests and in-message transport transfer.
Attachment on macOS now wraps a MachPort instead of a file descriptor,
converting between the two via fileport_makeport()/fileport_makefd().
The LibIPC socket transport tests are disabled on macOS since they are
socket-specific.
The RequestPipe uses a socketpair for streaming response body data
from RequestServer to WebContent. On macOS, the default socket buffer
size for AF_LOCAL sockets is only ~8KB, which meant every read/write
syscall could only transfer ~8KB at a time. For large responses, this
resulted in thousands of tiny reads with significant per-read overhead.
Increase the send and receive buffer sizes to 512KB, matching the
approach already used by IPC::TransportSocket. This dramatically
improves throughput for large response bodies -- for example, fetching
a 25MB file from localhost went from ~850ms to ~25ms in testing.
Previously, `create_paired()` returned two full Transport objects, and
callers would immediately call `from_transport()` on the remote side to
extract its underlying fd. This wasted resources: the remote
Transport's IO thread, wakeup pipes, and send queue were initialized
only to be torn down without ever sending or receiving a message.
Now `create_paired()` returns `{Transport, TransportHandle}` — the
remote side is born as a lightweight handle containing just the raw fd,
skipping all unnecessary initialization.
Also replace `release_underlying_transport_for_transfer()` (which
returned a raw int fd) with `release_for_transfer()` (which returns a
TransportHandle directly), hiding the socket implementation detail
from callers including MessagePort.
This fixes a race condition where a WebSocket would report a fatal
connection error instead of a clean close when the server dropped the
underlying connection immediately after sending a Close frame.
This fixes various timeouts in WPT, such as in:
https://wpt.live/websockets/Send-null.any.worker.html?wss
Add IPC::TransportHandle as an abstraction for passing IPC
transports through .ipc messages. This replaces IPC::File at
all sites where a transport (not a generic file) is being
transferred between processes.
TransportHandle provides from_transport(),
clone_from_transport(), and create_transport() methods that
encapsulate the fd-to-socket-to-transport conversion in one
place. This is preparatory work for Mach port support on
macOS -- when that lands, only TransportHandle's internals
need to change while all .ipc definitions and call sites
remain untouched.
Consolidate the repeated socketpair + adopt + configure pattern from
4 call sites into a single Transport::create_paired() factory method.
This fixes inconsistent error handling and socket configuration across
call sites, and prepares for future mach port support on macOS.
The Fetch spec defines HTTP whitespace as tab, LF, CR, and space.
Previously, trim_whitespace was also stripping vertical tab (U+000B)
and form feed (U+000C), which are not HTTP whitespace characters.
Switch to HTTP::normalize_header_value which matches the fetch
definition.
Fixes 4 subtests for WPT test:
https://wpt.live/cors/origin.htm
It's possible for the client (WebContent) to stop a request while the
request is waiting for an async callback to be invoked. So we cannot
assume the request itself will still be alive once these callbacks are
finally invoked.
We have a common pattern of creating a `WeakPtr<T>` from a reference and
passing that into a lambda, to then take the strong ref when the lambda
is executed. Add `weak_callback(Weakable, lambda)` that returns a lambda
that only invokes the callback if a strong ref exists, and passes it as
the first argument.
Especially for the disk cache, it always felt a bit sketchy to create
these objects on the global stack. We now create them in RequestServer's
main() where we can be more certain of destruction order, and pass them
to ConnectionFromClient instances.
This adds a settings box to about:settings to allow users to limit the
disk cache size. This will override the default 5 GiB limit. We do not
automatically delete cache data if the new limit is suddenly less than
the used disk space; this will happen on the next request. This allows
multiple changes to the settings in a row without thrashing the cache.
In the future, we can add more toggles, such as disabling the disk
cache altogether.
Substituted responses were missing the Access-Control-Allow-Origin
header, causing the CORS check to fail and the script to be treated as
a network error.
Fix this by adding `Access-Control-Allow-Origin: *` to all substituted
responses.
Also include the script's src attribute in the error message when a
script element's result is null, to make debugging easier.
The caching RFC is quite strict about the format of date strings. If we
received a revalidation attribute with an invalid date string, we would
previously fail a runtime assertion. This was because to start a
revalidation request, we would simply check for the presence of any
revalidation header; but then when we issued the request, we would fail
to parse the header, and end up with all attributes being null.
We now don't parse the revalidation attributes at all. Whatever we
receive in the Last-Modified response header is what we will send in the
If-Modified-Since request header, verbatim. For better or worse, this is
how other browsers behave. So if the server sends us an invalid date
string, it can receive its own date format for revalidation.
We currently attach HTTP cookie headers from LibWeb within Fetch. This
has the downside that the cookie IPC, and the infrastructure around it,
are all synchronous. This blocks the WebContent process entirely while
the cookie is being retrieved, for every request on a page.
We now attach cookie headers from RequestServer. The state machine in
RequestServer::Request allows us to easily do this work asynchronously.
We can also skip this work entirely when the response is served from
disk cache.
Note that we will continue to parse cookies in the WebContent process.
If something goes awry during parsing. we limit the damage to that
process, instead of the UI or RequestServer.
Also note that WebSocket requests still have cookie headers attached
attached from LibWeb. This will be handled in a future patch.
In the future, we may want to introduce a memory cache for cookies in
RequestServer to avoid IPC altogether as able.
This change makes it so that signal handlers aren't registered on
windows. The reason for that is that the aim of these handlers was to
catch signals that came from another process, but the windows signal
implementation doesn't actually support signaling another process. So we
will need to use a different mechanism for this.
We don't want to be terminated when we write to a pipe that's closed,
so set SIGPIPE to be ignored in main. We already pass MSG_NOSIGNAL when
sending over our socket in RequestPipe, so this is a secondary measure.
However, cURL assumes by default that SIGPIPE is unhandled, so before
any operations that interact with pipes, they set their own handler,
interact with the pipe, then restore the original handler. Since we now
ignore the signal, we can just tell cURL not to do this extra work.
The only-if-cached directive currently behaves differently in the HTTP
Caching RFC compared to the Fetch spec. In the former, we must return
an HTTP 504 response if we do not find a cache entry. In the latter, we
must return a network error.
Note that similar to commit aa1517b727, we
cannot test the only-if-cached directive. We implement a same-origin
restriction aligned with the Fetch API that prevents our test infra from
excercising this directive.
During process exit, static variables and thread-local variables are
destroyed in an unpredictable order. The connections HashMap was static,
so when destroyed during static destruction, it would destroy
ConnectionFromClient objects whose notifiers would try to unregister
from thread data that may have already been destroyed.
Fix this by moving the connections HashMap from static storage to stack
storage in ladybird_main(). This guarantees it will be destroyed before
the function returns, while the event loop and all thread data are still
fully alive.
If a request failed, or was stopped, do not attempt to write the cache
entry footer to disk. Note that at this point, the cache index will not
have been created, thus this entry will not be used in the future. We do
still delete any partial file on disk.
This serves as a more general fix for the issue addressed in commit
9f2ac14521.
We now partition the HTTP disk cache based on the Vary response header.
If a cached response contains a Vary header, we look for each of the
header names in the outgoing HTTP request. The outgoing request must
match every header value in the original request for the cache entry
to be used; otherwise, a new request will be issued, and a separate
cache entry will be created.
Note that we must now defer creating the disk cache file itself until we
have received the response headers. The Vary key is computed from these
headers, and affects the partitioned disk cache file name.
There are further optimizations we can make here. If we have a Vary
mismatch, we could find the best candidate cached response and issue a
conditional HTTP request. The content server may then respond with an
HTTP 304 if the mismatched request headers are actually okay. But for
now, if we have a Vary mismatch, we issue an unconditional request as
a purely correctness-oriented patch.
We need to store request headers in order to handle Vary mismatches.
(Note we should also be using BLOB for header storage in sqlite, as they
are not necessarily UTF-8.)
If the cache mode is no-store, we must not interact with the cache at
all.
If the cache mode is reload, we must not use any cached response.
If the cache-mode is only-if-cached or force-cache, we are permitted
to respond with stale cache responses.
Note that we currently cannot test only-if-cached in test-web. Setting
this mode also requires setting the cors mode to same-origin, but our
http-test-server infra requires setting the cors mode to cors.
We currently disable the disk cache because the WPT runner will run more
than one RequestServer process at a time. The SQLite database does not
handle this concurrent read/write access well.
We will now enable the disk cache with a per-process database. This is
needed to ensure that WPT Fetch cache tests are sufficiently handled by
RequestServer.
