Their cache action only works on their runners. For jobs that run on
other runners, we have use the default cache action. At least until they
update their cache product to work or fallback on other runners.
Everything that's not self-hosted or macOS is now pointing to
Blacksmith.sh. Nightly jobs and JS artifact builds use 8VCPU machines,
while regular integration builds & tests use 16VCPU machines.
The people over at Blacksmith.sh have generously offered usage of their
runners for our organization, so let's try to switch over some simple
workflows. The runners should be drop-in replacements.
When we try to retrieve benchmark results in the webhook call, we cannot
use the `head_sha` parameter since the workflow run might have a
different `head_sha` associated with it than the upstream workflow run.
This can happen when the JS repl binary workflow runs, a new commit is
pushed to master, followed by a JS benchmarks workflow run causing this
latter run to be associated with a different commit ID.
This extends the webhook payload to include the current run ID, which
can eventually be used by the webhook script to specifically download
the benchmark results associated with the current run.
Additionally, this changes the JS artifact download to use the upstream
run ID which seems nicer to do anyway.
This job uses the windows_ci_ninja preset to build just the
components and unit tests that are known to work with ClangCL on the
amd64-pc-windows-msvc target triple.
As a nightly job, its failures are non-blocking for any PRs, though
they should be fixed eventually or the job will get turned off by
email-annoyed maintainers.
For our js-benchmarks and libjs-test262 workflow runs, we already know
that they're provisioned with these repositories and can skip adding the
key and repo altogether.
We were using both wget and curl arbitrarily; use curl exclusively since
that is installed by default on our machines and containers. Fixes the
js-benchmarks workflow.
This is a workaround for the deprecation of the cache v1 REST API that
was replaced with a new protobuf RPC based API this month. vcpkg was
using the private cache backend API without the knowledge of the GitHub
actions team, and was thus broken by the deprecation.
While we wait for Microsoft to talk to Microsoft to get this feature
restored, we can use the raw actions/cache step to get almost the same
cache behavior. The only difference is that the cache will be less fine
grained than the per-package cache that VCPKG_BINARY_SOURCES of x-gha
was giving us before.
Chaining workflows does not cause the subsequently spawned workflow runs
to use the same event, but rather it uses the latest head SHA based on
the branch it runs on. This would cause the JS benchmarks jobs to not be
able to find artifacts (if a new JS repl workflow was started before the
previous one could finish) and/or assign the wrong commit SHA to the
benchmark results.
Since `github.event` contains information about the original workflow
run that spawned the JS benchmarks jobs, we can take the commit SHA from
there and use it to download the correct artifact.
We had concurrency set on the JS artifacts and JS benchmarks workflows
causing them to not run in parallel for the same combination of
(workflow, OS name). You'd expect that this causes a FIFO queue to exist
of the jobs to run sequentially, but in reality GitHub maintains a
single job to prioritize and cancels all others. We don't want that for
our artifacts and benchmarks: we want them to run on each push.
For example, a new push could have workflows getting cancelled because
someone restarted a previously failed workflow, resulting in the
following message:
"Canceling since a higher priority waiting request for [..] exists"
By removing the concurrency setting from these workflows, we make use of
all available runners to execute the jobs and potentially run some of
them in parallel. For the benchmarks however, we currently only have one
matching self-hosted runner per job, and as such they are still not run
in parallel.
This introduces a matrix for the js-benchmarks workflow and runs both
the Linux x86_64 and macOS arm64 JS repl builds against our benchmarks
repository.
The workflow-webhook action that was being used didn't work on macOS or
machines without Docker, so let's create the payload ourselves, sign it
and send it over using plain old `curl`.
It might be useful to have these artifacts, even for older commits. As
an added bonus, this causes the JS benchmarks to run as well giving us
more datapoints.
In practice this does not make a big difference, but technically it
could happen that a second JS Repl artifact was built before the first
JS Benchmarks job is executed. So make sure to filter on commit ID.
This workflow starts after a successful js-artifacts workflow, picks up
the JS repl binary and runs our js-benchmarks tool. It does not yet
publish or otherwise store the benchmark results, but it's a start!
This reverts commits bf333eaea2 and
6f69a445bd.
The commit linter needs to run on event `pull_request_target` to have
access to its secret token, which means we cannot have a dependency on
that job from another workflow that is run as a result of the
`pull_request` event.
Additionally, the linters were no longer run for first-time
contributors. This isn't a huge problem but it was nice that a
preliminary check took place before running the full CI on their PRs.
This requires us to always run the CI job and check the individual jobs'
results, since only having `needs:` will not work when `lint_commits` is
potentially skipped.
We can prevent running builds unnecessarily by requiring the linters to
succeed first. If either the code or commit linter fails, it means the
author of the PR needs to rework their branch and after pushing their
changes, we need to do a full new CI run anyway.
This was a relic from the SerenityOS CI, where architecture meant
what architecture to build Serenity for. For just ladybird, we might
want to build ladybird for multiple architectures per OS.
Originally we used the `paths-ignore` feature to skip the Lagom jobs if
no source code was changed. Afterwards, we enabled the 'required checks'
feature to prevent merging pull requests that have failing checks.
Unfortunately, marking a check as required means it always needs to be
successfully executed, even if a conditional caused it to be skipped.
By using the `paths-filter` action we were able to add the conditional
to the job instead, theoretically causing the 'required checks' feature
to start working again for documentation-only changes. As it turns out,
not the job, but the steps should get the new conditionals.
As that requires adding an `if` to every individual step and the
conditional execution has caused enough headaches since its
introduction, let's remove it for now and take the unfortunate execution
of CI jobs for granted.
Becuase we're using dynamic libraries, our configuration is
classified as a "community triplet". To not confuse vcpkg
maintainers when developers create bug reports, name them
properly. This means that the default triplet detection is now
kind of useless, so we have to invent our own for these triplets.
Apart from the fact that this workflow is failing every time, we don't
need to check the branch we're on since it's only invoked for the
default branch of the repository. We can also remove the `always()`
since there are no job dependencies nor do we want this to be
uncancelable.
Matching `master` can be a bit simpler and prevent invoking the job
altogether. Additionally, we don't need an `always()` here: we don't
have job dependencies and we want these jobs to be cancelable.
We need to invoke the CI jobs every time, even if nothing relevant has
changed, because we marked them as required status checks for PRs. If
they are not invoked, the associated status checks remain in a 'pending'
state indefinitely not allowing us to (auto-)merge the PR.
This action adds and removes the new "conflicts" label to indicate
whether a pull request is in need of conflict resolution. This both
automatically informs the author of the PR that this is the case, if
they have notifications enabled that is, and makes for an easier
evaluation of the PR queue.
What `latest-stable` means exactly also depends on the image version
we're running the workflow on, and unfortunately this can vary wildly
between GitHub runners.
Fixate the version to 16.1 for now. This version will need to be updated
as soon as we want to increase the minimum supported compiler version.
The stale action has a weird interaction with the cache which requires
the `actions: write` permission. Without this, it is unable to overwrite
any existing cache key.
We want to remind both contributors and maintainers that PRs should be
up-to-date, reviewed and merged. Stale bot does a pretty good job of
notifying the right people :^)
Even in a successful LibWeb test run, on Linux, we currently always have
LSAN output. Causing this step to fail is currently making every run of
CI fail.
If LibWeb tests did fail, they should exit with a non-zero status code
on their own, causing the pipeline to fail.
Instead of writing ASAN and UBSAN output into the same stream we use for
test logging, direct them to log files, named asan.log.$PID and
ubsan.log.$PID, and then output them in a separate CI job that runs
afterwards. This should hopefully make it easier to see which tests are
failing.
The downside is that it's now harder to tell which tests the *SAN errors
are related to.
From this action's documentation, "0 means using default retention."
There's no reason for this to have a different retention duration than
the CI results themselves.
Without this, if two CI runs on Linux both fail and want to upload
screenshots, we get an error like this on the second:
> Error: Failed to CreateArtifact: Received non-retryable error: Failed
> request: (409) Conflict: an artifact with this name already exists on
> the workflow run
Using all the inputs as part of the name should make this kind of
conflict impossible.
Our GCC pipeline is regularly timing out. When it doesn't time out, it
finishes just under the default 25 minute limit. Let's bump the timeout
to 30 minutes to give it a bit more wiggle room.
This reverts commit aeaa284be3.
This didn't actually help. It appears we actually just get stuck, so
increasing the timeout to an hour just makes CI take longer.
We seem to be brushing up against the 1500s limit (25 minutes) on both
macOS and Linux+GCC workflows. As we keep importing more WPT tests, we
will exasperate the issue.
For some reason, Microsoft have decided to remove Xcode 16 from macOS 14
images. We require Xcode 16 for Swift 6.
See: https://github.com/actions/runner-images/issues/10703
Because macOS 15 images are still in preview, their availability is much
lower than macOS 14 images. To hopefully alleviate the amount of time we
are waiting in the runner queue, for now this only upgrades the workflow
which uses Swift.
In addition to changing the build-type dependent build directories, we
can take this opportunity to move the vcpkg cache directory to the Build
folder itself. This probably isn't 100% needed, but it ensures that no
leftover artifacts are used from non-dynamic vcpkg builds, and it's also
generally nice to have all build artifacts under Build.
This change helps to ensure issue authors include the details needed to
reproduce problems — including repro steps, expected behavior, a reduced
test case, and logs/backtrace.
This uses the setup-python action in the setup action to install python
3.12, and removes the --break-system-packages hack that the system pip
requires.
Merely specifying `self-hosted` is not enough - to get closer to
reproducible timings, we want these jobs to run on dedicated hardware.
The `test262-runner` label was introduced for runners that offer this.
swift-format is only packaged for homebrew, Arch, and nixpkgs at the
moment. Rather than installing swiftly and a swift toolchain, let's
change the job to run on macOS.
This loader supports whatever format libavformat and libavcodec can
handle. Currently only seekable streams are supported, and we still have
some limitations as to the number of channels and sample format.
Plays all non-streaming audio files at:
https://tools.woolyss.com/html5-audio-video-tester/
This is needed to allow using clang plugins during PR workflows. They
currently are not included because they resulted in a clean build on
each run. With this, they should be cached.
The plugins build that runs on master currently uses the same ccache key
as non-plugin builds in PRs. This means all clang PR builds are not able
to use ccache.
Move the plugins build to a separate job. This job also has sanitizers
disabled to make the plugins build quicker.
In theory the clang module map should not have absolute paths for the
headers. Other Swift projects seem to use the -ivfsoverlay feature of
clang to work around this, but it seems difficult to get to work.
It’s possible but unlikely that a push of generated notes back to the
remote can fail with a message like
> [remote rejected] ... cannot > lock ref 'refs/notes/commits': is at
> f3648f50bb but expected 47686bf473
See https://github.com/LadybirdBrowser/ladybird/actions/runs/10054314539
So this change makes the call to “git push” for notes not return 0 even
if it fails — because it’s not actually a fatal error when it happens,
and not something we need to stop and fix. Instead, it fixes itself.
Specifically: If one CI job for a PR merge/push to master generates
some notes but fails to push them to the origin, the notes don’t get
dropped on the floor and lost.
Instead, the notes-generator tool looks at the entire history; and if it
finds commits that don’t have notes — even if those commits are not part
of the PR push/merge it’s operating on — it generates notes for those.
In other words, if notes for one PR push/job fail for some reason to get
pushed back to the remote, they get regenerated by the next PR push/job.
libqt6svg6-dev - We don't use Qt's SVG support any longer.
qt6-multimedia-dev - We already install libpulse-dev, so the Qt6
multimedia package is unused.
This change makes the notes-push.yml·workflow fetch the entire history
on each push, rather than just the one HEAD commit it otherwise sees.
Because we push multiple commits from PR merges, git-gloss need access
on each push to an arbitrary number of commits (however many commits
were pushed in a PR that got merged). There’s no easy way from GH
Actions to know how may commits got pushed at the same time. So we
instead fetch the whole history.
This change, for each commit pushed/merged to master:
- causes new git Notes with GitHub PR/issue/reviewer/author links to be
auto-generated for that commit
- pushes the updated refs/notes/commits tree+references back to the repo
AK will depend on some vcpkg dependencies, so the Lagom tools build will
need to know how to use vcpkg. We can do this by sym-linking vcpkg.json
to Meta/Lagom (as vcpkg.json has to be in the CMake source directory).
We also need a CMakePresets.json in the source directory, which can just
include the root file. The root CMakePresets then needs to define paths
relative to ${fileDir} rather than ${sourceDir}.
Trying to build VulkanLoader from source is a giant headache of
unnecessary packages. Every modern distro has vulkan packages, let's
depend on those instead of trying to build something for both wayland
and X11.
VP9 continues to function, but this also allows AV1 to be decoded. With
this commit, H.264 is still non-functional, as the decoder requires
some extra initial data from the track definition in the Matroska file.
LibLocale was split off from LibUnicode a couple years ago to reduce the
number of applications on SerenityOS that depend on CLDR data. Now that
we use ICU, both LibUnicode and LibLocale are actually linking in this
data. And since vcpkg gives us static libraries, both libraries are over
30MB in size.
This patch reverts the separation and merges LibLocale into LibUnicode
again. We now have just one library that includes the ICU data.
Further, this will let LibUnicode share the locale cache that previously
would only exist in LibLocale.
Most users will be building with Xcode Clang on macOS anyway, as our
build scripts default to the system compiler if it's new enough. We
already have an upstream Clang-based workflow on Linux, so we won't lose
any compiler coverage by switching to Apple Clang on macOS.
This should help us avoid build breakages like #186.
This changes the Sanitizer configs to build all the vcpkg dependencies
with our specified CFLAGS and CXXFLAGS for ASAN and UBSAN.
Unfortunately, we can't yet enable actually compiling them with
sanitizers enabled, because this causes test failures that need to be
investigated.
Unlike in Serenity, let's just run this job once per PR rather than on
every job. This lets us install fewer dependencies on the build-and-test
pipelines.
Turns out vcpkg does not yet support Universal binaries. While they are
working on it, lets produce arm64 binaries only for now to unblock the
pipeline.
Rather than removing LibLocale entirely, we will use it as a wrapper
around ICU (which has some C-like interfaces, and uses UTF-16 for its
string types). Using ICU will provide better web compatibility overall,
and will let us implement features we were previously unable to (e.g.
Intl.Collator requires data that is not in the JSON export of the CLDR).
And hook it into ladybird.sh for convenience. The script will set up
PATH and other environment variables automatically.
On CI, vcpkg is theoretically already installed on Linux machines, but
not with the right environment variables, and not on macOS. So this also
makes CI use this script to bootstrap vcpkg.
We're now getting errors on CI due to gcc-13 being missing. We can
probably be smarter about what packages we install, depending on the
workflow being run. But let's first unblock CI.
The error we get is a bit strange and inconsistent. Some CI runners seem
to already have gcc-13 installed. Others don't and can't find the gcc-13
package without the test Ubuntu toolchain PPA.
Now both /bin/zcat and /bin/gunzip are symlinks to /bin/gzip, and we
essentially running it in decompression mode through these symlinks.
This ensures we don't maintain 2 versions of code to decompress Gzipped
data anymore, and handle the use case of gzipped-streaming input only
once in the codebase.
Static analysis is great, but these workflows have not worked in a long
time, and no one was looking at the results. Our PVS Studio license and
our Sonar Cloud token have expired. Remove the workflows (at least for
now) so we don't waste CI runners and cache space. If someone is
motivated to revive these, they can revert this commit.
On the macOS 14 runners on GitHub actions, attempting to play audio (by
way of AudioOutputUnitStart) will open a pop-up asking for microphone
permission. This prevents any calling test to hang until they error out
with MACH_SEND_TIMED_OUT. This works around the issue by explicitly
enabling microphone access to all applications.
Just calling it "cmake" doesn't really describe what it is for, and a
future commit will add a lagom.yml. So let's call this serenity.yml to
make it clear that it tests SerenityOS itself.
This also renames the workflow, and updates some matrix orderings, to be
easier to distinguish between this job and the upcoming lagom.yml,
Unfortunately a composite action cannot have a `post:` step like
JavaScript actions are allowed to have, so we need to explicitly call
the post/save actions ourselves from the workflow file when we want to
save Toolchain/QEMU/ccache caches.
Co-Authored-By: Timothy Flynn <trflynn89@pm.me>
If there is a cache miss while downloading the ccache from GitHub/Azure,
the .ccache directory won't exist when we try to update the modification
time of its contents. Configure the ccache size first, which will create
the .ccache directory if it doesn't exist.
The JIT compiler was an interesting experiment, but ultimately the
security & complexity cost of doing arbitrary code generation at runtime
is far too high.
In subsequent commits, the bytecode format will change drastically, and
instead of rewriting the JIT to fit the new bytecode, this patch simply
removes the JIT instead.
Other engines, JavaScriptCore in particular, have already proven that
it's possible to handle the vast majority of contemporary web content
with an interpreter. They are currently ~5x faster than us on benchmarks
when running without a JIT. We need to catch up to them before
considering performance techniques with a heavy security cost.
The newer version installs a specific version of octokit, since the
latest version no longer supports the ruby installation in github
actions vms. This should resolve the CI issues.
These tests almost always fail, and all we do is try to boot. Disable it
for now until it can pass more reliably. Note we still compile aarch64,
so the build shouldn't break unnoticed.
This change introduces a new 2D graphics library that uses OpenGL to
perform painting operations. For now, it has extremely limited
functionality and supports only rectangle painting, but we have to
start somewhere.
Since this library is intended to be used by LibWeb, where the
WebContent process does not have an associated window, painting occurs
in an offscreen buffer created using EGL.
For now it is only possible to compile this library on linux.
Offscreen context creation on SerenityOS and MacOS will have to be
implemented separately in the future.
Co-Authored-By: Andreas Kling <awesomekling@gmail.com>
This version contains my patch that adds support for the proprietary
VideoCore mailbox message for reading the kernel command line, so
patches aren't needed anymore.
This is taking over 4 minutes to run on CI, and checks only a dozen
manpages, which is of limited benefit.
Leaving the manpage-generation code in place for now.
The test runner script sets the `halt_on_error=1` `UBSAN_OPTIONS` flag
already, this just makes it a compile-time decision. This should
alleviate some of the slowness of running on-target tests without
hardware acceleration.
