dmaax/ladybird
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ladybird/Libraries/LibRegex/RegexByteCode.cpp

525 lines
22 KiB
C++
Raw Normal View History

LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
/*
* Copyright (c) 2020, Emanuel Sprung <emanuel.sprung@gmail.com>
*
Everything: Move to SPDX license identifiers in all files. SPDX License Identifiers are a more compact / standardized way of representing file license information. See: https://spdx.dev/resources/use/#identifiers This was done with the `ambr` search and replace tool. ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
2021-04-22 05:24:48 -03:00
* SPDX-License-Identifier: BSD-2-Clause
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
*/
#include "RegexByteCode.h"
LibRegex: Account for uppercase characters in insensitive patterns
2025-07-11 10:06:40 -03:00
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 16:18:08 -03:00
#include <AK/CharacterTypes.h>
Everywhere: Fix badly-formatted includes In 7c5e30daaa615ad3a2ef55222423a747ac0a1227, the focus was "only" on Userland/Libraries/, whereas this commit cleans up the remaining headers in the repo, and any new badly-formatted include.
2023-01-02 11:15:14 -03:00
#include <AK/StringBuilder.h>
LibRegex+LibUnicode: Begin implementing Unicode property escapes This supports some binary property matching. It does not support any properties not yet parsed by LibUnicode, nor does it support value matching (such as Script_Extensions=Latin).
2021-07-29 15:18:51 -03:00
#include <LibUnicode/CharacterTypes.h>
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
namespace regex {
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 14:32:29 -03:00
StringView execution_result_name(ExecutionResult result)
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
{
switch (result) {
#define __ENUMERATE_EXECUTION_RESULT(x) \
case ExecutionResult::x: \
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 14:32:29 -03:00
return #x##sv;
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
ENUMERATE_EXECUTION_RESULTS
#undef __ENUMERATE_EXECUTION_RESULT
default:
Everywhere: Rename ASSERT => VERIFY (...and ASSERT_NOT_REACHED => VERIFY_NOT_REACHED) Since all of these checks are done in release builds as well, let's rename them to VERIFY to prevent confusion, as everyone is used to assertions being compiled out in release. We can introduce a new ASSERT macro that is specifically for debug checks, but I'm doing this wholesale conversion first since we've accumulated thousands of these already, and it's not immediately obvious which ones are suitable for ASSERT.
2021-02-23 16:42:32 -03:00
VERIFY_NOT_REACHED();
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 14:32:29 -03:00
return "<Unknown>"sv;
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
}
}
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 14:32:29 -03:00
StringView opcode_id_name(OpCodeId opcode)
LibRegex: Make infinite repetitions short-circuit on empty matches This makes (addmittedly weird) patterns like `(a*)*` work correctly without going into an infinite fork loop.
2021-09-05 19:08:47 -03:00
{
switch (opcode) {
#define __ENUMERATE_OPCODE(x) \
case OpCodeId::x: \
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 14:32:29 -03:00
return #x##sv;
LibRegex: Make infinite repetitions short-circuit on empty matches This makes (addmittedly weird) patterns like `(a*)*` work correctly without going into an infinite fork loop.
2021-09-05 19:08:47 -03:00
ENUMERATE_OPCODES
#undef __ENUMERATE_OPCODE
default:
VERIFY_NOT_REACHED();
Everywhere: Add sv suffix to strings relying on StringView(char const*) Each of these strings would previously rely on StringView's char const* constructor overload, which would call __builtin_strlen on the string. Since we now have operator ""sv, we can replace these with much simpler versions. This opens the door to being able to remove StringView(char const*). No functional changes.
2022-07-11 14:32:29 -03:00
return "<Unknown>"sv;
LibRegex: Make infinite repetitions short-circuit on empty matches This makes (addmittedly weird) patterns like `(a*)*` work correctly without going into an infinite fork loop.
2021-09-05 19:08:47 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
}
LibRegex: Support matching unicode multi-character sequences
2025-11-09 09:35:16 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
StringView fork_if_condition_name(ForkIfCondition condition)
{
switch (condition) {
#define __ENUMERATE_FORK_IF_CONDITION(x) \
case ForkIfCondition::x: \
return #x##sv;
ENUMERATE_FORK_IF_CONDITIONS
#undef __ENUMERATE_FORK_IF_CONDITION
default:
return "<Unknown>"sv;
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
}
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
StringView boundary_check_type_name(BoundaryCheckType ty)
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
{
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
switch (ty) {
#define __ENUMERATE_BOUNDARY_CHECK_TYPE(x) \
case BoundaryCheckType::x: \
return #x##sv;
ENUMERATE_BOUNDARY_CHECK_TYPES
#undef __ENUMERATE_BOUNDARY_CHECK_TYPE
default:
VERIFY_NOT_REACHED();
return "<Unknown>"sv;
LibRegex: Support non-ASCII case-insensitive character comparisons Specifically, when the Unicode flag is set, use Unicode-aware case folding to case-insensitively compare code points.
2023-11-08 12:30:55 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
}
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
StringView character_compare_type_name(CharacterCompareType ch_compare_type)
{
switch (ch_compare_type) {
#define __ENUMERATE_CHARACTER_COMPARE_TYPE(x) \
case CharacterCompareType::x: \
return #x##sv;
ENUMERATE_CHARACTER_COMPARE_TYPES
#undef __ENUMERATE_CHARACTER_COMPARE_TYPE
default:
VERIFY_NOT_REACHED();
return "<Unknown>"sv;
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
}
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
StringView character_class_name(CharClass ch_class)
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
{
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
switch (ch_class) {
#define __ENUMERATE_CHARACTER_CLASS(x) \
case CharClass::x: \
return #x##sv;
ENUMERATE_CHARACTER_CLASSES
#undef __ENUMERATE_CHARACTER_CLASS
default:
VERIFY_NOT_REACHED();
return "<Unknown>"sv;
LibRegex+Everywhere: Make LibRegex more unicode-aware This commit makes LibRegex (mostly) capable of operating on any of the three main string views: - StringView for raw strings - Utf8View for utf-8 encoded strings - Utf32View for raw unicode strings As a result, regexps with unicode strings should be able to properly handle utf-8 and not stop in the middle of a code point. A future commit will update LibJS to use the correct type of string depending on the flags.
2021-07-17 21:37:01 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
}
LibRegex+Everywhere: Make LibRegex more unicode-aware This commit makes LibRegex (mostly) capable of operating on any of the three main string views: - StringView for raw strings - Utf8View for utf-8 encoded strings - Utf32View for raw unicode strings As a result, regexps with unicode strings should be able to properly handle utf-8 and not stop in the middle of a code point. A future commit will update LibJS to use the correct type of string depending on the flags.
2021-07-17 21:37:01 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
static bool is_word_character(u32 code_point, bool case_insensitive, bool unicode_mode)
{
if (is_ascii_alphanumeric(code_point) || code_point == '_')
LibRegex: Partially implement string compare for Utf32View
2021-07-17 16:52:12 -03:00
return true;
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
if (case_insensitive && unicode_mode) {
auto canonical = Unicode::canonicalize(code_point, unicode_mode);
if (is_ascii_alphanumeric(canonical) || canonical == '_')
return true;
LibRegex: Avoid slicing a RegexStringView in non-unicode Compare ops Getting a single code point is much faster than slicing into the string.
2023-07-28 14:27:51 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
return false;
}
size_t ByteCode::s_next_checkpoint_serial_id { 0 };
u32 s_next_string_table_serial { 1 };
static u32 s_next_string_set_table_serial { 1 };
StringSetTable::StringSetTable()
: m_serial(s_next_string_set_table_serial++)
{
}
LibRegex: Partially implement string compare for Utf32View
2021-07-17 16:52:12 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
StringSetTable::~StringSetTable()
{
if (m_serial == s_next_string_set_table_serial - 1 && m_u8_tries.is_empty())
--s_next_string_set_table_serial;
}
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
StringSetTable::StringSetTable(StringSetTable const& other)
: m_serial(s_next_string_set_table_serial++)
{
for (auto const& entry : other.m_u8_tries)
m_u8_tries.set(entry.key, MUST(const_cast<StringSetTrie&>(entry.value).deep_copy()));
for (auto const& entry : other.m_u16_tries)
m_u16_tries.set(entry.key, MUST(const_cast<StringSetTrie&>(entry.value).deep_copy()));
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
StringSetTable& StringSetTable::operator=(StringSetTable const& other)
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
{
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
if (this != &other) {
m_u8_tries.clear();
m_u16_tries.clear();
for (auto const& entry : other.m_u8_tries)
m_u8_tries.set(entry.key, MUST(const_cast<StringSetTrie&>(entry.value).deep_copy()));
for (auto const& entry : other.m_u16_tries)
m_u16_tries.set(entry.key, MUST(const_cast<StringSetTrie&>(entry.value).deep_copy()));
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
return *this;
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
bool matches_character_class(CharClass character_class, u32 ch, bool insensitive, bool unicode_mode)
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
{
constexpr auto is_space_or_line_terminator = [](u32 code_point) {
LibRegex: Support non-ASCII whitespace characters when matching \s or \S ECMA-262 defines \s as: Return the CharSet containing all characters corresponding to a code point on the right-hand side of the WhiteSpace or LineTerminator productions. The LineTerminator production is simply: U+000A, U+000D, U+2028, or U+2029. Unfortunately there isn't a Unicode property that covers just those code points. The WhiteSpace production is: U+0009, U+000B, U+000C, U+FEFF, or any code point with the Space_Separator general category. If the Unicode generators are disabled, this will fall back to ASCII space code points.
2022-02-05 12:43:13 -03:00
if ((code_point == 0x0a) || (code_point == 0x0d) || (code_point == 0x2028) || (code_point == 0x2029))
return true;
if ((code_point == 0x09) || (code_point == 0x0b) || (code_point == 0x0c) || (code_point == 0xfeff))
return true;
LibUnicode: Replace code point general categories with ICU
2024-06-21 12:24:00 -03:00
return Unicode::code_point_has_space_separator_general_category(code_point);
LibRegex: Support non-ASCII whitespace characters when matching \s or \S ECMA-262 defines \s as: Return the CharSet containing all characters corresponding to a code point on the right-hand side of the WhiteSpace or LineTerminator productions. The LineTerminator production is simply: U+000A, U+000D, U+2028, or U+2029. Unfortunately there isn't a Unicode property that covers just those code points. The WhiteSpace production is: U+0009, U+000B, U+000C, U+FEFF, or any code point with the Space_Separator general category. If the Unicode generators are disabled, this will fall back to ASCII space code points.
2022-02-05 12:43:13 -03:00
};
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
switch (character_class) {
case CharClass::Alnum:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_ascii_alphanumeric(ch);
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
case CharClass::Alpha:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_ascii_alpha(ch);
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
case CharClass::Blank:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_ascii_blank(ch);
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
case CharClass::Cntrl:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_ascii_control(ch);
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
case CharClass::Digit:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_ascii_digit(ch);
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
case CharClass::Graph:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_ascii_graphical(ch);
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
case CharClass::Lower:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_ascii_lower_alpha(ch) || (insensitive && is_ascii_upper_alpha(ch));
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
case CharClass::Print:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_ascii_printable(ch);
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
case CharClass::Punct:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_ascii_punctuation(ch);
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
case CharClass::Space:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_space_or_line_terminator(ch);
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
case CharClass::Upper:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_ascii_upper_alpha(ch) || (insensitive && is_ascii_lower_alpha(ch));
LibRegex: Implement an ECMA262-compatible parser This also adds support for lookarounds and individually-negated comparisons. The only unimplemented part of the parser spec is the unicode stuff.
2020-11-27 13:03:53 -03:00
case CharClass::Word:
LibRegex: Implement Unicode case-insensitive matching Previously, case-insensitive regex matching used ASCII-only case conversion (to_ascii_lowercase) even for Unicode characters. Now we implement Canonicalize abstract operation, so we can case-fold Unicode characters properly during case-insensitive matching.
2026-02-15 11:08:52 -03:00
return is_word_character(ch, insensitive, unicode_mode);
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
case CharClass::Xdigit:
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
return is_ascii_hex_digit(ch);
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
}
LibRegex: Fully interpret the Compare Op when looking for overlaps We had a really naive and simplistic implementation, which lead to various issues where the optimiser incorrectly rewrote the regex to use atomic groups; this commit fixes that.
2022-07-04 13:41:15 -03:00
VERIFY_NOT_REACHED();
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
ByteString opcode_arguments_string(OpCodeId id, ByteCodeValueType const* data, size_t ip, MatchState const& state, ByteCodeBase const& bytecode)
{
// argument(N) = data[ip + 1 + N]
auto arg = [&](size_t n) -> ByteCodeValueType { return data[ip + 1 + n]; };
auto sz = opcode_size(id, data, ip);
switch (id) {
case OpCodeId::SaveModifiers:
return ByteString::formatted("new_modifiers={:#x}", arg(0));
case OpCodeId::RestoreModifiers:
case OpCodeId::Exit:
case OpCodeId::FailForks:
case OpCodeId::PopSaved:
case OpCodeId::Save:
case OpCodeId::Restore:
case OpCodeId::CheckBegin:
case OpCodeId::CheckEnd:
return ByteString::empty();
case OpCodeId::GoBack:
return ByteString::formatted("count={}", arg(0));
case OpCodeId::SetStepBack:
return ByteString::formatted("step={}", static_cast<i64>(arg(0)));
case OpCodeId::IncStepBack:
return ByteString::formatted("inc step back");
case OpCodeId::CheckStepBack:
return ByteString::formatted("check step back");
case OpCodeId::CheckSavedPosition:
return ByteString::formatted("check saved back");
case OpCodeId::Jump:
return ByteString::formatted("offset={} [&{}]", static_cast<ssize_t>(arg(0)), ip + sz + static_cast<ssize_t>(arg(0)));
case OpCodeId::ForkJump:
return ByteString::formatted("offset={} [&{}], sp: {}", static_cast<ssize_t>(arg(0)), ip + sz + static_cast<ssize_t>(arg(0)), state.string_position);
case OpCodeId::ForkReplaceJump:
return ByteString::formatted("offset={} [&{}], sp: {}", static_cast<ssize_t>(arg(0)), ip + sz + static_cast<ssize_t>(arg(0)), state.string_position);
case OpCodeId::ForkStay:
return ByteString::formatted("offset={} [&{}], sp: {}", static_cast<ssize_t>(arg(0)), ip + sz + static_cast<ssize_t>(arg(0)), state.string_position);
case OpCodeId::ForkReplaceStay:
return ByteString::formatted("offset={} [&{}], sp: {}", static_cast<ssize_t>(arg(0)), ip + sz + static_cast<ssize_t>(arg(0)), state.string_position);
case OpCodeId::CheckBoundary:
return ByteString::formatted("kind={} ({})", static_cast<unsigned long>(arg(0)), boundary_check_type_name(static_cast<BoundaryCheckType>(arg(0))));
case OpCodeId::ClearCaptureGroup:
case OpCodeId::SaveLeftCaptureGroup:
case OpCodeId::SaveRightCaptureGroup:
case OpCodeId::Checkpoint:
return ByteString::formatted("id={}", arg(0));
case OpCodeId::FailIfEmpty:
return ByteString::formatted("checkpoint={}", arg(0));
case OpCodeId::SaveRightNamedCaptureGroup:
return ByteString::formatted("name_id={}, id={}", arg(0), arg(1));
case OpCodeId::RSeekTo: {
auto ch = arg(0);
if (ch <= 0x7f)
return ByteString::formatted("before '{}'", ch);
return ByteString::formatted("before u+{:04x}", arg(0));
}
case OpCodeId::Compare:
return ByteString::formatted("argc={}, args={} ", arg(0), arg(1));
case OpCodeId::CompareSimple: {
StringBuilder builder;
auto type = static_cast<CharacterCompareType>(arg(1));
builder.append(character_compare_type_name(type));
switch (type) {
case CharacterCompareType::Char: {
auto ch = arg(2);
if (is_ascii_printable(ch))
builder.append(ByteString::formatted(" '{:c}'", static_cast<char>(ch)));
LibRegex: Implement Unicode case-insensitive matching Previously, case-insensitive regex matching used ASCII-only case conversion (to_ascii_lowercase) even for Unicode characters. Now we implement Canonicalize abstract operation, so we can case-fold Unicode characters properly during case-insensitive matching.
2026-02-15 11:08:52 -03:00
else
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
builder.append(ByteString::formatted(" 0x{:x}", ch));
break;
LibRegex: Implement Unicode case-insensitive matching Previously, case-insensitive regex matching used ASCII-only case conversion (to_ascii_lowercase) even for Unicode characters. Now we implement Canonicalize abstract operation, so we can case-fold Unicode characters properly during case-insensitive matching.
2026-02-15 11:08:52 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
case CharacterCompareType::String: {
auto string_index = arg(2);
auto string = bytecode.get_u16_string(string_index);
builder.appendff(" \"{}\"", string);
break;
LibRegex: Implement Unicode case-insensitive matching Previously, case-insensitive regex matching used ASCII-only case conversion (to_ascii_lowercase) even for Unicode characters. Now we implement Canonicalize abstract operation, so we can case-fold Unicode characters properly during case-insensitive matching.
2026-02-15 11:08:52 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
case CharacterCompareType::CharClass: {
auto character_class = static_cast<CharClass>(arg(2));
builder.appendff(" {}", character_class_name(character_class));
break;
LibRegex: Implement Unicode case-insensitive matching Previously, case-insensitive regex matching used ASCII-only case conversion (to_ascii_lowercase) even for Unicode characters. Now we implement Canonicalize abstract operation, so we can case-fold Unicode characters properly during case-insensitive matching.
2026-02-15 11:08:52 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
case CharacterCompareType::Reference: {
auto ref = arg(2);
builder.appendff(" number={}", ref);
break;
LibRegex: Add a basic optimization pass This currently tries to convert forking loops to atomic groups, and unify the left side of alternations.
2021-09-12 10:00:27 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
case CharacterCompareType::NamedReference: {
auto ref = arg(2);
builder.appendff(" named_number={}", ref);
break;
}
case CharacterCompareType::GeneralCategory:
case CharacterCompareType::Property:
case CharacterCompareType::Script:
case CharacterCompareType::ScriptExtension:
case CharacterCompareType::StringSet: {
builder.appendff(" value={}", arg(2));
break;
LibRegex: Skip multi-op compare overhead when not necessary
2025-12-22 10:00:19 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
case CharacterCompareType::LookupTable: {
auto count_sensitive = arg(2);
auto count_insensitive = arg(3);
for (size_t j = 0; j < count_sensitive; ++j) {
auto range = static_cast<CharRange>(arg(4 + j));
builder.appendff(" {:x}-{:x}", range.from, range.to);
}
if (count_insensitive > 0) {
builder.append(" [insensitive ranges:"sv);
for (size_t j = 0; j < count_insensitive; ++j) {
auto range = static_cast<CharRange>(arg(4 + count_sensitive + j));
builder.appendff(" {:x}-{:x}", range.from, range.to);
}
builder.append(" ]"sv);
LibRegex: Skip multi-op compare overhead when not necessary
2025-12-22 10:00:19 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
break;
LibRegex: Skip multi-op compare overhead when not necessary
2025-12-22 10:00:19 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
case CharacterCompareType::CharRange: {
auto value = arg(2);
auto range = static_cast<CharRange>(value);
builder.appendff(" {:x}-{:x}", range.from, range.to);
break;
}
default:
break;
}
return builder.to_byte_string();
}
case OpCodeId::Repeat: {
auto repeat_id = arg(2);
auto reps = repeat_id < state.repetition_marks.size() ? state.repetition_marks.at(repeat_id) : 0;
return ByteString::formatted("offset={} [&{}] count={} id={} rep={}, sp: {}",
static_cast<ssize_t>(arg(0)),
ip - arg(0),
arg(1) + 1,
repeat_id,
reps + 1,
state.string_position);
}
case OpCodeId::ResetRepeat: {
auto repeat_id = arg(0);
auto reps = repeat_id < state.repetition_marks.size() ? state.repetition_marks.at(repeat_id) : 0;
return ByteString::formatted("id={} rep={}", repeat_id, reps + 1);
}
case OpCodeId::JumpNonEmpty:
return ByteString::formatted("{} offset={} [&{}], cp={}",
opcode_id_name(static_cast<OpCodeId>(arg(2))),
static_cast<ssize_t>(arg(0)), ip + sz + static_cast<ssize_t>(arg(0)),
arg(1));
case OpCodeId::ForkIf:
return ByteString::formatted("{} {} offset={} [&{}]",
opcode_id_name(static_cast<OpCodeId>(arg(1))),
fork_if_condition_name(static_cast<ForkIfCondition>(arg(2))),
static_cast<ssize_t>(arg(0)), ip + sz + static_cast<ssize_t>(arg(0)));
LibRegex: Skip multi-op compare overhead when not necessary
2025-12-22 10:00:19 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
VERIFY_NOT_REACHED();
LibRegex: Skip multi-op compare overhead when not necessary
2025-12-22 10:00:19 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
Vector<ByteString> compare_variable_arguments_to_byte_string(ByteCodeValueType const* data, size_t ip, MatchState const& state, ByteCodeBase const& bytecode, Optional<MatchInput const&> input)
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
{
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
Vector<ByteString> result;
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
size_t offset = ip + 3;
LibRegex: Simplify ternary condition in RegexByteCode No functional changes.
2025-07-21 09:54:36 -03:00
RegexStringView const& view = input.has_value() ? input.value().view : StringView {};
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto argument_count = data[ip + 1]; // arguments_count for Compare
LibRegex: Cache pointer to flattened bytecode data in OpCode_Compare This avoids repeatedly checking if the bytecode has been flattened (which is always the case by the time we're executing). 1.05x speedup on Octane/regexp.js
2025-12-13 13:53:16 -03:00
for (size_t i = 0; i < argument_count; ++i) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto compare_type = static_cast<CharacterCompareType>(data[offset++]);
result.empend(ByteString::formatted("type={} [{}]", static_cast<size_t>(compare_type), character_compare_type_name(compare_type)));
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto string_start_offset = state.string_position_before_match;
LibRegex: Fix OOB access in Regex debug print
2020-11-28 17:03:45 -03:00
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
if (compare_type == CharacterCompareType::Char) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto ch = data[offset++];
Everywhere: Replace ctype.h to avoid narrowing conversions This replaces ctype.h with CharacterType.h everywhere I could find issues with narrowing conversions. While using it will probably make sense almost everywhere in the future, the most critical places should have been addressed.
2021-06-01 16:18:08 -03:00
auto is_ascii = is_ascii_printable(ch);
LibRegex: Implement section B.1.4. of the ECMA262 spec This allows the parser to deal with crazy patterns like the one in #5517.
2021-02-26 16:19:34 -03:00
if (is_ascii)
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(" value='{:c}'", static_cast<char>(ch)));
LibRegex: Implement section B.1.4. of the ECMA262 spec This allows the parser to deal with crazy patterns like the one in #5517.
2021-02-26 16:19:34 -03:00
else
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(" value={:x}", ch));
LibRegex: Implement section B.1.4. of the ECMA262 spec This allows the parser to deal with crazy patterns like the one in #5517.
2021-02-26 16:19:34 -03:00
LibRegex: Display correct position for Compare in REGEX_DEBUG When REGEX_DEBUG is enabled, LibRegex dumps a table of information regarding the state of the regex bytecode execution. The Compare opcode manipulates state.string_position directly, so the string_position value cannot be used to display where the comparison started; therefore, this patch introduces a new variable to keep track of where we were before the comparison happened.
2021-06-16 07:14:12 -03:00
if (!view.is_null() && view.length() > string_start_offset) {
LibRegex: Implement section B.1.4. of the ECMA262 spec This allows the parser to deal with crazy patterns like the one in #5517.
2021-02-26 16:19:34 -03:00
if (is_ascii) {
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(
LibRegex: Partially implement the ECMAScript unicodeSets proposal This skips the new string unicode properties additions, along with \q{}.
2022-07-20 15:52:07 -03:00
" compare against: '{}'",
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
view.substring_view(string_start_offset, string_start_offset > view.length() ? 0 : 1).to_byte_string()));
LibRegex: Implement section B.1.4. of the ECMA262 spec This allows the parser to deal with crazy patterns like the one in #5517.
2021-02-26 16:19:34 -03:00
} else {
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
auto str = view.substring_view(string_start_offset, string_start_offset > view.length() ? 0 : 1).to_byte_string();
LibRegex: Implement section B.1.4. of the ECMA262 spec This allows the parser to deal with crazy patterns like the one in #5517.
2021-02-26 16:19:34 -03:00
u8 buf[8] { 0 };
__builtin_memcpy(buf, str.characters(), min(str.length(), sizeof(buf)));
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(" compare against: {:x},{:x},{:x},{:x},{:x},{:x},{:x},{:x}",
LibRegex: Implement section B.1.4. of the ECMA262 spec This allows the parser to deal with crazy patterns like the one in #5517.
2021-02-26 16:19:34 -03:00
buf[0], buf[1], buf[2], buf[3], buf[4], buf[5], buf[6], buf[7]));
}
}
LibRegex: Implement an ECMA262-compatible parser This also adds support for lookarounds and individually-negated comparisons. The only unimplemented part of the parser spec is the unicode stuff.
2020-11-27 13:03:53 -03:00
} else if (compare_type == CharacterCompareType::Reference) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto ref = data[offset++];
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(" number={}", ref));
LibRegex: Add some more information to Compare::Reference debug output
2022-01-21 17:14:50 -03:00
if (input.has_value()) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
if (state.capture_group_matches_size() > input->match_index) {
auto match = state.capture_group_matches(input->match_index);
LibRegex: Add some more information to Compare::Reference debug output
2022-01-21 17:14:50 -03:00
if (match.size() > ref) {
auto& group = match[ref];
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(" left={}", group.left_column));
result.empend(ByteString::formatted(" right={}", group.left_column + group.view.length_in_code_units()));
result.empend(ByteString::formatted(" contents='{}'", group.view));
LibRegex: Add some more information to Compare::Reference debug output
2022-01-21 17:14:50 -03:00
} else {
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(" (invalid ref, max={})", match.size() - 1));
LibRegex: Add some more information to Compare::Reference debug output
2022-01-21 17:14:50 -03:00
}
} else {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
result.empend(ByteString::formatted(" (invalid index {}, max={})", input->match_index, state.capture_group_matches_size() - 1));
LibRegex: Fix backreferences to undefined capture groups Fixes handling of backreferences when the referenced capture group is undefined or hasn't participated in the match. CharacterCompareType::NamedReference is added to distinguish numbered (\1) from named (\k<name>) backreferences. Numbered backreferences use exact group lookup. Named backreferences search for participating groups among duplicates.
2025-07-23 15:48:34 -03:00
}
}
} else if (compare_type == CharacterCompareType::NamedReference) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto ref = data[offset++];
LibRegex: Fix backreferences to undefined capture groups Fixes handling of backreferences when the referenced capture group is undefined or hasn't participated in the match. CharacterCompareType::NamedReference is added to distinguish numbered (\1) from named (\k<name>) backreferences. Numbered backreferences use exact group lookup. Named backreferences search for participating groups among duplicates.
2025-07-23 15:48:34 -03:00
result.empend(ByteString::formatted(" named_number={}", ref));
if (input.has_value()) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
if (state.capture_group_matches_size() > input->match_index) {
auto match = state.capture_group_matches(input->match_index);
LibRegex: Fix backreferences to undefined capture groups Fixes handling of backreferences when the referenced capture group is undefined or hasn't participated in the match. CharacterCompareType::NamedReference is added to distinguish numbered (\1) from named (\k<name>) backreferences. Numbered backreferences use exact group lookup. Named backreferences search for participating groups among duplicates.
2025-07-23 15:48:34 -03:00
if (match.size() > ref) {
auto& group = match[ref];
result.empend(ByteString::formatted(" left={}", group.left_column));
result.empend(ByteString::formatted(" right={}", group.left_column + group.view.length_in_code_units()));
result.empend(ByteString::formatted(" contents='{}'", group.view));
} else {
result.empend(ByteString::formatted(" (invalid ref {}, max={})", ref, match.size() - 1));
}
} else {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
result.empend(ByteString::formatted(" (invalid index {}, max={})", input->match_index, state.capture_group_matches_size() - 1));
LibRegex: Add some more information to Compare::Reference debug output
2022-01-21 17:14:50 -03:00
}
}
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
} else if (compare_type == CharacterCompareType::String) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto str_id = data[offset++];
auto string = bytecode.get_u16_string(str_id);
LibRegex: Fuse consecutive single-char Compares into a String Compare This avoids huge instruction decoding and dispatch overhead, ~40x performance improvement for /(^|x)ppp/.
2025-12-15 20:33:28 -03:00
result.empend(ByteString::formatted(" value=\"{}\"", string));
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
if (!view.is_null() && view.length() > state.string_position)
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(
LibRegex: Partially implement the ECMAScript unicodeSets proposal This skips the new string unicode properties additions, along with \q{}.
2022-07-20 15:52:07 -03:00
" compare against: \"{}\"",
LibRegex: Fuse consecutive single-char Compares into a String Compare This avoids huge instruction decoding and dispatch overhead, ~40x performance improvement for /(^|x)ppp/.
2025-12-15 20:33:28 -03:00
input.value().view.substring_view(string_start_offset, string_start_offset + string.length_in_code_units() > view.length() ? 0 : string.length_in_code_units()).to_byte_string()));
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
} else if (compare_type == CharacterCompareType::CharClass) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto character_class = static_cast<CharClass>(data[offset++]);
result.empend(ByteString::formatted(" ch_class={} [{}]", static_cast<size_t>(character_class), character_class_name(character_class)));
if (!view.is_null() && view.length() > state.string_position)
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(
LibRegex: Partially implement the ECMAScript unicodeSets proposal This skips the new string unicode properties additions, along with \q{}.
2022-07-20 15:52:07 -03:00
" compare against: '{}'",
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
input.value().view.substring_view(string_start_offset, state.string_position > view.length() ? 0 : 1).to_byte_string()));
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
} else if (compare_type == CharacterCompareType::CharRange) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto value = static_cast<CharRange>(data[offset++]);
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(" ch_range={:x}-{:x}", value.from, value.to));
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
if (!view.is_null() && view.length() > state.string_position)
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(
LibRegex: Partially implement the ECMAScript unicodeSets proposal This skips the new string unicode properties additions, along with \q{}.
2022-07-20 15:52:07 -03:00
" compare against: '{}'",
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
input.value().view.substring_view(string_start_offset, state.string_position > view.length() ? 0 : 1).to_byte_string()));
LibRegex: Use a match table for character classes Generate a sorted, compressed series of ranges in a match table for character classes, and use a binary search to find the matches. This is about a 3-4x speedup for character class match performance. :^)
2021-10-03 12:31:25 -03:00
} else if (compare_type == CharacterCompareType::LookupTable) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto count_sensitive = data[offset++];
auto count_insensitive = data[offset++];
LibRegex: Account for uppercase characters in insensitive patterns
2025-07-11 10:06:40 -03:00
for (size_t j = 0; j < count_sensitive; ++j) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto range = static_cast<CharRange>(data[offset++]);
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.append(ByteString::formatted(" {:x}-{:x}", range.from, range.to));
LibRegex: Use a match table for character classes Generate a sorted, compressed series of ranges in a match table for character classes, and use a binary search to find the matches. This is about a 3-4x speedup for character class match performance. :^)
2021-10-03 12:31:25 -03:00
}
LibRegex: Account for uppercase characters in insensitive patterns
2025-07-11 10:06:40 -03:00
if (count_insensitive > 0) {
result.append(" [insensitive ranges:");
for (size_t j = 0; j < count_insensitive; ++j) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto range = static_cast<CharRange>(data[offset++]);
LibRegex: Account for uppercase characters in insensitive patterns
2025-07-11 10:06:40 -03:00
result.append(ByteString::formatted(" {:x}-{:x}", range.from, range.to));
}
result.append(" ]");
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
if (!view.is_null() && view.length() > state.string_position)
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(
LibRegex: Partially implement the ECMAScript unicodeSets proposal This skips the new string unicode properties additions, along with \q{}.
2022-07-20 15:52:07 -03:00
" compare against: '{}'",
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
input.value().view.substring_view(string_start_offset, state.string_position > view.length() ? 0 : 1).to_byte_string()));
LibRegex: Partially implement the ECMAScript unicodeSets proposal This skips the new string unicode properties additions, along with \q{}.
2022-07-20 15:52:07 -03:00
} else if (compare_type == CharacterCompareType::GeneralCategory
|| compare_type == CharacterCompareType::Property
|| compare_type == CharacterCompareType::Script
LibRegex: Add missing StringSet cases
2025-11-27 07:42:17 -03:00
|| compare_type == CharacterCompareType::ScriptExtension
|| compare_type == CharacterCompareType::StringSet) {
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
auto value = data[offset++];
Everywhere: Rename {Deprecated => Byte}String This commit un-deprecates DeprecatedString, and repurposes it as a byte string. As the null state has already been removed, there are no other particularly hairy blockers in repurposing this type as a byte string (what it _really_ is). This commit is auto-generated: $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \ Meta Ports Ladybird Tests Kernel) $ perl -pie 's/\bDeprecatedString\b/ByteString/g; s/deprecated_string/byte_string/g' $xs $ clang-format --style=file -i \ $(git diff --name-only | grep \.cpp\|\.h) $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-16 11:19:34 -03:00
result.empend(ByteString::formatted(" value={}", value));
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
}
}
return result;
}
LibRegex: Implement and use a REPEAT operation for bytecode repetition Currently, when we need to repeat an instruction N times, we simply add that instruction N times in a for-loop. This doesn't scale well with extremely large values of N, and ECMA-262 allows up to N = 2^53 - 1. Instead, add a new REPEAT bytecode operation to defer this loop from the parser to the runtime executor. This allows the parser to complete sans any loops (for this instruction), and allows the executor to bail early if the repeated bytecode fails. Note: The templated ByteCode methods are to allow the Posix parsers to continue using u32 because they are limited to N = 2^20.
2021-08-12 12:02:46 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
Vector<CompareTypeAndValuePair> flat_compares_at(ByteCodeValueType const* data, size_t ip, bool is_simple)
LibRegex: Make infinite repetitions short-circuit on empty matches This makes (addmittedly weird) patterns like `(a*)*` work correctly without going into an infinite fork loop.
2021-09-05 19:08:47 -03:00
{
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
Vector<CompareTypeAndValuePair> result;
LibRegex: Make infinite repetitions short-circuit on empty matches This makes (addmittedly weird) patterns like `(a*)*` work correctly without going into an infinite fork loop.
2021-09-05 19:08:47 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
size_t offset = ip + (is_simple ? 2 : 3);
auto argument_count = is_simple ? 1 : data[ip + OpArgs::Compare::arguments_count];
LibRegex: Add a basic optimization pass This currently tries to convert forking loops to atomic groups, and unify the left side of alternations.
2021-09-12 10:00:27 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
for (size_t i = 0; i < argument_count; ++i) {
auto compare_type = (CharacterCompareType)data[offset++];
LibRegex: Add a basic optimization pass This currently tries to convert forking loops to atomic groups, and unify the left side of alternations.
2021-09-12 10:00:27 -03:00
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
if (compare_type == CharacterCompareType::Char) {
auto ch = data[offset++];
result.append({ compare_type, ch });
} else if (compare_type == CharacterCompareType::Reference) {
auto ref = data[offset++];
result.append({ compare_type, ref });
} else if (compare_type == CharacterCompareType::NamedReference) {
auto ref = data[offset++];
result.append({ compare_type, ref });
} else if (compare_type == CharacterCompareType::String) {
auto string_index = data[offset++];
result.append({ compare_type, string_index });
} else if (compare_type == CharacterCompareType::CharClass) {
auto character_class = data[offset++];
result.append({ compare_type, character_class });
} else if (compare_type == CharacterCompareType::CharRange) {
auto value = data[offset++];
result.append({ compare_type, value });
} else if (compare_type == CharacterCompareType::LookupTable) {
auto count_sensitive = data[offset++];
auto count_insensitive = data[offset++];
for (size_t j = 0; j < count_sensitive; ++j)
result.append({ CharacterCompareType::CharRange, data[offset++] });
offset += count_insensitive; // Skip insensitive ranges
} else if (compare_type == CharacterCompareType::GeneralCategory
|| compare_type == CharacterCompareType::Property
|| compare_type == CharacterCompareType::Script
|| compare_type == CharacterCompareType::ScriptExtension
|| compare_type == CharacterCompareType::StringSet) {
auto value = data[offset++];
result.append({ compare_type, value });
} else {
result.append({ compare_type, 0 });
LibRegex: Add a basic optimization pass This currently tries to convert forking loops to atomic groups, and unify the left side of alternations.
2021-09-12 10:00:27 -03:00
}
LibRegex: Make infinite repetitions short-circuit on empty matches This makes (addmittedly weird) patterns like `(a*)*` work correctly without going into an infinite fork loop.
2021-09-05 19:08:47 -03:00
}
LibRegex: Replace virtual-inheritance dispatch with a switch interpreter
2026-03-17 14:05:08 -03:00
return result;
LibRegex: Make infinite repetitions short-circuit on empty matches This makes (addmittedly weird) patterns like `(a*)*` work correctly without going into an infinite fork loop.
2021-09-05 19:08:47 -03:00
}
LibRegex: Add a regular expression library This commit is a mix of several commits, squashed into one because the commits before 'Move regex to own Library and fix all the broken stuff' were not fixable in any elegant way. The commits are listed below for "historical" purposes: - AK: Add options/flags and Errors for regular expressions Flags can be provided for any possible flavour by adding a new scoped enum. Handling of flags is done by templated Options class and the overloaded '|' and '&' operators. - AK: Add Lexer for regular expressions The lexer parses the input and extracts tokens needed to parse a regular expression. - AK: Add regex Parser and PosixExtendedParser This patchset adds a abstract parser class that can be derived to implement different parsers. A parser produces bytecode to be executed within the regex matcher. - AK: Add regex matcher This patchset adds an regex matcher based on the principles of the T-REX VM. The bytecode pruduced by the respective Parser is put into the matcher and the VM will recursively execute the bytecode according to the available OpCodes. Possible improvement: the recursion could be replaced by multi threading capabilities. To match a Regular expression, e.g. for the Posix standard regular expression matcher use the following API: ``` Pattern<PosixExtendedParser> pattern("^.*$"); auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle EXPECT(result.count == 1); EXPECT(result.matches.at(0).view.starts_with("Well")); EXPECT(result.matches.at(0).view.end() == "!"); result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line EXPECT(result.count == 2); EXPECT(result.matches.at(0).view == "Well, hello friends!"); EXPECT(result.matches.at(1).view == "Hello World!"); EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources. ``` - AK: Rework regex to work with opcodes objects This patchsets reworks the matcher to work on a more structured base. For that an abstract OpCode class and derived classes for the specific OpCodes have been added. The respective opcode logic is contained in each respective execute() method. - AK: Add benchmark for regex - AK: Some optimization in regex for runtime and memory - LibRegex: Move regex to own Library and fix all the broken stuff Now regex works again and grep utility is also in place for testing. This commit also fixes the use of regex.h in C by making `regex_t` an opaque (-ish) type, which makes its behaviour consistent between C and C++ compilers. Previously, <regex.h> would've blown C compilers up, and even if it didn't, would've caused a leak in C code, and not in C++ code (due to the existence of `OwnPtr` inside the struct). To make this whole ordeal easier to deal with (for now), this pulls the definitions of `reg*()` into LibRegex. pros: - The circular dependency between LibC and LibRegex is broken - Eaiser to test (without accidentally pulling in the host's libc!) cons: - Using any of the regex.h functions will require the user to link -lregex - The symbols will be missing from libc, which will be a big surprise down the line (especially with shared libs). Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
}
Reference in a new issue Copy permalink