LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
/*
|
2026-03-25 06:52:59 -03:00
|
|
|
|
* Copyright (c) 2026-present, the Ladybird developers.
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
*
|
2021-04-22 05:24:48 -03:00
|
|
|
|
* SPDX-License-Identifier: BSD-2-Clause
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
*/
|
|
|
|
|
|
|
LibRegex: Restore broad ECMA-262 test coverage
Port a broad slice of the old TestRegex.cpp coverage to the new
ECMAScriptRegex API so we keep exercising the Rust engine with the
same kinds of parser and matcher edge cases that used to live in the
legacy C++ suite.
Restore checks for parser acceptance and rejection, lookbehind
captures, quantified alternations, zero-width backreferences,
undefined backreferences, optional groups with empty matches,
modifier groups, Unicode properties and Unicode sets, empty-match
loops, and long fork chains.
Leave out the old Posix tests and bytecode-dump assertions that were
specific to the removed engine internals. One old boolean expectation
for duplicate named groups also did not translate cleanly to the new
search-style API, so keep the surrounding duplicate-name coverage in
capture-focused tests instead.
Verify the port with ./Build/sanitizers/bin/TestRegex.
2026-03-25 17:41:37 -03:00
|
|
|
|
#include <AK/Array.h>
|
2026-03-27 19:12:47 -03:00
|
|
|
|
#include <AK/StringBuilder.h>
|
LibRegex: Restore broad ECMA-262 test coverage
Port a broad slice of the old TestRegex.cpp coverage to the new
ECMAScriptRegex API so we keep exercising the Rust engine with the
same kinds of parser and matcher edge cases that used to live in the
legacy C++ suite.
Restore checks for parser acceptance and rejection, lookbehind
captures, quantified alternations, zero-width backreferences,
undefined backreferences, optional groups with empty matches,
modifier groups, Unicode properties and Unicode sets, empty-match
loops, and long fork chains.
Leave out the old Posix tests and bytecode-dump assertions that were
specific to the removed engine internals. One old boolean expectation
for duplicate named groups also did not translate cleanly to the new
search-style API, so keep the surrounding duplicate-name coverage in
capture-focused tests instead.
Verify the port with ./Build/sanitizers/bin/TestRegex.
2026-03-25 17:41:37 -03:00
|
|
|
|
#include <AK/Utf16String.h>
|
2026-03-25 06:52:59 -03:00
|
|
|
|
#include <LibTest/TestCase.h>
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
#include <LibRegex/ECMAScriptRegex.h>
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
|
2026-06-22 08:07:20 -03:00
|
|
|
|
static ErrorOr<regex::ECMAScriptRegex, String> compile_regex_result(Utf16View pattern, regex::ECMAScriptCompileFlags flags = {})
|
|
|
|
|
|
{
|
|
|
|
|
|
return regex::ECMAScriptRegex::compile(pattern, flags);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
static ErrorOr<regex::ECMAScriptRegex, String> compile_regex_result(StringView pattern, regex::ECMAScriptCompileFlags flags = {})
|
|
|
|
|
|
{
|
|
|
|
|
|
auto utf16_pattern = Utf16String::from_utf8(pattern);
|
|
|
|
|
|
return compile_regex_result(utf16_pattern.utf16_view(), flags);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
LibRegex: Restore broad ECMA-262 test coverage
Port a broad slice of the old TestRegex.cpp coverage to the new
ECMAScriptRegex API so we keep exercising the Rust engine with the
same kinds of parser and matcher edge cases that used to live in the
legacy C++ suite.
Restore checks for parser acceptance and rejection, lookbehind
captures, quantified alternations, zero-width backreferences,
undefined backreferences, optional groups with empty matches,
modifier groups, Unicode properties and Unicode sets, empty-match
loops, and long fork chains.
Leave out the old Posix tests and bytecode-dump assertions that were
specific to the removed engine internals. One old boolean expectation
for duplicate named groups also did not translate cleanly to the new
search-style API, so keep the surrounding duplicate-name coverage in
capture-focused tests instead.
Verify the port with ./Build/sanitizers/bin/TestRegex.
2026-03-25 17:41:37 -03:00
|
|
|
|
static regex::ECMAScriptRegex compile_regex(StringView pattern, regex::ECMAScriptCompileFlags flags = {})
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
return MUST(compile_regex_result(pattern, flags));
|
LibRegex: Restore broad ECMA-262 test coverage
Port a broad slice of the old TestRegex.cpp coverage to the new
ECMAScriptRegex API so we keep exercising the Rust engine with the
same kinds of parser and matcher edge cases that used to live in the
legacy C++ suite.
Restore checks for parser acceptance and rejection, lookbehind
captures, quantified alternations, zero-width backreferences,
undefined backreferences, optional groups with empty matches,
modifier groups, Unicode properties and Unicode sets, empty-match
loops, and long fork chains.
Leave out the old Posix tests and bytecode-dump assertions that were
specific to the removed engine internals. One old boolean expectation
for duplicate named groups also did not translate cleanly to the new
search-style API, so keep the surrounding duplicate-name coverage in
capture-focused tests instead.
Verify the port with ./Build/sanitizers/bin/TestRegex.
2026-03-25 17:41:37 -03:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
static bool compile_succeeds(StringView pattern, regex::ECMAScriptCompileFlags flags = {})
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
return !compile_regex_result(pattern, flags).is_error();
|
LibRegex: Restore broad ECMA-262 test coverage
Port a broad slice of the old TestRegex.cpp coverage to the new
ECMAScriptRegex API so we keep exercising the Rust engine with the
same kinds of parser and matcher edge cases that used to live in the
legacy C++ suite.
Restore checks for parser acceptance and rejection, lookbehind
captures, quantified alternations, zero-width backreferences,
undefined backreferences, optional groups with empty matches,
modifier groups, Unicode properties and Unicode sets, empty-match
loops, and long fork chains.
Leave out the old Posix tests and bytecode-dump assertions that were
specific to the removed engine internals. One old boolean expectation
for duplicate named groups also did not translate cleanly to the new
search-style API, so keep the surrounding duplicate-name coverage in
capture-focused tests instead.
Verify the port with ./Build/sanitizers/bin/TestRegex.
2026-03-25 17:41:37 -03:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
static bool matches(StringView pattern, StringView subject, regex::ECMAScriptCompileFlags flags = {})
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex(pattern, flags);
|
|
|
|
|
|
auto utf16_subject = Utf16String::from_utf8(subject);
|
|
|
|
|
|
auto result = regex.test(utf16_subject, 0);
|
|
|
|
|
|
EXPECT(result != regex::MatchResult::LimitExceeded);
|
|
|
|
|
|
return result == regex::MatchResult::Match;
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
static Optional<Utf16View> capture_group(regex::ECMAScriptRegex const& regex, Utf16View input, unsigned group_index)
|
|
|
|
|
|
{
|
|
|
|
|
|
auto start = regex.capture_slot(group_index * 2);
|
|
|
|
|
|
auto end = regex.capture_slot(group_index * 2 + 1);
|
|
|
|
|
|
if (start < 0 || end < 0)
|
|
|
|
|
|
return {};
|
|
|
|
|
|
return input.substring_view(start, end - start);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
static void expect_capture_eq(regex::ECMAScriptRegex const& regex, Utf16View input, unsigned group_index, StringView expected)
|
|
|
|
|
|
{
|
|
|
|
|
|
auto capture = capture_group(regex, input, group_index);
|
|
|
|
|
|
EXPECT(capture.has_value());
|
|
|
|
|
|
if (capture.has_value())
|
|
|
|
|
|
EXPECT(*capture == expected);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
static void expect_capture_unmatched(regex::ECMAScriptRegex const& regex, unsigned group_index)
|
|
|
|
|
|
{
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(group_index * 2), -1);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(group_index * 2 + 1), -1);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
TEST_CASE(compile_rejects_invalid_pattern)
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = compile_regex_result("("sv, {});
|
2026-03-25 06:52:59 -03:00
|
|
|
|
EXPECT(regex.is_error());
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
TEST_CASE(exec_tracks_named_capture_slots)
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("(?<word>foo)(bar)"sv, {}));
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
EXPECT_EQ(regex.capture_count(), 2u);
|
|
|
|
|
|
EXPECT_EQ(regex.total_groups(), 3u);
|
|
|
|
|
|
EXPECT_EQ(regex.named_groups().size(), 1u);
|
|
|
|
|
|
EXPECT_EQ(regex.named_groups()[0].name, "word"sv);
|
|
|
|
|
|
EXPECT_EQ(regex.named_groups()[0].index, 1u);
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
EXPECT_EQ(regex.exec(u"foobar"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(0), 0);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(1), 6);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(2), 0);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(3), 3);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(4), 3);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(5), 6);
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
TEST_CASE(exec_reports_unmatched_optional_groups)
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("(foo)?bar"sv, {}));
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
EXPECT_EQ(regex.exec(u"bar"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(0), 0);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(1), 3);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(2), -1);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(3), -1);
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-25 16:21:36 -03:00
|
|
|
|
TEST_CASE(ascii_backed_inputs_preserve_match_results)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("(?<word>foo)(bar)"sv, {}));
|
2026-03-25 16:21:36 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec("foobar"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(0), 0);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(1), 6);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(2), 0);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(3), 3);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(4), 3);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(5), 6);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.test("foobar"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all("foobar foobar"sv, 0), 2);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).start, 0);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).end, 6);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).start, 7);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).end, 13);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
TEST_CASE(test_honors_ignore_case)
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("casesensitive"sv, { .ignore_case = true }));
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
EXPECT_EQ(regex.test(u"CaseSensitive"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"something else"sv, 0), regex::MatchResult::NoMatch);
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-27 18:11:29 -03:00
|
|
|
|
TEST_CASE(ascii_ignore_case_literal_search_preserves_behavior)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("zfvr"sv, { .ignore_case = true }));
|
2026-03-27 18:11:29 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec("...ZFVR..."sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(0), 3);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(1), 7);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.find_all("zfvr ZFVR zFVr"sv, 0), 3);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).start, 0);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).start, 5);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(2).start, 10);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(ascii_ignore_case_literal_search_handles_punctuation_prefixes)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("##yv22##"sv, { .ignore_case = true }));
|
2026-03-27 18:11:29 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.find_all("##YV22## and ##yv22##"sv, 0), 2);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).start, 0);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).end, 8);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).start, 13);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).end, 21);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-27 18:18:21 -03:00
|
|
|
|
TEST_CASE(ascii_ignore_case_literal_alternation_preserves_behavior)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("##yv22##|zfvr|puebzr"sv, { .ignore_case = true }));
|
2026-03-27 18:18:21 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.find_all("##YV22## zFVr PUEBZR"sv, 0), 3);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).start, 0);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).end, 8);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).start, 9);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).end, 13);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(2).start, 14);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(2).end, 20);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(ascii_ignore_case_literal_alternation_respects_source_order)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("foo|f"sv, { .ignore_case = true }));
|
2026-03-27 18:18:21 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec("FoO"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(0), 0);
|
|
|
|
|
|
EXPECT_EQ(regex.capture_slot(1), 3);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(unicode_ignore_case_literal_alternation_preserves_behavior)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("s|k"sv, { .ignore_case = true, .unicode = true }));
|
2026-03-27 18:18:21 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"\u017F"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"\u212A"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-27 18:22:15 -03:00
|
|
|
|
TEST_CASE(word_boundary_literal_preserves_behavior)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("\\bfoo\\b"sv, {}));
|
2026-03-27 18:22:15 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.find_all("foo foo-bar barfoo foo2 _foo foo_"sv, 0), 2);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).start, 0);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).end, 3);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).start, 4);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).end, 7);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(ascii_ignore_case_word_boundary_literal_preserves_behavior)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("\\bzfvr\\b"sv, { .ignore_case = true }));
|
2026-03-27 18:22:15 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.find_all("ZFVR zfvr1 _ZFVR zFVr"sv, 0), 2);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).start, 0);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).end, 4);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).start, 17);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).end, 21);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(unicode_ignore_case_word_boundary_literal_preserves_behavior)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("\\bk\\b"sv, { .ignore_case = true, .unicode = true }));
|
2026-03-27 18:22:15 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"\u212A"sv, 0), regex::MatchResult::Match);
|
LibRegex: Inline \w and \d ranges in legacy positive char classes
Previously, a character class containing any builtin (\d, \w, \s)
forced the compiler down the slow "complex class" path, which emits
a disjunction of alternatives and backtracks at runtime.
For non-unicode, non-unicode-sets, non-negated classes, \w and \d
can be inlined as their raw ASCII code-point ranges. The resulting
class stays on the fast path and compiles into a single sorted
CharClass instruction.
The unicode/unicode_sets and negation guards are required for
correctness: with the /u + /i flags, \w gains non-ASCII members
via case folding (e.g. U+017F, U+212A), and negated classes have
a separate, smarter compilation path.
2026-04-21 10:22:00 -03:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(mixed_positive_class_with_word_builtin_preserves_legacy_ignore_case_behavior)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("[\\w\\$]+"sv, { .ignore_case = true }));
|
LibRegex: Inline \w and \d ranges in legacy positive char classes
Previously, a character class containing any builtin (\d, \w, \s)
forced the compiler down the slow "complex class" path, which emits
a disjunction of alternatives and backtracks at runtime.
For non-unicode, non-unicode-sets, non-negated classes, \w and \d
can be inlined as their raw ASCII code-point ranges. The resulting
class stays on the fast path and compiles into a single sorted
CharClass instruction.
The unicode/unicode_sets and negation guards are required for
correctness: with the /u + /i flags, \w gains non-ASCII members
via case folding (e.g. U+017F, U+212A), and negated classes have
a separate, smarter compilation path.
2026-04-21 10:22:00 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.test("AZ_09$"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"\u017F"sv, 0), regex::MatchResult::NoMatch);
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"\u212A"sv, 0), regex::MatchResult::NoMatch);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(mixed_positive_class_with_digit_builtin_preserves_behavior)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("[A-Z\\d-]+"sv, { .ignore_case = true }));
|
LibRegex: Inline \w and \d ranges in legacy positive char classes
Previously, a character class containing any builtin (\d, \w, \s)
forced the compiler down the slow "complex class" path, which emits
a disjunction of alternatives and backtracks at runtime.
For non-unicode, non-unicode-sets, non-negated classes, \w and \d
can be inlined as their raw ASCII code-point ranges. The resulting
class stays on the fast path and compiles into a single sorted
CharClass instruction.
The unicode/unicode_sets and negation guards are required for
correctness: with the /u + /i flags, \w gains non-ASCII members
via case folding (e.g. U+017F, U+212A), and negated classes have
a separate, smarter compilation path.
2026-04-21 10:22:00 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.test("ABC-123"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.test("abc"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.test("!"sv, 0), regex::MatchResult::NoMatch);
|
2026-03-27 18:22:15 -03:00
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
TEST_CASE(find_all_returns_non_overlapping_matches)
|
LibRegex: Add a regular expression library
This commit is a mix of several commits, squashed into one because the
commits before 'Move regex to own Library and fix all the broken stuff'
were not fixable in any elegant way.
The commits are listed below for "historical" purposes:
- AK: Add options/flags and Errors for regular expressions
Flags can be provided for any possible flavour by adding a new scoped enum.
Handling of flags is done by templated Options class and the overloaded
'|' and '&' operators.
- AK: Add Lexer for regular expressions
The lexer parses the input and extracts tokens needed to parse a regular
expression.
- AK: Add regex Parser and PosixExtendedParser
This patchset adds a abstract parser class that can be derived to implement
different parsers. A parser produces bytecode to be executed within the
regex matcher.
- AK: Add regex matcher
This patchset adds an regex matcher based on the principles of the T-REX VM.
The bytecode pruduced by the respective Parser is put into the matcher and
the VM will recursively execute the bytecode according to the available OpCodes.
Possible improvement: the recursion could be replaced by multi threading capabilities.
To match a Regular expression, e.g. for the Posix standard regular expression matcher
use the following API:
```
Pattern<PosixExtendedParser> pattern("^.*$");
auto result = pattern.match("Well, hello friends!\nHello World!"); // Match whole needle
EXPECT(result.count == 1);
EXPECT(result.matches.at(0).view.starts_with("Well"));
EXPECT(result.matches.at(0).view.end() == "!");
result = pattern.match("Well, hello friends!\nHello World!", PosixFlags::Multiline); // Match line by line
EXPECT(result.count == 2);
EXPECT(result.matches.at(0).view == "Well, hello friends!");
EXPECT(result.matches.at(1).view == "Hello World!");
EXPECT(pattern.has_match("Well,....")); // Just check if match without a result, which saves some resources.
```
- AK: Rework regex to work with opcodes objects
This patchsets reworks the matcher to work on a more structured base.
For that an abstract OpCode class and derived classes for the specific
OpCodes have been added. The respective opcode logic is contained in
each respective execute() method.
- AK: Add benchmark for regex
- AK: Some optimization in regex for runtime and memory
- LibRegex: Move regex to own Library and fix all the broken stuff
Now regex works again and grep utility is also in place for testing.
This commit also fixes the use of regex.h in C by making `regex_t`
an opaque (-ish) type, which makes its behaviour consistent between
C and C++ compilers.
Previously, <regex.h> would've blown C compilers up, and even if it
didn't, would've caused a leak in C code, and not in C++ code (due to
the existence of `OwnPtr` inside the struct).
To make this whole ordeal easier to deal with (for now), this pulls the
definitions of `reg*()` into LibRegex.
pros:
- The circular dependency between LibC and LibRegex is broken
- Eaiser to test (without accidentally pulling in the host's libc!)
cons:
- Using any of the regex.h functions will require the user to link -lregex
- The symbols will be missing from libc, which will be a big surprise
down the line (especially with shared libs).
Co-Authored-By: Ali Mohammad Pur <ali.mpfard@gmail.com>
2020-04-26 09:45:10 -03:00
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("aba"sv, {}));
|
2026-02-03 13:15:08 -03:00
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
EXPECT_EQ(regex.find_all(u"aba aba"sv, 0), 2);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).start, 0);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).end, 3);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).start, 4);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).end, 7);
|
2026-02-03 13:15:08 -03:00
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
TEST_CASE(unicode_property_matching_works)
|
2026-02-03 13:15:08 -03:00
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("\\p{ASCII}+"sv, { .unicode = true }));
|
2026-02-03 13:15:08 -03:00
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
EXPECT_EQ(regex.test(u"ASCII"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"😀"sv, 0), regex::MatchResult::NoMatch);
|
2026-02-03 13:15:08 -03:00
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
TEST_CASE(end_anchored_suffix_patterns_preserve_behavior)
|
2026-02-03 13:15:08 -03:00
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("(.*)\\/client-(.*)\\.js$"sv, {}));
|
2026-02-03 13:15:08 -03:00
|
|
|
|
|
2026-03-25 06:52:59 -03:00
|
|
|
|
EXPECT_EQ(regex.test(u"https://cdn.example.com/assets/client-main.js"sv, 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"<script src=\"/assets/client-main.js\"></script>"sv, 0), regex::MatchResult::NoMatch);
|
2026-02-03 13:15:08 -03:00
|
|
|
|
}
|
LibRegex: Restore broad ECMA-262 test coverage
Port a broad slice of the old TestRegex.cpp coverage to the new
ECMAScriptRegex API so we keep exercising the Rust engine with the
same kinds of parser and matcher edge cases that used to live in the
legacy C++ suite.
Restore checks for parser acceptance and rejection, lookbehind
captures, quantified alternations, zero-width backreferences,
undefined backreferences, optional groups with empty matches,
modifier groups, Unicode properties and Unicode sets, empty-match
loops, and long fork chains.
Leave out the old Posix tests and bytecode-dump assertions that were
specific to the removed engine internals. One old boolean expectation
for duplicate named groups also did not translate cleanly to the new
search-style API, so keep the surrounding duplicate-name coverage in
capture-focused tests instead.
Verify the port with ./Build/sanitizers/bin/TestRegex.
2026-03-25 17:41:37 -03:00
|
|
|
|
|
2026-03-27 17:55:36 -03:00
|
|
|
|
TEST_CASE(leading_start_or_separator_prefix_preserves_behavior)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("(?:^|;)\\s*foo=([^;]*)"sv, {}));
|
2026-03-27 17:55:36 -03:00
|
|
|
|
|
|
|
|
|
|
{
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("foo=bar"sv);
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "bar"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
{
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("a=1; foo=bar; baz=qux"sv);
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "bar"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"a=1; baz=qux"sv, 0), regex::MatchResult::NoMatch);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
2026-03-27 19:12:47 -03:00
|
|
|
|
TEST_CASE(required_literal_prefilter_preserves_assignment_extractors)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("(?:^|;)\\s*foo=([^;]*)"sv, {}));
|
2026-03-27 19:12:47 -03:00
|
|
|
|
|
|
|
|
|
|
{
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("a=1; bar=baz; foo=qux"sv);
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "qux"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"a=1; bar=baz; quux=7"sv, 0), regex::MatchResult::NoMatch);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(ascii_ignore_case_required_literal_prefilter_preserves_behavior)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("\\bfoo\\s*=\\s*([^;]*)"sv, { .ignore_case = true }));
|
2026-03-27 19:12:47 -03:00
|
|
|
|
|
|
|
|
|
|
{
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("FOO = Bar"sv);
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "Bar"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"bar = baz"sv, 0), regex::MatchResult::NoMatch);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(required_literal_prefilter_handles_common_substrings_across_alternatives)
|
|
|
|
|
|
{
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result("(\\$\\{name\\})|(\\$name\\b)"sv, {}));
|
2026-03-27 19:12:47 -03:00
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.find_all("${name} $name"sv, 0), 2);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).start, 0);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).end, 7);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).start, 8);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(1).end, 13);
|
|
|
|
|
|
EXPECT_EQ(regex.test(u"${other} $other"sv, 0), regex::MatchResult::NoMatch);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(required_literal_prefilter_compiles_large_exact_quantifiers)
|
|
|
|
|
|
{
|
|
|
|
|
|
StringBuilder pattern_builder;
|
|
|
|
|
|
pattern_builder.append("(?:ab){"sv);
|
|
|
|
|
|
pattern_builder.appendff("{}", 1'000'000);
|
|
|
|
|
|
pattern_builder.append("}"sv);
|
|
|
|
|
|
auto pattern = MUST(pattern_builder.to_string());
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT(compile_succeeds(pattern));
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(required_literal_prefilter_compiles_long_literal_alternations)
|
|
|
|
|
|
{
|
|
|
|
|
|
StringBuilder branch_builder;
|
|
|
|
|
|
branch_builder.append_repeated("a"sv, 1'024);
|
|
|
|
|
|
auto shared_prefix = MUST(branch_builder.to_string());
|
|
|
|
|
|
|
|
|
|
|
|
StringBuilder pattern_builder;
|
|
|
|
|
|
pattern_builder.append(shared_prefix);
|
|
|
|
|
|
pattern_builder.append("b|"sv);
|
|
|
|
|
|
pattern_builder.append(shared_prefix);
|
|
|
|
|
|
pattern_builder.append("c"sv);
|
|
|
|
|
|
auto pattern = MUST(pattern_builder.to_string());
|
|
|
|
|
|
|
2026-06-22 08:07:20 -03:00
|
|
|
|
auto regex = MUST(compile_regex_result(pattern, {}));
|
2026-03-27 19:12:47 -03:00
|
|
|
|
|
|
|
|
|
|
StringBuilder subject_builder;
|
|
|
|
|
|
subject_builder.append(shared_prefix);
|
|
|
|
|
|
subject_builder.append("c"sv);
|
|
|
|
|
|
auto matching_subject = MUST(subject_builder.to_string());
|
|
|
|
|
|
|
|
|
|
|
|
subject_builder.trim(1);
|
|
|
|
|
|
subject_builder.append("d"sv);
|
|
|
|
|
|
auto missing_subject = MUST(subject_builder.to_string());
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.test(Utf16String::from_utf8(matching_subject), 0), regex::MatchResult::Match);
|
|
|
|
|
|
EXPECT_EQ(regex.test(Utf16String::from_utf8(missing_subject), 0), regex::MatchResult::NoMatch);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
LibRegex: Restore broad ECMA-262 test coverage
Port a broad slice of the old TestRegex.cpp coverage to the new
ECMAScriptRegex API so we keep exercising the Rust engine with the
same kinds of parser and matcher edge cases that used to live in the
legacy C++ suite.
Restore checks for parser acceptance and rejection, lookbehind
captures, quantified alternations, zero-width backreferences,
undefined backreferences, optional groups with empty matches,
modifier groups, Unicode properties and Unicode sets, empty-match
loops, and long fork chains.
Leave out the old Posix tests and bytecode-dump assertions that were
specific to the removed engine internals. One old boolean expectation
for duplicate named groups also did not translate cleanly to the new
search-style API, so keep the surrounding duplicate-name coverage in
capture-focused tests instead.
Verify the port with ./Build/sanitizers/bin/TestRegex.
2026-03-25 17:41:37 -03:00
|
|
|
|
TEST_CASE(restored_ecmascript_parse_coverage)
|
|
|
|
|
|
{
|
|
|
|
|
|
struct Test {
|
|
|
|
|
|
StringView pattern;
|
|
|
|
|
|
bool should_compile { true };
|
|
|
|
|
|
regex::ECMAScriptCompileFlags flags {};
|
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
static constexpr Test tests[] {
|
|
|
|
|
|
{ "^hello.$"sv },
|
|
|
|
|
|
{ "\\x"sv },
|
|
|
|
|
|
{ "\\x1"sv },
|
|
|
|
|
|
{ "\\x1"sv, false, { .unicode = true } },
|
|
|
|
|
|
{ "\\x11"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "\\"sv, false },
|
|
|
|
|
|
{ "(?"sv, false },
|
|
|
|
|
|
{ "\\u1234"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "[\\u1234]"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "\\u1"sv, false, { .unicode = true } },
|
|
|
|
|
|
{ "[\\u1]"sv, false, { .unicode = true } },
|
|
|
|
|
|
{ "{1}"sv, false },
|
|
|
|
|
|
{ "{1,2}"sv, false },
|
|
|
|
|
|
{ "\\uxxxx"sv, false, { .unicode = true } },
|
|
|
|
|
|
{ "\\u{10ffff}"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "\\u{110000}"sv, false, { .unicode = true } },
|
|
|
|
|
|
{ "\\p{ASCII}"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "\\p{}"sv, false, { .unicode = true } },
|
|
|
|
|
|
{ "\\p{AsCiI}"sv, false, { .unicode = true } },
|
|
|
|
|
|
{ "(?<a>a)(?<a>b)"sv, false },
|
|
|
|
|
|
{ "(?:(?<x>a)|(?<y>a)(?<x>b))(?:(?<z>c)|(?<z>d))"sv },
|
|
|
|
|
|
{ "(?<1a>a)"sv, false },
|
|
|
|
|
|
{ "(?<$$_$$>a)"sv },
|
|
|
|
|
|
{ "(?<ÿ>a)"sv },
|
|
|
|
|
|
{ "(?<𝓑𝓻𝓸𝔀𝓷>a)"sv },
|
|
|
|
|
|
{ "(?ii:a)"sv, false },
|
|
|
|
|
|
{ "(?-:a)"sv, false },
|
|
|
|
|
|
{ "(?i)"sv, false },
|
|
|
|
|
|
{ "(?-i)"sv, false },
|
|
|
|
|
|
{ "["sv, false },
|
|
|
|
|
|
{ "[ -"sv, false },
|
|
|
|
|
|
{ "[[x[]]]"sv, true, { .unicode_sets = true } },
|
|
|
|
|
|
{ "[\\w--x]"sv, true, { .unicode_sets = true } },
|
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
for (auto const& test : tests)
|
|
|
|
|
|
EXPECT_EQ(compile_succeeds(test.pattern, test.flags), test.should_compile);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(restored_ecmascript_match_coverage)
|
|
|
|
|
|
{
|
|
|
|
|
|
struct Test {
|
|
|
|
|
|
StringView pattern;
|
|
|
|
|
|
StringView subject;
|
|
|
|
|
|
bool should_match { true };
|
|
|
|
|
|
regex::ECMAScriptCompileFlags flags {};
|
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
static constexpr Test tests[] {
|
|
|
|
|
|
{ "^hello.$"sv, "hello1"sv },
|
|
|
|
|
|
{ "^h{0,1}ello.$"sv, "ello1"sv },
|
|
|
|
|
|
{ "^hell\\x6f1$"sv, "hello1"sv },
|
|
|
|
|
|
{ "^hel(?<LO>l.)1$"sv, "hello1"sv },
|
|
|
|
|
|
{ "\\b.*\\b"sv, "hello1"sv },
|
|
|
|
|
|
{ "bar(?=f.)foo"sv, "barfoo"sv },
|
|
|
|
|
|
{ "bar(?=foo)bar"sv, "barbar"sv, false },
|
|
|
|
|
|
{ "bar(?!foo)bar"sv, "barbar"sv },
|
|
|
|
|
|
{ "bar(?!bar)bar"sv, "barbar"sv, false },
|
|
|
|
|
|
{ "bar.*(?<=foo)"sv, "barbar"sv, false },
|
|
|
|
|
|
{ "bar.*(?<!foo)"sv, "barbar"sv },
|
|
|
|
|
|
{ "(?:)"sv, ""sv },
|
|
|
|
|
|
{ "(?<=.{3})f"sv, "abcdef"sv },
|
|
|
|
|
|
{ "(?<=.{3})f"sv, "abc😀ef"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "a(?=.(?=c)|b)b"sv, "ab"sv },
|
|
|
|
|
|
{ "(?=)(?=\\d)"sv, "smart"sv, false },
|
|
|
|
|
|
{ "(?<!.*q.*?)(?<=h.*)THIS(?=.*!)"sv, "hey THIS does match!"sv },
|
|
|
|
|
|
{ "(.*a)?(x)"sv, "x"sv },
|
|
|
|
|
|
{ "^\\w*[\\u212A]"sv, "K"sv, true, { .ignore_case = true, .unicode = true } },
|
|
|
|
|
|
{ "^a*A\\d"sv, "aaaa5"sv, true, { .ignore_case = true } },
|
|
|
|
|
|
{ "^\\u{017f}*s$"sv, "ſs"sv, true, { .ignore_case = true, .unicode = true } },
|
|
|
|
|
|
{ "(a+)+b"sv, "aaaaaaaaaaaaaaaaaaaaaaaaa"sv, false },
|
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
for (auto const& test : tests)
|
|
|
|
|
|
EXPECT_EQ(matches(test.pattern, test.subject, test.flags), test.should_match);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(restored_lookbehind_capture_coverage)
|
|
|
|
|
|
{
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(?<=(a|cc))b"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("ccb"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "b"sv);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "cc"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("((?<=\\b)[d-f]{3})"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("abc def"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "def"sv);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "def"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(?<=(b+))c"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("abbbbbbc"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "c"sv);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "bbbbbb"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(?<=((?:b\\d{2})+))c"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("ab12b23b34c"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "c"sv);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "b12b23b34"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(restored_inversion_state_in_char_class_coverage)
|
|
|
|
|
|
{
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("[\\S\\s]"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("hello"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "h"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("[^\\S\\n]"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("\n"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::NoMatch);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("[^\\S]"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("\t"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "\t"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(restored_quantified_alternation_capture_coverage)
|
|
|
|
|
|
{
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("^(a|a?)+$"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("a"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "a"sv);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "a"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("^(a|a?)+$"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("aa"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "aa"sv);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "a"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(restored_zero_width_backreference_coverage)
|
|
|
|
|
|
{
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(a*)b\\1+"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("baaac"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "b"sv);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, ""sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(x)?\\1y"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("y"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "y"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 1);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(?!(y)y)(\\1)z"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("xyyz"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "z"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 1);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 2, ""sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(restored_backreference_to_undefined_capture_groups)
|
|
|
|
|
|
{
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(?:(?<x>a)|(?<x>b))\\k<x>"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("bb"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "bb"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 1);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 2, "b"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(?:(?:(?<x>a)|(?<x>b))\\k<x>){2}"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("aabb"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "aabb"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 1);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 2, "b"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(?:(?<x>a)|(?<x>b))\\k<x>"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("aa"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "aa"sv);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "a"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 2);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(.*?)a(?!(a+)b\\2c)\\2(.*)"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("baaabaac"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "baaabaac"sv);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "ba"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 2);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 3, "abaac"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("^(?:(?<a>x)|(?<a>y)|z)\\k<a>$"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("z"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "z"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 1);
|
|
|
|
|
|
expect_capture_unmatched(regex, 2);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("^(?:(?<a>x)|(?<a>y)|z){2}\\k<a>$"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("xz"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "xz"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 1);
|
|
|
|
|
|
expect_capture_unmatched(regex, 2);
|
|
|
|
|
|
}
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(restored_optional_groups_with_empty_matches)
|
|
|
|
|
|
{
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("^(.*)(.*)?$"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("a"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "a"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 2);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("()?"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8(""sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_unmatched(regex, 1);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(z)((a+)?(b+)?(c))*"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("zaacbbbcac"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 1, "z"sv);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 2, "ac"sv);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 3, "a"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 4);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 5, "c"sv);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(?:(?=(abc)))?a"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("abc"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "a"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 1);
|
|
|
|
|
|
}
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("^(?:(?=(abc))){0,1}a"sv);
|
|
|
|
|
|
auto subject = Utf16String::from_utf8("abc"sv);
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.exec(subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
expect_capture_eq(regex, subject, 0, "a"sv);
|
|
|
|
|
|
expect_capture_unmatched(regex, 1);
|
|
|
|
|
|
}
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(restored_ecmascript_modifier_coverage)
|
|
|
|
|
|
{
|
|
|
|
|
|
struct Test {
|
|
|
|
|
|
StringView pattern;
|
|
|
|
|
|
StringView subject;
|
|
|
|
|
|
bool should_match { true };
|
|
|
|
|
|
regex::ECMAScriptCompileFlags flags {};
|
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
static constexpr Test tests[] {
|
|
|
|
|
|
{ "a(?i:b)c"sv, "aBc"sv },
|
|
|
|
|
|
{ "a(?i:b)c"sv, "aBC"sv, false },
|
|
|
|
|
|
{ "a(?s:.)c"sv, "a\nc"sv },
|
|
|
|
|
|
{ "(?ims:a.b)"sv, "A\nB"sv },
|
|
|
|
|
|
{ "(?i:a(?-i:b)c)"sv, "AbC"sv },
|
|
|
|
|
|
{ "(?i:a(?-i:b)c)"sv, "ABC"sv, false },
|
|
|
|
|
|
{ "a(?-i:b)c"sv, "AbC"sv, true, { .ignore_case = true } },
|
|
|
|
|
|
{ "a(?-i:b)c"sv, "ABC"sv, false, { .ignore_case = true } },
|
|
|
|
|
|
{ "x.(?m:^a)"sv, "x\na"sv, true, { .dot_all = true } },
|
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
for (auto const& test : tests)
|
|
|
|
|
|
EXPECT_EQ(matches(test.pattern, test.subject, test.flags), test.should_match);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(restored_unicode_property_and_sets_coverage)
|
|
|
|
|
|
{
|
|
|
|
|
|
struct Test {
|
|
|
|
|
|
StringView pattern;
|
|
|
|
|
|
StringView subject;
|
|
|
|
|
|
bool should_match { true };
|
|
|
|
|
|
regex::ECMAScriptCompileFlags flags {};
|
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
static constexpr Test tests[] {
|
|
|
|
|
|
{ "\\p{ASCII}"sv, "a"sv, false },
|
|
|
|
|
|
{ "\\p{ASCII}"sv, "p{ASCII}"sv },
|
|
|
|
|
|
{ "\\p{ASCII}"sv, "a"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "\\p{ASCII}"sv, "😀"sv, false, { .unicode = true } },
|
|
|
|
|
|
{ "\\P{ASCII}"sv, "a"sv, false, { .unicode = true } },
|
|
|
|
|
|
{ "\\P{ASCII}"sv, "😀"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "\\p{ASCII_Hex_Digit}"sv, "1"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "\\P{ASCII_Hex_Digit}"sv, "x"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "\\p{General_Category=Cased_Letter}"sv, "A"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "\\P{Cased_Letter}"sv, "9"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "\\p{sc=Latin}"sv, "A"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "\\u{1f600}"sv, "😀"sv, true, { .unicode = true } },
|
|
|
|
|
|
{ "[\\w--x]"sv, "x"sv, false, { .unicode_sets = true } },
|
|
|
|
|
|
{ "[\\w--x]"sv, "y"sv, true, { .unicode_sets = true } },
|
|
|
|
|
|
{ "[\\w&&x]"sv, "x"sv, true, { .unicode_sets = true } },
|
|
|
|
|
|
{ "[[0-9\\w]--x--6]"sv, "6"sv, false, { .unicode_sets = true } },
|
|
|
|
|
|
{ "[[0-9\\w]--x--6]"sv, "9"sv, true, { .unicode_sets = true } },
|
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
for (auto const& test : tests)
|
|
|
|
|
|
EXPECT_EQ(matches(test.pattern, test.subject, test.flags), test.should_match);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(restored_empty_match_and_loop_coverage)
|
|
|
|
|
|
{
|
|
|
|
|
|
static constexpr StringView patterns[] {
|
|
|
|
|
|
"(a*)*"sv,
|
|
|
|
|
|
"(a*?)*"sv,
|
|
|
|
|
|
"(a*)*?"sv,
|
|
|
|
|
|
"(?:)*?"sv,
|
|
|
|
|
|
"(a?)+$"sv,
|
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
|
|
for (auto pattern : patterns)
|
|
|
|
|
|
EXPECT(matches(pattern, ""sv));
|
|
|
|
|
|
|
|
|
|
|
|
auto regex = compile_regex(".*"sv, { .global = true });
|
|
|
|
|
|
auto subject = Utf16String::from_utf8(""sv);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all(subject, 0), 1);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).start, 0);
|
|
|
|
|
|
EXPECT_EQ(regex.find_all_match(0).end, 0);
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
TEST_CASE(restored_long_fork_chain_coverage)
|
|
|
|
|
|
{
|
|
|
|
|
|
auto regex = compile_regex("(?:aa)*"sv);
|
|
|
|
|
|
auto subject = MUST(String::repeated('a', 1000));
|
|
|
|
|
|
auto utf16_subject = Utf16String::from_utf8(subject.bytes_as_string_view());
|
|
|
|
|
|
|
|
|
|
|
|
EXPECT_EQ(regex.test(utf16_subject, 0), regex::MatchResult::Match);
|
|
|
|
|
|
}
|