Understanding how a regex engine processes patterns can significantly improve your ability to write efficient and accurate regular expressions. By learning the internal mechanics, you’ll be better equipped to troubleshoot and refine your regex patterns, reducing frustration and guesswork when tackling complex tasks.
Types of Regex Engines
There are two primary types of regex engines:
Text-Directed Engines (also known as DFA - Deterministic Finite Automaton) Regex-Directed Engines (also known as NFA - Non-Deterministic Finite Automaton) All the regex flavors discussed in this tutorial utilize regex-directed engines. This type is more popular because it supports features like lazy quantifiers and backreferences, which are not possible in text-directed engines.
Examples of Text-Directed Engines:
awk egrep flex lex MySQL Procmail Note: Some versions of awk and egrep use regex-directed engines.
How to Identify the Engine Type
To determine whether a regex engine is text-directed or regex-directed, you can apply a simple test using the pattern:
«regex|regex not» Apply this pattern to the string "regex not":
If the result is "regex", the engine is regex-directed. If the result is "regex not", the engine is text-directed. The difference lies in how eager the engine is to find matches. A regex-directed engine is eager and will report the leftmost match, even if a better match exists later in the string.
The Regex-Directed Engine Always Returns the Leftmost Match
A crucial concept to grasp is that a regex-directed engine will always return the leftmost match. This behavior is essential to understand because it affects how the engine processes patterns and determines matches.
How It Works
When applying a regex to a string, the engine starts at the first character of the string and tries every possible permutation of the regex at that position. If all possibilities fail, the engine moves to the next character and repeats the process.
For example, consider applying the pattern «cat» to the string:
"He captured a catfish for his cat." Here’s a step-by-step breakdown:
The engine starts at the first character "H" and tries to match "c" from the pattern. This fails. The engine moves to "e", then space, and so on, failing each time until it reaches the fourth character "c". At "c", it tries to match the next character "a" from the pattern with the fifth character of the string, which is "a". This succeeds. The engine then tries to match "t" with the sixth character, "p", but this fails. The engine backtracks and resumes at the next character "a", continuing the process. Finally, at the 15th character in the string, it matches "c", then "a", and finally "t", successfully finding a match for "cat". Key Point
The engine reports the first valid match it finds, even if a better match could be found later in the string. In this case, it matches the first three letters of "catfish" rather than the standalone "cat" at the end of the string.
Why?
At first glance, the behavior of the regex-directed engine may seem similar to a basic text search routine. However, as we introduce more complex regex tokens, you’ll see how the internal workings of the engine have a profound impact on the matches it returns.
Understanding this behavior will help you avoid surprises and leverage the full power of regex for more effective and efficient text processing.

Regular expressions can also match non-printable characters using special sequences. Here are some common examples:
«\t»: Tab character (ASCII 0x09) «\r»: Carriage return (ASCII 0x0D) «\n»: Line feed (ASCII 0x0A) «\a»: Bell (ASCII 0x07) «\e»: Escape (ASCII 0x1B) «\f»: Form feed (ASCII 0x0C) «\v»: Vertical tab (ASCII 0x0B) Keep in mind that Windows text files use "\r\n" to terminate lines, while UNIX text files use "\n".
Hexadecimal and Unicode Characters
You can include any character in your regex using its hexadecimal or Unicode code point. For example:
«\x09»: Matches a tab character (same as «\t»). «\xA9»: Matches the copyright symbol (©) in the Latin-1 character set. «\u20AC»: Matches the euro currency sign (€) in Unicode. Additionally, most regex flavors support control characters using the syntax «\cA» through «\cZ», which correspond to Control+A through Control+Z. For example:
«\cM»: Matches a carriage return, equivalent to «\r». In XML Schema regex, the token «\c» is a shorthand for matching any character allowed in an XML name.
When working with Unicode regex engines, it’s best to use the «\uFFFF» notation to ensure compatibility with a wide range of characters.

To go beyond matching literal text, regex engines reserve certain characters for special functions. These are known as metacharacters. The following characters have special meanings in most regex flavors discussed in this tutorial:
[ \ ^ $ . | ? * + ( ) If you need to use any of these characters as literals in your regex, you must escape them with a backslash (). For instance, to match "1+1=2", you would write the regex as:
«1\+1=2» Without the backslash, the plus sign would be interpreted as a quantifier, causing unexpected behavior. For example, the regex «1+1=2» would match "111=2" in the string "123+111=234" because the plus sign is interpreted as "one or more of the preceding character."
Escaping Special Characters
To escape a metacharacter, simply prepend it with a backslash (). For example:
«.» matches a literal dot. «*» matches a literal asterisk. «+» matches a literal plus sign. Most regex flavors also support the \Q...\E escape sequence. This treats everything between \Q and \E as literal characters. For example:
«\Q*\d+*\E» This pattern matches the literal text "\d+". If the \E is omitted at the end, it is assumed. This syntax is supported by many engines, including Perl, PCRE, Java, and JGsoft, but it may have quirks in older Java versions.
Special Characters in Programming Languages
If you're a programmer, you might expect characters like single and double quotes to be special characters in regex. However, in most regex engines, they are treated as literal characters.
In programming, you must be mindful of characters that your language treats specially within strings. These characters will be processed by the compiler before being passed to the regex engine. For instance:
To use the regex «1+1=2» in C++ code, you would write it as "1\+1=2". The compiler converts the double backslashes into a single backslash for the regex engine. To match a Windows file path like "c:\temp", the regex would be «c:\temp», and in C++ code, it would be written as "c:\\temp". Refer to the specific language documentation to understand how to handle regex patterns within your code.

The simplest regular expressions consist of literal characters. A literal character is a character that matches itself. For example, the regex «a» will match the first occurrence of the character "a" in a string. Consider the string "Jack is a boy": this pattern will match the "a" after the "J".
It’s important to note that the regex engine doesn’t care where the match occurs within a word unless instructed otherwise. If you want to match entire words, you’ll need to use word boundaries, a concept we’ll cover later.
Similarly, the regex «cat» will match the word "cat" in the string "About cats and dogs." This pattern consists of three literal characters in sequence: «c», «a», and «t». The regex engine looks for these characters in the specified order.
Case Sensitivity
By default, most regex engines are case-sensitive. This means that the pattern «cat» will not match "Cat" unless you explicitly configure the engine to perform a case-insensitive search.

A regular expression engine is a software component that processes regex patterns, attempting to match them against a given string. Typically, you won’t interact directly with the engine. Instead, it operates behind the scenes within applications and programming languages, which invoke the engine as needed to apply the appropriate regex patterns to your data or files.
Variations Across Regex Engines
As is often the case in software development, not all regex engines are created equal. Different engines support different regex syntaxes, often referred to as regex flavors. This tutorial focuses on the Perl 5 regex flavor, widely considered the most popular and influential. Many modern engines, including the open-source PCRE (Perl-Compatible Regular Expressions) engine, closely mimic Perl 5’s syntax but may introduce slight variations. Other notable engines include:
.NET Regular Expression Library Java’s Regular Expression Package (included from JDK 1.4 onwards) Whenever significant differences arise between flavors, this guide will highlight them, ensuring you understand which features are specific to Perl-derived engines.
Getting Hands-On with Regex
You can start experimenting with regular expressions in any text editor that supports regex functionality. One recommended option is EditPad Pro, which offers a robust regex engine in its evaluation version.
To try it out:
Copy and paste the text from this page into EditPad Pro. From the menu, select Search > Show Search Panel to open the search pane at the bottom. In the Search Text box, type «regex». Check the Regular expression option. Click Find First to locate the first match. Use Find Next to jump to subsequent matches. When there are no more matches, the Find Next button will briefly flash. A More Advanced Example
Let’s take it a step further. Try searching for the following regex pattern:
«reg(ular expressions?|ex(p|es)?)» This pattern matches all variations of the term "regex" used on this page, whether singular or plural. Without regex, you’d need to perform five separate searches to achieve the same result. With regex, one pattern does the job, saving you significant time and effort.
For instance, in EditPad Pro, select Search > Count Matches to see how many times the regex matches the text. This feature showcases the power of regex for efficient text processing.
Why Use Regex in Programming?
For programmers, regexes offer both performance and productivity benefits:
Efficiency: Even a basic regex engine can outperform state-of-the-art plain text search algorithms by applying a pattern once instead of running multiple searches. Reduced Development Time: Checking if a user’s input resembles a valid email address can be accomplished with a single line of code in languages like Perl, PHP, Java, or .NET, or with just a few lines when using libraries like PCRE in C. By incorporating regex into your workflows and applications, you can achieve faster, more efficient text processing and validation tasks.