Regular expressions let you specify pattern strings and perform searches and substitutions. Regular expressions are not easy to master, but they can be a powerful tool for processing text. Python offers rich regular expression functionality through the built-in re
module.
A regular expression (RE) is built from a string that represents a pattern. With RE functionality, you can examine any string with the pattern, and see which parts of the string, if any, match the pattern.
The re
module supplies Python’s RE functionality. The compile
function builds an RE object from a pattern string and optional flags. The methods of an RE object look for matches of the RE in a string or perform substitutions. The re
module also exposes functions equivalent to an RE object’s methods, but with the RE’s pattern string as the first argument.
REs can be difficult to master, and this book does not purport to teach them; we cover only the ways in which you can use REs in Python. For general coverage of REs, we recommend the book Mastering Regular Expressions, by Jeffrey Friedl (O’Reilly). Friedl’s book offers thorough coverage of REs at both tutorial and advanced levels. Many tutorials and references on REs can also be found online, including an excellent, detailed tutorial in the online docs. Sites like Pythex and regex101 let you test your REs interactively.
In v3, by default, REs work in two ways: when applied to str
instances, an RE matches accordingly (for example, a Unicode character c
is deemed to be “a letter” if 'LETTER' in unicodedata.name(c)
); when applied to bytes
instances, an RE matches in terms of ASCII (for example, a byte character c
is deemed to be “a letter” if c in string.ascii_letters
). In v2, REs, by default, match in terms of ASCII, whether applied to str
or unicode
instances; you can override this behavior by explicitly specifying optional flags such as re.U
, covered in “Optional Flags”. For example, in v3:
import
re
(
re
.
findall
(
r
'
w+
'
,
'
cittá
'
)
)
# prints
['cittá']
while, in v2:
import
re
(
re
.
findall
(
r
'
w+
'
,
u
'
cittá
'
)
)
# prints
['citt']
(
re
.
findall
(
r
'
w+
'
,
u
'
cittá
'
,
re
.
U
)
)
# prints
['cittxe1']
The pattern string representing a regular expression follows a specific syntax:
Alphabetic and numeric characters stand for themselves. An RE whose pattern is a string of letters and digits matches the same string.
Many alphanumeric characters acquire special meaning in a pattern when they are preceded by a backslash ().
Punctuation works the other way around: self-matching when escaped, special meaning when unescaped.
The backslash character is matched by a repeated backslash (i.e., pattern \
).
Since RE patterns often contain backslashes, it’s best to always specify them using raw-string syntax (covered in “Strings”). Pattern elements (such as r' '
, equivalent to the nonraw string literal '\t'
) do match the corresponding special characters (in this case, the tab character ' '
, AKA chr(9)
); so, you can use raw-string syntax even when you need a literal match for such special characters.
Table 9-1 lists the special elements in RE pattern syntax. The exact meanings of some pattern elements change when you use optional flags, together with the pattern string, to build the RE object. The optional flags are covered in “Optional Flags”.
Element | Meaning |
---|---|
|
Matches any single character except |
|
Matches start of string (if |
|
Matches end of string (if |
|
Matches zero or more cases of the previous RE; greedy (match as many as possible) |
|
Matches one or more cases of the previous RE; greedy (match as many as possible) |
|
Matches zero or one case of the previous RE; greedy (match one if possible) |
|
Nongreedy versions of |
|
Matches between |
|
Matches between |
|
Matches any one of a set of characters contained within the brackets |
|
Matches one character not contained within the brackets after the caret |
|
Matches either the preceding RE or the following RE |
|
|
|
Alternate way to set optional flags; no effect on matcha |
|
Like |
|
Like |
|
Matches whatever was previously matched by group named |
|
Content of parentheses is just a comment; no effect on match |
|
Lookahead assertion: matches if RE |
|
Negative lookahead assertion: matches if RE |
|
Lookbehind assertion: matches if there is a match ending at the current position for RE |
|
Negative lookbehind assertion: matches if there is no match ending at the current position for RE |
|
Matches whatever was previously matched by group numbered |
|
Matches an empty string, but only at the start of the whole string |
|
Matches an empty string, but only at the start or end of a word (a maximal sequence of alphanumeric characters; see also |
|
Matches an empty string, but not at the start or end of a word |
|
Matches one digit, like the set |
|
Matches one nondigit, like the set |
|
Matches a whitespace character, like the set |
|
Matches a nonwhitespace character, like the set |
|
Matches one alphanumeric character; unless in Unicode mode, or |
|
Matches one nonalphanumeric character, the reverse of |
|
Matches an empty string, but only at the end of the whole string |
|
Matches one backslash character |
a Always place the |
Use raw-string syntax for all RE pattern literals, and for them only: this ensures you’ll never forget to escape a backslash (), and improves code readability as it makes your RE pattern literals stand out.
.*
as a substring of a regular expression’s pattern string means “any number of repetitions (zero or more) of any character.” In other words, .*
matches any substring of a target string, including the empty substring. .+
is similar, but matches only a nonempty substring. For example:
r
'pre.*post'
matches a string containing a substring 'pre'
followed by a later substring 'post'
, even if the latter is adjacent to the former (e.g., it matches both 'prepost'
and 'pre23post'
). On the other hand:
r
'pre.+post'
matches only if 'pre'
and 'post'
are not adjacent (e.g., it matches 'pre23post'
but does not match 'prepost'
). Both patterns also match strings that continue after the 'post'
. To constrain a pattern to match only strings that end with 'post'
, end the pattern with . For example:
r
'pre.*post'
matches 'prepost'
, but not 'preposterous'
.
All of these examples are greedy, meaning that they match the substring beginning with the first occurrence of 'pre'
all the way to the last occurrence of 'post'
. When you care about what part of the string you match, you may want to specify nongreedy matching, meaning to match the substring beginning with the first occurrence of 'pre'
but only up to the first following occurrence of 'post'
.
For example, when the string is 'preposterous and post facto'
, the greedy RE pattern r'pre.*post'
matches the substring 'preposterous and post'
; the non-greedy variant r'pre.*?post'
matches just the substring 'prepost'
.
Another frequently used element in RE patterns is , which matches a word boundary. To match the word
'his'
only as a whole word and not its occurrences as a substring in such words as 'this'
and 'history'
, the RE pattern is:
r
'his'
with word boundaries both before and after. To match the beginning of any word starting with 'her'
, such as 'her'
itself and 'hermetic'
, but not words that just contain 'her'
elsewhere, such as 'ether'
or 'there'
, use:
r
'her'
with a word boundary before, but not after, the relevant string. To match the end of any word ending with 'its'
, such as 'its'
itself and 'fits'
, but not words that contain 'its'
elsewhere, such as 'itsy'
or 'jujitsu'
, use:
r
'its'
with a word boundary after, but not before, the relevant string. To match whole words thus constrained, rather than just their beginning or end, add a pattern element w*
to match zero or more word characters. To match any full word starting with 'her'
, use:
r
'herw*'
To match just the first three letters of any word starting with 'her'
, but not the word 'her'
itself, use a negative word boundary B
:
r
'herB'
To match any full word ending with 'its'
, including 'its'
itself, use:
r
'w*its'
You denote sets of characters in a pattern by listing the characters within brackets ([]
). In addition to listing characters, you can denote a range by giving the first and last characters of the range separated by a hyphen (-
). The last character of the range is included in the set, differently from other Python ranges. Within a set, special characters stand for themselves, except ,
]
, and -
, which you must escape (by preceding them with a backslash) when their position is such that, if unescaped, they would form part of the set’s syntax. You can denote a class of characters within a set by escaped-letter notation, such as d
or S
. in a set means a backspace character (
chr(8)
), not a word boundary. If the first character in the set’s pattern, right after the [
, is a caret (^
), the set is complemented: such a set matches any character except those that follow ^
in the set pattern notation.
A frequent use of character sets is to match a word using a definition of which characters can make up a word that differs from w
’s default (letters and digits). To match a word of one or more characters, each of which can be a letter, an apostrophe, or a hyphen, but not a digit (e.g., "Finnegan-O'Hara"
), use:
r
"[a-zA-Z'-]+"
It’s not strictly necessary to escape the hyphen with a backslash in this case, since its position at the end of the set makes the situation syntactically unambiguous. However, the backslash is advisable because it makes the pattern more readable by visually distinguishing the hyphen that you want to have as a character in the set from those used to denote ranges.
A vertical bar (|
) in a regular expression pattern, used to specify alternatives, has low syntactic precedence. Unless parentheses change the grouping, |
applies to the whole pattern on either side, up to the start or end of the pattern, or to another |
. A pattern can be made up of any number of subpatterns joined by |
. To match such an RE, the first subpattern is tried, and if it matches, the others are skipped. If the first subpattern doesn’t match, the second subpattern is tried, and so on. |
is neither greedy nor nongreedy: it just doesn’t take the length of the match into account.
Given a list L
of words, an RE pattern that matches any one of the words is:
'|'
.
join
(
r
'
{}
'
.
format
(
word
)
for
word
in
L
)
If the items of L
can be more general strings, not just words, you need to escape each of them with the function re.escape
(covered in Table 9-3), and you may not want the word boundary markers on either side. In this case, use the following RE pattern:
'
|
'
.
join
(
re
.
escape
(
s
)
for
s
in
L
)
A regular expression can contain any number of groups, from none to 99 (or even more, but only the first 99 groups are fully supported). Parentheses in a pattern string indicate a group. Element (?P<id>...)
also indicates a group, and gives the group a name, id
, that can be any Python identifier. All groups, named and unnamed, are numbered from left to right, 1 to 99; “group 0” means the whole RE.
For any match of the RE with a string, each group matches a substring (possibly an empty one). When the RE uses |
, some groups may not match any substring, although the RE as a whole does match the string. When a group doesn’t match any substring, we say that the group does not participate in the match. An empty string (''
) is used as the matching substring for any group that does not participate in a match, except where otherwise indicated later in this chapter. For example:
r
'(.+)1+'
matches a string made up of two or more repetitions of any nonempty substring. The (.+)
part of the pattern matches any nonempty substring (any character, one or more times) and defines a group, thanks to the parentheses. The 1+
part of the pattern matches one or more repetitions of the group, and anchors the match to the end of the string.
A regular expression pattern element with one or more of the letters iLmsux
between (?
and )
lets you set RE options within the pattern, rather than by the flags
argument to the compile
function of the re
module. Options apply to the whole RE, no matter where the options element occurs in the pattern.
In particular, placement at the start is mandatory if x
is among the options, since x
changes the way Python parses the pattern. In 3.6, options not at the start of the pattern produce a deprecation warning.
Using the explicit flags
argument is more readable than placing an options element within the pattern. The flags
argument to the function compile
is a coded integer built by bitwise ORing (with Python’s bitwise OR operator, |
) one or more of the following attributes of the module re
. Each attribute has both a short name (one uppercase letter), for convenience, and a long name (an uppercase multiletter identifier), which is more readable and thus normally preferable:
I
or IGNORECASE
Makes matching case-insensitive
L
or LOCALE
Causes w
, W
, , and
B
matches to depend on what the current locale deems alphanumeric; deprecated in v3
M
or MULTILINE
Makes the special characters ^
and $
match at the start and end of each line (i.e., right after/before a newline), as well as at the start and end of the whole string (A
and always match only the start and end of the whole string)
S
or DOTALL
Causes the special character .
to match any character, including a newline
U
or UNICODE
Makes w
, W
, , and
B
matches depend on what Unicode deems alphanumeric; deprecated in v3
X
or VERBOSE
Causes whitespace in the pattern to be ignored, except when escaped or in a character set, and makes a #
character in the pattern begin a comment that lasts until the end of the line
For example, here are three ways to define equivalent REs with function compile
, covered in Table 9-3. Each of these REs matches the word “hello” in any mix of upper- and lowercase letters:
import
re
r1
=
re
.
compile
(
r
'(?i)hello'
)
r2
=
re
.
compile
(
r
'hello'
,
re
.
I
)
r3
=
re
.
compile
(
r
'hello'
,
re
.
IGNORECASE
)
The third approach is clearly the most readable, and thus the most maintainable, even though it is slightly more verbose. The raw-string form is not necessary here, since the patterns do not include backslashes; however, using raw strings is innocuous, and we recommend you always do that for RE patterns, to improve clarity and readability.
Option re.VERBOSE
(or re.X
) lets you make patterns more readable and understandable by appropriate use of whitespace and comments. Complicated and verbose RE patterns are generally best represented by strings that take up more than one line, and therefore you normally want to use the triple-quoted raw-string format for such pattern strings. For example, to match old-style Python v2 int
literals:
repat_num1
=
r
'(0[0-7]*|0x[da-fA-F]+|[1-9]d*)L?'
repat_num2
=
r
'''(?x) # pattern matching int literals
(0 [0-7]* | # octal: leading 0, 0+ octal digits
0x [da-fA-F]+ | # hex: 0x, then 1+ hex digits
[1-9] d* ) # decimal: leading non-0, 0+ digits
L? # optional trailing L, end of string
'''
The two patterns defined in this example are equivalent, but the second one is made more readable and understandable by the comments and the free use of whitespace to visually group portions of the pattern in logical ways.
So far, we’ve been using regular expressions to match strings. For example, the RE with pattern r'box'
matches strings such as 'box'
and 'boxes'
, but not 'inbox'
. In other words, an RE match is implicitly anchored at the start of the target string, as if the RE’s pattern started with A
.
Often, you’re interested in locating possible matches for an RE anywhere in the string, without anchoring (e.g., find the r'box'
match inside such strings as 'inbox'
, as well as in 'box'
and 'boxes'
). In this case, the Python term for the operation is a search, as opposed to a match. For such searches, use the search
method of an RE object; the match
method deals with matching only from the start. For example:
import
re
r1
=
re
.
compile
(
r
'
box
'
)
if
r1
.
match
(
'
inbox
'
)
:
(
'
match succeeds
'
)
else
:
(
'
match fails
'
)
# prints:
match fails
if
r1
.
search
(
'
inbox
'
)
:
(
'
search succeeds
'
)
# prints:
search succeeds
else
:
(
'
search fails
'
)
The pattern elements ensuring that a regular expression search (or match) is anchored at string start and string end are A
and , respectively. More traditionally, elements
^
for start and $
for end are also used in similar roles. ^
is the same as A
, and $
is the same as , for RE objects that are not multiline (i.e., that do not contain pattern element
(?m)
and are not compiled with the flag re.M
or re.MULTILINE
). For a multiline RE, however, ^
anchors at the start of any line (i.e., either at the start of the whole string or at any position right after a newline character
). Similarly, with a multiline RE, $
anchors at the end of any line (i.e., either at the end of the whole string or at any position right before a
). On the other hand, A
and anchor at the start and end of the string whether the RE object is multiline or not. For example, here’s a way to check whether a file has any lines that end with digits:
import
re
digatend
=
re
.
compile
(
r
'd$'
,
re
.
MULTILINE
)
with
open
(
'afile.txt'
)
as
f
:
if
digatend
.
search
(
f
.
read
()):
(
'some lines end with digits'
)
else
:
(
'no line ends with digits'
)
A pattern of r'd
'
is almost equivalent, but in that case the search fails if the very last character of the file is a digit not followed by an end-of-line character. With the preceding example, the search succeeds if a digit is at the very end of the file’s contents, as well as in the more usual case where a digit is followed by an end-of-line character.
A regular expression object r
has the following read-only attributes that detail how r
was built (by the function compile
of the module re
, covered in Table 9-3):
flags
The flags
argument passed to compile
, or 0
when flags
is omitted (in v2; in v3, the default when flags
is omitted is re.UNICODE
)
groupindex
A dictionary whose keys are group names as defined by elements (?P<id>...)
; the corresponding values are the named groups’ numbers
pattern
The pattern string from which r
is compiled
These attributes make it easy to get back from a compiled RE object to its pattern string and flags, so you never have to store those separately.
An RE object r
also supplies methods to locate matches for r
within a string, as well as to perform substitutions on such matches (Table 9-2). Matches are generally represented by special objects, covered in “Match Objects”.
findall |
When
When
When
|
finditer |
|
match |
Returns an appropriate match object when a substring of
|
search |
Returns an appropriate match object for the leftmost substring of
|
split |
Returns a list
When
If
|
sub |
Returns a copy of
Without the final When
When
|
subn |
|
Match objects are created and returned by the methods match
and search
of a regular expression object, and are the items of the iterator returned by the method finditer
. They are also implicitly created by the methods sub
and subn
when the argument repl
is callable, since in that case the appropriate match object is passed as the argument on each call to repl
. A match object m
supplies the following read-only attributes that detail how m
was created:
pos
The start
argument that was passed to search
or match
(i.e., the index into s
where the search for a match began)
endpos
The end
argument that was passed to search
or match
(i.e., the index into s
before which the matching substring of s
had to end)
lastgroup
The name of the last-matched group (None
if the last-matched group has no name, or if no group participated in the match)
lastindex
The integer index (1 and up) of the last-matched group (None
if no group participated in the match)
re
The RE object r
whose method created m
string
The string s
passed to finditer
, match
, search
, sub
, or subn
A match object m
also supplies several methods, detailed in Table 9-3.
end, span, start |
These methods return the limit indices, within |
expand |
Returns a copy of |
group |
When called with a single argument When |
groups |
Returns a tuple with one item per group in |
groupdict |
Returns a dictionary whose keys are the names of all named groups in |
The re
module supplies the attributes listed in “Optional Flags”. It also provides one function for each method of a regular expression object (findall
, finditer
, match
, search
, split
, sub
, and subn
), each with an additional first argument, a pattern string that the function implicitly compiles into an RE object. It’s usually better to compile pattern strings into RE objects explicitly and call the RE object’s methods, but sometimes, for a one-off use of an RE pattern, calling functions of the module re
can be slightly handier. For example, to count the number of occurrences of 'hello'
in any mix of cases, one concise, function-based way is:
import
re
_
,
count
=
re
.
subn
(
r
'(?i)hello'
,
''
,
astring
)
(
'Found'
,
count
,
'occurrences of "hello"'
)
In such cases, RE options (here, case insensitivity) must be encoded as RE pattern elements (here (?i)
): the functions of the re
module do not accept a flags
argument. The re
module internally caches RE objects it creates from the patterns passed to functions; to purge the cache and reclaim some memory, call re.purge()
.
The re
module also supplies error
, the class of exceptions raised upon errors (generally, errors in the syntax of a pattern string), and two more functions:
compile |
Creates and returns an RE object, parsing string |
escape |
Returns a copy of string |