Regex

8.2. Regex#

Hide code cell source

 
import sys
from pathlib import Path

current = Path.cwd()
for parent in [current, *current.parents]:
    if (parent / '_config.yml').exists():
        project_root = parent  # ← Add project root, not chapters
        break
else:
    project_root = Path.cwd().parent.parent

sys.path.insert(0, str(project_root))

from shared import thinkpython, diagram, jupyturtle, structshape
from shared.download import download

# Register as top-level modules so direct imports work in subsequent cells
sys.modules['thinkpython'] = thinkpython
sys.modules['diagram'] = diagram
sys.modules['jupyturtle'] = jupyturtle

String methods allow you to search and manipulate text to a certain extent. A regular expression (regex) is a sequence of characters that defines a search pattern to search, match, and manipulate text in more powerful ways that go beyond simple string methods. For example:

Task

Use

Check if string starts with ‘http’

str.startswith()

Replace all spaces with underscores

str.replace()

Extract all email addresses from text

regex

Validate a phone number format

regex

Find words matching a complex pattern

regex

The rule of thumb: if the pattern is fixed and simple, use string methods. If the pattern is variable or complex, use regex.

For example, to search a pattern in a text, you can use the find() string method, which returns an index if it finds the pattern.

text = "I am Dracula; and I bid you welcome, Mr. Harker,\
    to my house."
pattern = 'Dracula'
text.find(pattern)

5

8.2.1. Backslashes in Regex Patterns#

In Escape Sequences and Raw Strings you saw that Python treats a backslash in a string literal as the start of an escape sequence, so "\n" is a single newline character. Regex patterns also use backslashes, but for their own purposes: \d means “any digit,” and \. means “a literal dot.” The Metacharacters section below covers these sequences.

So a regex pattern passes through two layers of backslash processing:

  1. Python reads the string literal and turns escape sequences such as \n, \t, and \b into single characters.

  2. The regex engine then reads the resulting string and gives its own meaning to sequences such as \d, \w, and \b.

8.2.1.1. Raw Strings#

A raw string, prefixed with r, turns off the first layer: Python keeps every backslash as an ordinary character, so the regex engine receives exactly the pattern you typed. That is why regex patterns are almost always written as raw strings.

The two layers disagree about \b. To Python, "\b" is a backspace character; to the regex engine, \b is a word boundary. Only the raw string gives the regex engine the pattern you meant:

import re

text = "cat catfish concatenate"

print(re.findall("\bcat\b", text))    # []: Python turned each \b into a backspace character
print(re.findall(r"\bcat\b", text))   # ['cat']: the regex engine received \b, a word boundary

[]
['cat']

Without a raw string, you have to double every backslash so that one survives the first layer. In the cell below, compare how many backslashes each version needs to produce the same characters:

print('\\\\')                  # normal string: each \\ becomes one backslash, so this prints \\
print(r'\\')                   # raw string: both backslashes are kept, so this also prints \\
print(r'\\\n')                 # raw string: three backslashes and an n, printed as \\\n
print(r'\\\n' == '\\\\\\n')    # True: the normal string needs six backslashes for the same four characters

\\
\\
\\\n
True

In short, a raw string tells Python to treat each backslash as an ordinary character, so you can write a regex pattern once, exactly as the regex engine should see it.

8.2.2. The re Module#

Python’s built-in re module provides regex support. The 6 most commonly used regex functions are:

Function

Description

Sample Syntax

Return

re.search()

Find first match anywhere in the string

re.search(pattern, text)

Match object or None

re.match()

Match only at the start of the string

re.match(pattern, text)

Match object or None

re.findall()

Find all matches; return as a list

re.findall(pattern, text)

list of strings

re.sub()

Find and replace matches

re.sub(pattern, repl, text)

str

re.split()

Split string on a pattern

re.split(pattern, text)

list of strings

re.fullmatch()

Match the entire string against the pattern

re.fullmatch(pattern, text)

Match object or None

8.2.2.1. The Match object#

re.search(), re.match(), and re.fullmatch() functions return a Match object when the pattern matches.

For example,

  • returns: re.search(pattern, text) scans through text and returns

    • a Match object for the first location where pattern is found.

    • If the pattern is not found anywhere in the string, it returns None.

  • Match: A Match object has the following commonly used attributes and methods:

Attribute / Method

Description

Example

.group()

Returns the matched substring

m.group() → 'Dracula'

.start()

Index where the match begins

m.start() → 5

.end()

Index where the match ends

m.end() → 12

.span()

Tuple of (start, end)

m.span() → (5, 12)

.string

The original string that was searched

m.string → 'I am Dracula...'

 
import re

text = "I am Dracula; and I bid you welcome, Mr. Harker, to my house."
pattern = 'Dracula'

result = re.search(pattern, text)     ### pattern: Dracula; text: the line
result                              ### the Match object

<re.Match object; span=(5, 12), match='Dracula'>

If the pattern appears in the text, search returns a Match object that contains the results of the search.

  1. String: Among other information, it has a variable named string that contains the text that was searched.

  2. Group: It also provides a method called group that returns the part of the text that matched the pattern.

  3. Span and Start/End: And it provides a method called span that returns the index in the text where the pattern starts and ends.

print(result.string)
print(result.group())
print(result.start())
print(result.end())
print(result.span())

I am Dracula; and I bid you welcome, Mr. Harker, to my house.
Dracula
5
12
(5, 12)

Note

.group() returns the matched substring from the text: the portion of the text that the pattern matched against. In simple cases like re.search('Dracula', text), the match equals the pattern string. But with a pattern that uses metacharacters (covered in the Metacharacters section below), such as a pattern for any dollar amount, .group() returns the actual text that matched, like '$42.99', not the pattern expression itself.

If the pattern doesn’t appear in the text, the return value from search is None. So you can check whether the search was successful by checking whether the result is None.

result = re.search('Count', text)
print(result)

result is None

None

True

The next cell previews two symbols with special meaning in a pattern, . and ?; the Metacharacters section below explains them.

s = "This is a test of the regular expression system."
print(re.findall('is', s))  # ['is', 'is']
print(re.findall('is.', s)) # ['is ', 'is ']    ### 'is' followed by any character (space in this case)
print(re.findall('is.?', s)) # ['is ', 'is ']   ### 'is' followed by zero or one character (space in this case)
print(re.findall('is.?', s, re.IGNORECASE)) # ['is ', 'is '] ### same as above, but case-insensitive   
print(re.findall('is.?', s, re.IGNORECASE | re.DOTALL)) # ['is ', 'is ']    ### same as above, but also makes '.' match newline characters (not relevant in this case since there are no newlines)

['is', 'is']
['is ', 'is ']
['is ', 'is ']
['is ', 'is ']
['is ', 'is ']

The other functions in the table take a pattern the same way. The cell below runs findall(), search(), sub(), and split() with literal patterns, which match only the exact characters you type. The Metacharacters section shows how to write patterns that match variable text, such as any price or any run of spaces.

import re

text = "The cat sat on the mat with another cat."

# findall: get all matches
print(re.findall("cat", text))              # ['cat', 'cat']

# search: first match object
m = re.search("cat", text)
print(m.group())                            # 'cat'
print(m.start(), m.end())                   # position in string

# sub: replace
print(re.sub("cat", "dog", text))           # every 'cat' becomes 'dog'

# split
print(re.split(", ", "one, two, three"))    # ['one', 'two', 'three']

['cat', 'cat']
cat
4 7
The dog sat on the mat with another dog.
['one', 'two', 'three']

### Exercise: The Match Object
# Difficulty: Basic
import re
text = "Customer ID: 4892, Order date: 2024-03-15"
# 1. Use re.search() to find the word 'Order' in text
# 2. Print the matched string, start index, end index, and span
# 3. Search text for 'Invoice' and print whether the result is None
### Your code starts here



### Your code ends here

Hide code cell source

 
### Solution
import re
text = "Customer ID: 4892, Order date: 2024-03-15"
m = re.search('Order', text)
print(m.group())
print(m.start())
print(m.end())
print(m.span())
print(re.search('Invoice', text) is None)

Order
19
24
(19, 24)
True

8.2.3. Metacharacters#

Metacharacters are characters that carry special meaning inside a regex pattern: instead of matching themselves literally, they instruct the regex engine to do something specific, like match any character, mark a boundary, or repeat a pattern. There are 14 of them in Python’s re module; the table below lists each bracket pair, {}, [], and (), as one row. You need to escape them if you want them to be regular characters.

Type

Character

Meaning

Example

Wildcard

.

Matches any character except newline

c.t → cat, cot

Anchor

^

Start of string

^Hello

Anchor

$

End of string

end$

Quantifier

*

0 or more repetitions

a*

Quantifier

+

1 or more repetitions

a+

Quantifier

?

Optional (0 or 1) / makes quantifier lazy

colou?r

Quantifier

{}

Specific repetition range

o{2}

Character Class Delimiters

[]

Defines a set of allowed characters

[a-z]

Grouping Delimiters

()

Groups patterns and captures matches

(cat| dog)

Escape

\

Escapes metacharacters or forms special sequences

\d, \w, \.

Alternation

|

Logical OR between patterns

cat  | dog

 
import re

To match a metacharacter literally, escape it with a backslash: \. matches a real dot, and \$ matches a real dollar sign. Compare the patterns below:

text = "Total: $19.95 (was $24.50)"
print(re.findall(r"\$", text))                    # ['$', '$']: \$ is a literal dollar sign

print(re.findall(r"19.95", "19.95 or 19x95"))     # ['19.95', '19x95']: . matches any character
print(re.findall(r"19\.95", "19.95 or 19x95"))    # ['19.95']: \. matches only a real dot

['$', '$']
['19.95', '19x95']
['19.95']

The sections below present the metacharacters in groups.

8.2.3.1. Quantifiers#

Quantifiers tell the regex engine how many times the preceding character, group, or character class should match.

Quantifier

Meaning

Example

Matches

*

0 or more

ab*

a, ab, abb, abbb

+

1 or more

ab+

ab, abb, abbb (not a)

?

0 or 1

ab?

a or ab only

{n}

Exactly n

o{2}

oo

{n,}

n or more

o{2,}

oo, ooo, oooo…

{n,m}

Between n & m

o{2,3}

oo, ooo

By default quantifiers are greedy (match as much as possible). Add ? to make them lazy.

text = "<b>bold</b> and <i>italic</i>"

# Greedy — matches as much as possible
print(re.findall(r"<.+>", text))     # ['<b>bold</b> and <i>italic</i>']

# Lazy — matches as little as possible
print(re.findall(r"<.+?>", text))    # ['<b>', '</b>', '<i>', '</i>']

# Exact and ranged quantifiers
print(re.findall(r"o{2}", "bot boot booot"))     # ['oo', 'oo']
print(re.findall(r"o{2,3}", "bot boot booot"))   # ['oo', 'ooo']

['<b>bold</b> and <i>italic</i>']
['<b>', '</b>', '<i>', '</i>']
['oo', 'oo']
['oo', 'ooo']

In <.+>, the + is greedy. It matches as many characters as possible while still allowing the overall pattern to succeed. So it gobbles everything from the first < all the way to the last >.

Adding ? after a quantifier switches it to lazy mode: instead of matching as much as possible, it now matches as little as possible. So <.+?> still needs at least one character (that’s the +), but stops at the earliest > it can find.

8.2.3.2. Greedy vs Non-greedy#

Quantifiers like * and + are greedy by default. Add ? to make them non-greedy.

text_block = """Title: Notes
Email: ALICE@example.com
Email: bob@Example.org"""

# IGNORECASE
emails = re.findall(r'[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}', text_block, flags=re.IGNORECASE)
print(emails)

# Greedy vs non-greedy on tags
html = "<b>bold</b><i>italic</i>"
print(re.findall(r'<.*>', html))     # greedy
print(re.findall(r'<.*?>', html))    # non-greedy

['ALICE@example.com', 'bob@Example.org']
['<b>bold</b><i>italic</i>']
['<b>', '</b>', '<i>', '</i>']

### Exercise: Greedy and Non-greedy Quantifiers
# Difficulty: Basic
text = "The color and the colour look the same."
html = "<x>1</x><x>2</x>"
# 1. Use the ? quantifier to find both spellings, 'color' and 'colour', in text.
# 2. Extract the tags from html non-greedily: ['<x>', '</x>', '<x>', '</x>'].
# 3. Run the same tag pattern greedily and compare the result.
### Your code starts here



### Your code ends here

Hide code cell source

 
### Solution
text = "The color and the colour look the same."
html = "<x>1</x><x>2</x>"

print(re.findall(r"colou?r", text))   # ['color', 'colour']
print(re.findall(r"<.*?>", html))     # ['<x>', '</x>', '<x>', '</x>']
print(re.findall(r"<.*>", html))      # ['<x>1</x><x>2</x>']: greedy grabs as much as it can

['color', 'colour']
['<x>', '</x>', '<x>', '</x>']
['<x>1</x><x>2</x>']

8.2.3.3. Anchors#

Anchors don’t match characters; they match positions in the string.

Anchor

Meaning

^

Start of string (or line with re.MULTILINE)

$

End of string (or line)

\b

Word boundary

\B

Non-word boundary

 
# ^ and $
print(re.findall(r"^\w+", "Hello world"))     # ['Hello'] — only at start
print(re.findall(r"\w+$", "Hello world"))     # ['world'] — only at end

# Word boundary \b
text = "cat catfish concatenate"
print(re.findall(r"\bcat\b", text))           # ['cat'] — whole word only
print(re.findall(r"cat", text))               # ['cat', 'cat', 'cat'] — anywhere

# Multiline
multi = "line1\nline2\nline3"
print(re.findall(r"^\w+", multi, re.MULTILINE))  # ['line1', 'line2', 'line3']

['Hello']
['world']
['cat']
['cat', 'cat', 'cat']
['line1', 'line2', 'line3']

8.2.3.4. Character Classes#

Before writing larger patterns, it helps to know the core building blocks. Character classes match one character from a defined set. They’re written with square brackets [ ].

Pattern

Matches

[aeiou]

any single vowel

[a-z]

any lowercase letter

[A-Z]

any uppercase letter

[0-9]

any digit

[a-zA-Z0-9]

any alphanumeric character

[^aeiou]

any character not a vowel (^ negates)

Shorthand classes (work outside brackets too):

Pattern

Meaning

Example Match

.

Any character (except newline)

\d

digit

7

\w

word char (letter/digit/underscore)

A, x, 9, _

\s

whitespace

space, tab

Observe the escape sequence '\w'.

import re
s = "This is a regular expression."
print(re.findall(r'\w', s))     ### \w matches any alphanumeric character (letters, digits, and underscore)
print(re.findall(r'\w+', s))    ### + means "one or more occurrences of the preceding pattern"
print(re.findall(r'\w*', s))    ### * means "zero or more occurrences of the preceding pattern"

['T', 'h', 'i', 's', 'i', 's', 'a', 'r', 'e', 'g', 'u', 'l', 'a', 'r', 'e', 'x', 'p', 'r', 'e', 's', 's', 'i', 'o', 'n']
['This', 'is', 'a', 'regular', 'expression']
['This', '', 'is', '', 'a', '', 'regular', '', 'expression', '', '']

\\s matches these whitespace characters:

Character

Name

\\n

newline

\\t

tab

\\r

carriage return

space

\\f

form feed

\\v

vertical tab

Use raw strings like r'\d+' for regex patterns so backslashes are interpreted correctly.

import re

text = "Hello World 123! foo_bar"

print(re.findall(r"\d", text))        # individual digits
print(re.findall(r"\d+", text))       # consecutive digits
print(re.findall(r"\w+", text))       # words (incl. underscore)
print(re.findall(r"[A-Z][a-z]+", text))  # capitalized words
print(re.findall(r"[^a-zA-Z\s]+", text)) # non-alpha, non-space

['1', '2', '3']
['123']
['Hello', 'World', '123', 'foo_bar']
['Hello', 'World']
['123!', '_']

sample = "User_42 logged in at 09:30 on 2026-03-11"

print(re.findall(r'\d+', sample))                  # all digit runs
print(re.findall(r'[A-Za-z_]+', sample))            # word-like alphabetic tokens
print(re.findall(r'\d{2}:\d{2}', sample))        # HH:MM time
print(re.findall(r'\d{4}-\d{2}-\d{2}', sample))  # YYYY-MM-DD date

['42', '09', '30', '2026', '03', '11']
['User_', 'logged', 'in', 'at', 'on']
['09:30']
['2026-03-11']

### Exercise: Regex Syntax Essentials
# Difficulty: Basic
s = "IDs: A12, B7, C999"
# 1. Extract all uppercase letters
# 2. Extract all digit sequences
# 3. Extract letter+digit tokens like A12, B7, C999
### Your code starts here



### Your code ends here

Hide code cell source

 
### Solution
s = "IDs: A12, B7, C999"
print(re.findall(r'[A-Z]', s))
print(re.findall(r'\d+', s))
print(re.findall(r'[A-Z]\d+', s))

['I', 'D', 'A', 'B', 'C']
['12', '7', '999']
['A12', 'B7', 'C999']

### Exercise: Regex Escape Sequences
# Difficulty: Basic
s = "Price: $19.95, code=A_7, spaces here"
# 1. Extract all digit sequences
# 2. Extract all word tokens
# 3. Extract literal '$' and literal '.' matches
### Your code starts here



### Your code ends here

Hide code cell source

 
### Solution
import re

# Solution
s = "Price: $19.95, code=A_7, spaces here"
print(re.findall(r'\d+', s))
print(re.findall(r'\w+', s))
print(re.findall(r'\$', s))
print(re.findall(r'\.', s))

['19', '95', '7']
['Price', '19', '95', 'code', 'A_7', 'spaces', 'here']
['$']
['.']

8.2.3.5. Groups & Capturing#

Parentheses () group part of a pattern into a single unit. A capturing group also saves the matched text so you can extract or reuse it afterward. Use non-capturing groups (?:...) when you need grouping for structure but don’t need to extract the text. Named groups (?P<name>...) let you refer to captured text by name instead of number.

Syntax

Meaning

(...)

Capturing group

(?:...)

Non-capturing group

(?P<name>...)

Named group

|

Alternation (OR)

 
# Capturing groups
dates = "2024-01-15 and 2023-12-31"
print(re.findall(r"(\d{4})-(\d{2})-(\d{2})", dates))
# [('2024', '01', '15'), ('2023', '12', '31')]

# Named groups: m is one object from the search, so gives only one match, not all matches
m = re.search(r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})", dates)
print(m.group('year'), m.group('month'), m.group('day'))

# Alternation
print(re.findall(r"cat|dog", "I have a cat and a dog"))  # ['cat', 'dog']

# Using groups in sub()
print(re.sub(r"(\d{4})-(\d{2})-(\d{2})", r"\2/\3/\1", dates))
# '01/15/2024 and 12/31/2023'

[('2024', '01', '15'), ('2023', '12', '31')]
2024 01 15
['cat', 'dog']
01/15/2024 and 12/31/2023

8.2.3.6. Groups and Extraction#

Parentheses create capture groups. You can extract parts of a match with .group(1), .group(2), etc. Named groups can make patterns more readable.

record = "OrderID=4821; Customer=Alice; Total=$39.50"
pattern = r'OrderID=(\d+); Customer=([A-Za-z]+); Total=\$(\d+(?:\.\d{2})?)'
m = re.search(pattern, record)

print(m.group(0))  # full match
print(m.group(1))  # order id
print(m.group(2))  # customer
print(m.group(3))  # total amount

OrderID=4821; Customer=Alice; Total=$39.50
4821
Alice
39.50

### Exercise: Capture Groups
# Difficulty: Intermediate
line = "name=Bob,age=27,dept=Sales"
# 1. Use one regex with 3 capture groups to extract name, age, dept
# 2. Print each extracted value on its own line
### Your code starts here



### Your code ends here

Hide code cell source

 
### Solution
line = "name=Bob,age=27,dept=Sales"
m = re.search(r'name=([A-Za-z]+),age=(\d+),dept=([A-Za-z]+)', line)
print(m.group(1))
print(m.group(2))
print(m.group(3))

Bob
27
Sales

8.2.3.7. Alternation (OR)#

Use | to match one of multiple patterns.

import re

# pattern using alternation
pattern = r"\.(jpg|png|gif)$"

files = [
    "photo.jpg",
    "diagram.png",
    "animation.gif",
    "document.pdf",
    "archive.zip"
]

for file in files:
    if re.search(pattern, file):
        print(f"{file} -> valid image file")
    else:
        print(f"{file} -> not an image")

photo.jpg -> valid image file
diagram.png -> valid image file
animation.gif -> valid image file
document.pdf -> not an image
archive.zip -> not an image

import re

pattern = r"\b(coffee|tea)\b"

sentences = [
    "I like coffee in the morning.",
    "She prefers tea at night.",
    "He drinks water.",
    "Coffee is my favorite."
]

for sentence in sentences:
    match = re.search(pattern, sentence, re.IGNORECASE)

    if match:
        print(f"Found beverage: {match.group()}")
    else:
        print("No beverage found")

Found beverage: coffee
Found beverage: tea
No beverage found
Found beverage: Coffee

8.2.4. Advanced Topics#

The patterns so far cover most everyday matching. This section adds three tools for harder cases: flags that change how a pattern matches, compiled patterns for reuse, and lookaround assertions that check the text around a match without including it.

import re

8.2.4.1. Flags#

Flags change matching behavior:

Flag

Shorthand

Meaning

re.IGNORECASE

re.I

Case-insensitive matching

re.MULTILINE

re.M

^/$ match line start/end

re.DOTALL

re.S

. matches newline too

re.VERBOSE

re.X

Allow comments/whitespace in pattern

 
# IGNORECASE
print(re.findall(r"hello", "Hello HELLO hello", re.I))  # ['Hello', 'HELLO', 'hello']

# DOTALL — dot matches newline
text = "<div>\nsome content\n</div>"
print(re.findall(r"<div>.*</div>", text, re.DOTALL))  # matches across lines

# VERBOSE — write readable patterns with comments
email_pattern = re.compile(r"""
    [\w.+-]+       # username
    @              # at sign
    [\w-]+         # domain name
    \.             # dot
    [\w.]+         # TLD
""", re.VERBOSE)

print(email_pattern.findall("Contact us at hello@example.com or support@test.org"))

['Hello', 'HELLO', 'hello']
['<div>\nsome content\n</div>']
['hello@example.com', 'support@test.org']

### Exercise: Regex Flags
# Difficulty: Basic
import re
log = """INFO: Server started
error: disk full
WARNING: low memory
ERROR: connection lost"""
# 1. Use re.findall() with re.IGNORECASE | re.MULTILINE to
#    extract every line that begins with 'error'
# 2. Print the list of matches
### Your code starts here



### Your code ends here

Hide code cell source

 
### Solution
import re
log = """INFO: Server started
error: disk full
WARNING: low memory
ERROR: connection lost"""
results = re.findall(r'^error.*$', log, flags=re.IGNORECASE | re.MULTILINE)
print(results)

['error: disk full', 'ERROR: connection lost']

8.2.4.2. Compiled Patterns#

Use re.compile() when reusing the same pattern multiple times; it is more efficient and cleaner.

# Compile once, use many times
phone_pattern = re.compile(r"\b\d{3}[-.]\d{3}[-.]\d{4}\b")

texts = [
    "Call me at 123-456-7890",
    "My number is 987.654.3210",
    "No phone here",
    "Reach us at 555-123-4567 or 800-999-0000"
]

for t in texts:
    matches = phone_pattern.findall(t)
    if matches:
        print(f"Found: {matches} in '{t}'")

Found: ['123-456-7890'] in 'Call me at 123-456-7890'
Found: ['987.654.3210'] in 'My number is 987.654.3210'
Found: ['555-123-4567', '800-999-0000'] in 'Reach us at 555-123-4567 or 800-999-0000'

### Exercise: Compiled Patterns
# Difficulty: Basic
import re
emails = ['alice@example.com', 'not-an-email', 'bob@company.org', 'charlie_at_test.net']
# 1. Compile a regex pattern that matches a simple email address
# 2. Print each email with True or False using the compiled pattern
### Your code starts here



### Your code ends here

Hide code cell source

 
### Solution
import re
emails = ['alice@example.com', 'not-an-email', 'bob@company.org', 'charlie_at_test.net']
email_re = re.compile(r'[\w.+-]+@[\w-]+\.[\w.]+')
for e in emails:
    print(e, bool(email_re.match(e)))

alice@example.com True
not-an-email False
bob@company.org True
charlie_at_test.net False

8.2.4.3. Lookahead & Lookbehind#

Match a pattern only if it is (or isn’t) preceded/followed by another pattern, without including that other pattern in the match.

Syntax

Type

Meaning

(?=...)

Positive lookahead

Followed by

(?!...)

Negative lookahead

NOT followed by

(?<=...)

Positive lookbehind

Preceded by

(?<!...)

Negative lookbehind

NOT preceded by

 
# Positive lookahead — prices followed by USD
text = "100USD 200EUR 300USD"
print(re.findall(r"\d+(?=USD)", text))     # ['100', '300']

# Negative lookahead — naive version backtracks into the numbers
print(re.findall(r"\d+(?!USD)", text))     # ['10', '200', '30']

# Negative lookahead — also forbid a following digit
print(re.findall(r"\d+(?!\d|USD)", text))  # ['200']

# Positive lookbehind — extract amount after $
text2 = "Price: $42.99, discount: $5.00"
print(re.findall(r"(?<=\$)[\d.]+", text2)) # ['42.99', '5.00']

['100', '300']
['10', '200', '30']
['200']
['42.99', '5.00']

The naive negative lookahead returns '10' and '30' because of backtracking. When \d+ first matches all of 100, the lookahead sees USD and fails. The engine then gives back one digit and tries again with 10. Now the next character is 0, not USD, so the lookahead succeeds and the engine returns '10', a fragment of a number that is followed by USD.

The fix is to require that the number ends where the match ends. (?!\d|USD) rejects a match that is followed by another digit or by USD, so the engine can no longer stop in the middle of a number. A word boundary such as \b\d+\b does not help here: digits and letters are both word characters, so there is no boundary between 0 and U and that pattern would miss every amount. Excluding a following digit is the reliable choice for this text.

### Exercise: Lookahead & Lookbehind
# Difficulty: Intermediate
import re
text = "Alice scored 95pts, Bob scored 80pts, Charlie scored 73pts"
# 1. Use a positive lookahead to extract all numbers followed by 'pts'
# 2. Use a positive lookbehind to extract numbers preceded by 'scored '
### Your code starts here



### Your code ends here

Hide code cell source

 
### Solution
import re
text = "Alice scored 95pts, Bob scored 80pts, Charlie scored 73pts"
print(re.findall(r'\d+(?=pts)', text))        # positive lookahead
print(re.findall(r'(?<=scored )\d+', text))   # positive lookbehind

['95', '80', '73']
['95', '80', '73']

8.2.5. Applications#

The rest of this section applies regular expressions to a real text, Bram Stoker’s Dracula: first to clean the downloaded file, then to substitute text, and finally to validate input with re.fullmatch(). A quick reference table closes the section.

8.2.5.1. Cleaning Text#

Before you can search the text of Dracula, you need to download it from Project Gutenberg and remove the header and footer information.

import re

The code below downloads the Dracula text from Project Gutenberg and saves it to the data folder. Then it cleans the file and saves the cleaned version in the same folder. All subsequent analysis will use these files.

from pathlib import Path
from urllib.request import urlretrieve

data_dir = project_root / 'data'
data_dir.mkdir(parents=True, exist_ok=True)

# Download Dracula text to the project data folder
url = 'https://www.gutenberg.org/files/345/345-0.txt'
raw_path = data_dir / 'pg345.txt'
clean_path = data_dir / 'pg345_cleaned.txt'
if not raw_path.exists():
    urlretrieve(url, raw_path)
    print('Downloaded Dracula to', raw_path)
else:
    print('Dracula already downloaded:', raw_path)

Dracula already downloaded: /home/runner/work/py/py/data/pg345.txt

# download('https://www.gutenberg.org/cache/epub/345/pg345.txt');

def clean_file(input_file, output_file):
    reader = open(input_file, encoding='utf-8')
    writer = open(output_file, 'w')

    for line in reader:
        if is_special_line(line):
            break

    for line in reader:
        if is_special_line(line):
            break
        writer.write(line)
        
    reader.close()
    writer.close()

def is_special_line(line):
    """Return True if the line marks the start or end of the Gutenberg content."""
    return line.startswith('***')

# def is_special_line(line):
#     return line.strip().startswith('*** ')

# Clean the Dracula text and save to data/pg345_cleaned.txt
clean_file(raw_path, clean_path)
print('Cleaned file saved to', clean_path)

Cleaned file saved to /home/runner/work/py/py/data/pg345_cleaned.txt

With the cleaned file saved, you can search it line by line. Here’s a function that loops through the lines in the book until it finds one that matches the given pattern, and returns the Match object.

def find_first(pattern, path=clean_path):
    with open(path, encoding='utf8') as f:
        for line in f:
            result = re.search(pattern, line)
            if result is not None:
                return result

You can use it to find the first mention of a character.

result = find_first('Harker')
result.string

'CHAPTER I. Jonathan Harker’s Journal\n'

For this example, you didn’t have to use regular expressions; you could have done the same thing more easily with the in operator. But regular expressions can do things the in operator cannot.

For example, if the pattern includes the vertical bar character, '|', it can match either the sequence on the left or the sequence on the right. Suppose you want to find the first mention of Mina Murray in the book, but you are not sure whether the book refers to her by first name or last. You can use the following pattern, which matches either name.

pattern = 'Mina|Murray'
result = find_first(pattern)
result.string

'CHAPTER V. Letters—Lucy and Mina\n'

You can use a pattern like this to see how many times a character is mentioned by either name. Here’s a function that loops through the book and counts the number of lines that match the given pattern.

def count_matches(pattern, path=clean_path):
    count = 0
    with open(path, encoding='utf8') as f:
        for line in f:
            result = re.search(pattern, line)
            if result is not None:
                count += 1
    return count

Next, count how many times the book mentions Mina.

count_matches('Mina|Murray')

229

The special character '^' matches the beginning of a string, so you can find a line that starts with a given pattern.

result = find_first('^Dracula')
result.string

'Dracula, jumping to his feet, said:--\n'

And the special character '$' matches the end of a string, so you can find a line that ends with a given pattern (ignoring the newline at the end).

result = find_first('Harker$')
result.string

'by five o’clock, we must start off; for it won’t do to leave Mrs. Harker\n'

### Exercise: Download and Clean Text
# Difficulty: Intermediate
# 1. Use raw_path and clean_path to print whether each file exists
# 2. If clean_path does not exist, run clean_file(raw_path, clean_path)
# 3. Read clean_path with read_text() and print how many characters it contains
### Your code starts here



### Your code ends here

Hide code cell source

 
### Solution
print(raw_path.exists(), clean_path.exists())
if not clean_path.exists():
    clean_file(raw_path, clean_path)
print(len(clean_path.read_text(encoding='utf8')))

True True
845928

8.2.5.2. String substitution#

Bram Stoker was born in Ireland, and when Dracula was published in 1897, he was living in England. So you would expect him to use the British spelling of words like “centre” and “colour”. To check, you can use the following pattern, which matches either “centre” or the American spelling “center”.

pattern = 'cent(er|re)'

In this pattern, the parentheses enclose the part of the pattern the vertical bar applies to. So this pattern matches a sequence that starts with 'cent' and ends with either 'er' or 're'.

result = find_first(pattern)
result.string

'horseshoe of the Carpathians, as if it were the centre of some sort of\n'

As expected, he used the British spelling.

You can also check whether he used the British spelling of “colour”. The following pattern uses the special character '?', which means that the previous character is optional.

pattern = 'colou?r'

This pattern matches either “colour” with the 'u' or “color” without it.

result = find_first(pattern)
line = result.string
line

'undergarment with long double apron, front, and back, of coloured stuff\n'

Again, as expected, he used the British spelling.

Suppose you want to produce an edition of the book with American spellings. You can use the sub function in the re module, which does string substitution.

re.sub(pattern, 'color', line)

'undergarment with long double apron, front, and back, of colored stuff\n'

The first argument is the pattern you want to find and replace, the second is what you want to replace it with, and the third is the string you want to search. In the result, you can see that “colour” has been replaced with “color”.

# I used this function to search for lines to use as examples

def all_matches(pattern, path=clean_path):
    with open(path, encoding='utf8') as f:
        for line in f:
            result = re.search(pattern, line)
            if result:
                print(line.strip())

### e.g., 

all_matches('weather')

weather. As I stood, the driver jumped again into his seat and shook the
weatherworn, was still complete; but it was evidently many a day since
it is a buoy with a bell, which swings in bad weather, and sends in a
am awakened by her moving about the room. Fortunately, the weather is so
learn the weather signs. To-day is a grey day, and the sun as I write is
experienced here, with results both strange and unique. The weather had
kept watch on weather signs from the East Cliff, foretold in an emphatic
_22 July_.--Rough weather last three days, and all hands busy with
weather. Passed Gibralter and out through Straits. All well.
and entering on the Bay of Biscay with wild weather ahead, and yet last

weather influences as we know that the Count can bring to bear; and if
that I am fully armed as there may be wolves; the weather is getting

# Here's the pattern I used (which uses some features we haven't seen)

# names = r'(?<!\.\s)[A-Z][a-zA-Z]+'

# all_matches(names)

### Exercise: String Substitution
# Difficulty: Intermediate
sample = "The colour of the city centre changed overnight."
# 1. Replace British spellings with American spellings using regex:
#    colour -> color, centre -> center
# 2. Print the transformed sentence
### Your code starts here



### Your code ends here

Hide code cell source

 
### Solution
import re

sample = "The colour of the city centre changed overnight."
sample = re.sub(r'colou?r', 'color', sample)
sample = re.sub(r'cent(er|re)', 'center', sample)
print(sample)

The color of the city center changed overnight.

8.2.5.3. re.fullmatch() for Validation#

re.fullmatch(pattern, text) succeeds only if the entire string matches the pattern. This is the right tool for validation tasks (IDs, simple emails, phone formats, etc.).

employee_id_pattern = r'EMP-\d{4}'
ids = ['EMP-0001', 'EMP-12', 'AEMP-0001', 'EMP-12345']

for emp_id in ids:
    print(emp_id, bool(re.fullmatch(employee_id_pattern, emp_id)))

EMP-0001 True
EMP-12 False
AEMP-0001 False
EMP-12345 False

### Exercise: Full String Validation
# Difficulty: Intermediate
codes = ['CS-101', 'MATH-240', 'CS101', 'EE-7']
# A valid course code must be: 2-4 uppercase letters, a dash, then 3 digits.
# 1. Write the regex pattern
# 2. Print each code with True/False using re.fullmatch
### Your code starts here



### Your code ends here

Hide code cell source

 
### Solution
codes = ['CS-101', 'MATH-240', 'CS101', 'EE-7']
pattern = r'[A-Z]{2,4}-\d{3}'
for code_str in codes:
    print(code_str, bool(re.fullmatch(pattern, code_str)))

CS-101 True
MATH-240 True
CS101 False
EE-7 False

8.2.5.4. Quick Reference#

Characters

Pattern

Meaning

Example match

.

Any character (except newline)

c.t → cat, cot

\d

Digit

7

\w

Word character (letter, digit, underscore)

A, x, 9, _

\s

Whitespace (space, tab, newline)

[abc]

Character class: any one of a, b, c

a

[^abc]

Negated class: any character except a, b, c

d

Quantifiers

Pattern

Meaning

Example

*

0 or more

ab* → a, ab, abb

+

1 or more

ab+ → ab, abb

?

0 or 1 (optional)

colou?r → color, colour

{n}

Exactly n

\d{3} → 123

{n,m}

Between n and m

\d{2,4} → 12, 123

*? +?

Lazy (match as little as possible)

<.+?>

Anchors

Pattern

Meaning

^

Start of string (or line with re.M)

$

End of string (or line)

\b

Word boundary

Groups

Syntax

Meaning

(...)

Capturing group

(?:...)

Non-capturing group

(?P<name>...)

Named group

(?=...)

Lookahead

(?<=...)

Lookbehind

Flags

Flag

Shorthand

Meaning

re.IGNORECASE

re.I

Case-insensitive matching

re.MULTILINE

re.M

^/$ match line start/end

re.DOTALL

re.S

. matches newline too

re.VERBOSE

re.X

Allow comments/whitespace in pattern