Generators

11.2. Generators#

Learning goals — By the end of this section you will be able to:

Write generator functions using yield to produce sequences lazily
Explain why generators are memory-efficient compared to lists
Create generator expressions as a one-line alternative to generator functions
Use generators for infinite sequences (e.g., Fibonacci) and large-file processing
Chain generators into efficient data pipelines

11.2.1. Generator Functions #

A generator function looks like a regular function but uses yield instead of return. Each call to next() runs the function body until the next yield, then pauses — preserving all local variables.

def my_gen():
    yield 1   # pauses here on first next()
    yield 2   # pauses here on second next()
    yield 3   # pauses here on third next()
    # function returns → StopIteration is raised automatically

Key differences from a regular function:

Calling a generator function returns a generator object — it does not execute the body
Each next() resumes from where it left off
Values are produced lazily — only when requested

def count_up_to(max_count):
    """Yields integers 1, 2, ..., max_count."""
    n = 1
    while n <= max_count:
        yield n
        n += 1

# Calling the function returns a generator object — nothing executes yet
gen = count_up_to(5)
print(type(gen))          # <class 'generator'>

# Values are produced one at a time
print(next(gen))          # 1
print(next(gen))          # 2

# for loops consume the remaining values
for val in gen:
    print(val, end=' ')   # 3 4 5
print()

# A fresh generator starts from the beginning
print(list(count_up_to(5)))   # [1, 2, 3, 4, 5]

<class 'generator'>
1
2
3 4 5 
[1, 2, 3, 4, 5]

11.2.2. Generator Expressions #

A generator expression has the same syntax as a list comprehension, but uses () instead of []. The result is a generator, not a list; values are computed on demand.

# List comprehension: builds the entire list in memory immediately
squares_list = [x ** 2 for x in range(1_000_000)]   # about 40 MB in total

# Generator expression: produces values one at a time
squares_gen  = (x ** 2 for x in range(1_000_000))   # about 200 bytes

The list’s total has two parts. sys.getsizeof(squares_list) reports only the list’s own array of references, about 8 MB; the one million integer objects it points to take about 32 MB more. The generator stores none of the values, only its current state.

Use a generator expression whenever you only iterate through the result once.

11.2.2.1. `any()` and `all()` with Generator Expressions #

In Chapter 6 you used any() and all() with list comprehensions. Now that you know generator expressions, you can drop the brackets:

# list comprehension — builds the full list, then checks
any([x > 0 for x in values])

# generator expression — stops as soon as the result is known
any(x > 0 for x in values)

The generator version is more efficient for two reasons: it never builds an intermediate list, and both functions short-circuit — any() stops the moment it finds a truthy value; all() stops the moment it finds a falsy one.

data = [4, -1, 7, 0, 3, -2, 8]

# any() stops the moment it finds a True value — no need to check the rest
print(any(x < 0 for x in data))      # True  (stops at -1)

# all() stops the moment it finds a False value
print(all(x > 0 for x in data))      # False (stops at -1)

# Common data-validation patterns
scores = [82, 91, 74, 88, 65]
print(all(s >= 60 for s in scores))  # True  — all passed
print(any(s >= 90 for s in scores))  # True  — at least one A

True
False
True
True

### Exercise: any() and all() with Generator Expressions
sales = [1200, 850, 2100, 430, 975, 1600]

# 1. Use any() with a generator expression to check if any sale exceeds 2000.
# 2. Use all() with a generator expression to check if all sales are above 400.
# 3. Use next() with a generator expression to find the first sale above 1000.
### Your code starts here.



### Your code ends here.

True
True
1200

11.2.3. Generator Pipelines #

A generator pipeline is a chain of small generators in which each stage consumes the output of the stage before it. Every stage does one job: one produces raw records, the next parses them, the next filters them, and a final consumer such as sum(), list(), max(), or a for loop reduces the stream to a result.

The stages are lazy. Building the chain only creates generator objects; no record is read, parsed, or filtered until the consumer asks for its first value. Each request then travels back up the chain, and one value flows down through every stage before the next value is produced.

read_rows  ->  parse  ->  in_region  ->  (amount for ...)  ->  sum()
 raw lines     dicts      East only      numbers               total

11.2.3.1. Building a Pipeline #

The pipeline below totals the East region’s sales from a small block of CSV-style text. The text is an inline string, so you can run the example without any files. Each stage is a generator function that takes an iterable and yields new values, and the last stage is a generator expression that picks out the amount field.

SALES = """region,product,amount
East,apple,120
West,banana,85
East,cherry,210
West,apple,40
East,banana,95"""

def read_rows(text):
    """Stage 1: yield one raw line at a time, skipping the header."""
    lines = iter(text.splitlines())
    next(lines)                      # skip the header row
    for line in lines:
        yield line

def parse(rows):
    """Stage 2: turn each line into a dict with a numeric amount."""
    for row in rows:
        region, product, amount = row.split(",")
        yield {"region": region, "product": product, "amount": int(amount)}

def in_region(records, region):
    """Stage 3: keep only the records from one region."""
    for rec in records:
        if rec["region"] == region:
            yield rec

rows = read_rows(SALES)
records = parse(rows)
east = in_region(records, "East")
amounts = (rec["amount"] for rec in east)   # Stage 4: pick one field

print(type(amounts))   # <class 'generator'>  (nothing has run yet)
print(sum(amounts))    # 425  (sum() pulls every value through the chain)

<class 'generator'>
425

11.2.3.2. Tracing the Flow #

To see the order in which values move, insert a pass-through stage that prints each item and yields it unchanged. Because trace() is itself a generator, you can drop it between any two stages without changing the result.

def trace(label, items):
    """Pass items through unchanged, printing each one."""
    for item in items:
        print(f"{label:>5}: {item}")
        yield item

rows = trace("read", read_rows(SALES))
records = trace("parse", parse(rows))
east = trace("keep", in_region(records, "East"))

total = sum(rec["amount"] for rec in east)
print("total:", total)   # total: 425

 read: East,apple,120
parse: {'region': 'East', 'product': 'apple', 'amount': 120}
 keep: {'region': 'East', 'product': 'apple', 'amount': 120}
 read: West,banana,85
parse: {'region': 'West', 'product': 'banana', 'amount': 85}
 read: East,cherry,210
parse: {'region': 'East', 'product': 'cherry', 'amount': 210}
 keep: {'region': 'East', 'product': 'cherry', 'amount': 210}
 read: West,apple,40
parse: {'region': 'West', 'product': 'apple', 'amount': 40}
 read: East,banana,95
parse: {'region': 'East', 'product': 'banana', 'amount': 95}
 keep: {'region': 'East', 'product': 'banana', 'amount': 95}
total: 425

Read the trace from top to bottom. The first line is read, parsed, and kept before the second line is even read. The West records stop at the filter, so sum() never sees them. At any moment only one record is in flight, which is why the same pipeline works on a file with millions of lines: memory use depends on the size of one record, not the number of records.

11.2.3.3. Iterator Exhaustion #

A generator remembers its position, and it cannot rewind. Once it has raised StopIteration, it is exhausted: every later request ends immediately. This is called iterator exhaustion. It matters in pipelines because a second pass over the same generator silently produces nothing. sum() of an empty stream returns 0 and list() returns [], so no error warns you.

east = in_region(parse(read_rows(SALES)), "East")
amounts = (rec["amount"] for rec in east)

print(sum(amounts))    # 425
print(sum(amounts))    # 0   (the generator is already exhausted)
print(list(amounts))   # []

425
0
[]

There are two fixes, and you choose based on how much data there is:

Rebuild the pipeline. Wrap the chain in a function that returns a fresh generator each time you call it. Each call reads the source again, but memory use stays constant.
Materialize once. Call list() on the pipeline when you need the same values several times and they fit in memory. A list can be iterated as often as you like.

def east_amounts(text):
    """Build a fresh pipeline every time it is called."""
    records = in_region(parse(read_rows(text)), "East")
    return (rec["amount"] for rec in records)

# Fix 1: rebuild the pipeline for each pass
print(sum(east_amounts(SALES)))   # 425
print(max(east_amounts(SALES)))   # 210

# Fix 2: materialize once, then reuse the list
amounts = list(east_amounts(SALES))
print(amounts)                                    # [120, 210, 95]
print(sum(amounts), max(amounts), len(amounts))   # 425 210 3

425
210
[120, 210, 95]
425 210 3

### Exercise: Generator Pipeline
SCORES = """name,quiz,score
Alice,1,88
Bob,1,72
Carol,1,95
Alice,2,91
Bob,2,58
Carol,2,79"""
#   1. Write a generator function `parse_scores(text)` that skips the header
#      and yields one (name, score) tuple per line, with score as an int.
#   2. Write a function `passing(text, cutoff)` that returns a generator
#      expression over parse_scores(text) keeping only scores >= cutoff.
#   3. With cutoff 75, print how many scores pass and the highest passing
#      score. Build a fresh pipeline for each result.
### Your code starts here.




### Your code ends here.

[('Alice', 88), ('Carol', 95), ('Alice', 91), ('Carol', 79)]
4 95

11.2.4. Practical Use Cases #

Generators are most useful when a sequence is too large to build all at once, or has no end at all. The example below represents an endless sequence and takes only as many values as it needs.

11.2.4.1. Infinite sequences #

Generators can represent sequences that have no end — you just take as many values as you need.

import itertools

def fibonacci():
    """Infinite Fibonacci sequence generator."""
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Take only the values you need — never computes the rest
fib = fibonacci()
first_10 = list(itertools.islice(fib, 10))
print("First 10 Fibonacci:", first_10)  # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

# Find the first Fibonacci number > 1000
fib = fibonacci()
big = next(f for f in fib if f > 1000)
print("First Fibonacci > 1000:", big)   # 1597

First 10 Fibonacci: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
First Fibonacci > 1000: 1597

### Exercise: Running Total Generator
#   1. Write a generator function `running_total(numbers)` that yields the
#      cumulative sum after each element.
#      Example: running_total([1, 4, 2, 8]) yields 1, 5, 7, 15.
#   2. Write a generator expression `even_squares_gen` that yields the squares
#      of even numbers from 0 to 19 (i.e., 0, 4, 16, 36, 64, 100, ..., 324).
### Your code starts here.




### Your code ends here.

[1, 5, 7, 15]
[10, 7, 14]
[0, 4, 16, 36, 64, 100, 144, 196, 256, 324]

11.2.5. `yield from`#

yield from delegates yielding to another iterable (including another generator). It is essentially a shorthand for for item in sub: yield item, but also properly forwards send() and throw() calls to the sub-generator.

def flatten(nested):
    """Recursively flatten a nested list structure."""
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)  # delegate to recursive call
        else:
            yield item

data = [1, [2, 3], [4, [5, 6]], 7]
print(list(flatten(data)))   # [1, 2, 3, 4, 5, 6, 7]

[1, 2, 3, 4, 5, 6, 7]

# yield from with multiple sub-generators
def combined(*iterables):
    for it in iterables:
        yield from it

result = list(combined([1, 2], 'ab', range(3)))
print(result)   # [1, 2, 'a', 'b', 0, 1, 2]

[1, 2, 'a', 'b', 0, 1, 2]

11.2.6. More `itertools`#

The itertools module provides iterator building-blocks. Beyond islice, four more are especially useful in data processing.

Function	Description	Example
`chain(*iterables)`	Joins iterables end to end, lazily	`chain('AB', 'CD')` → A, B, C, D
`count(start, step)`	Infinite counter	`count(1, 2)` → 1, 3, 5, …
`cycle(iterable)`	Repeats iterable endlessly	`cycle('AB')` → A, B, A, B, …
`combinations(it, r)`	All r-length combos (no repeat)	`combinations('ABC', 2)` → AB AC BC
`permutations(it, r)`	All ordered r-length arrangements	`permutations('AB', 2)` → AB BA

import itertools

# count: infinite arithmetic sequence
evens = itertools.islice(itertools.count(0, 2), 5)
print('First 5 even numbers:', list(evens))  # [0, 2, 4, 6, 8]

# cycle: round-robin scheduling
servers = ['A', 'B', 'C']
requests = list(itertools.islice(itertools.cycle(servers), 7))
print('Request routing:', requests)   # ['A', 'B', 'C', 'A', 'B', 'C', 'A']

First 5 even numbers: [0, 2, 4, 6, 8]
Request routing: ['A', 'B', 'C', 'A', 'B', 'C', 'A']

chain() connects several iterables into one stream. It reads the first iterable to the end and then moves on to the next one, without copying anything into a new list. You can think of it as the library version of the combined() generator from the yield from section. Because chain() is lazy, it works with generators as well as lists: nothing is read from any input until you ask for values.

chain.from_iterable() does the same job for a single iterable whose items are themselves iterables, such as a list of lists.

import itertools

morning = ['apple', 'banana']                                  # a list
afternoon = (fruit.upper() for fruit in ['cherry', 'date'])    # a generator

all_fruits = itertools.chain(morning, afternoon)
print(type(all_fruits).__name__)   # chain  (nothing has been read yet)
print(list(all_fruits))            # ['apple', 'banana', 'CHERRY', 'DATE']

# chain.from_iterable: flatten one level of a list of lists
baskets = [[1, 2], [3], [4, 5]]
print(list(itertools.chain.from_iterable(baskets)))   # [1, 2, 3, 4, 5]

chain
['apple', 'banana', 'CHERRY', 'DATE']
[1, 2, 3, 4, 5]

import itertools

# combinations: choose without caring about order
teams = list(itertools.combinations(['Alice', 'Bob', 'Carol'], 2))
print('Possible pairs:', teams)
# [('Alice', 'Bob'), ('Alice', 'Carol'), ('Bob', 'Carol')]

# permutations: order matters (e.g. race positions)
podium = list(itertools.permutations(['Gold', 'Silver', 'Bronze'], 2))
print(f'Top-2 orderings: {len(podium)} total')
print(podium[:3])   # first 3 arrangements

Possible pairs: [('Alice', 'Bob'), ('Alice', 'Carol'), ('Bob', 'Carol')]
Top-2 orderings: 6 total
[('Gold', 'Silver'), ('Gold', 'Bronze'), ('Silver', 'Gold')]

### Exercise: itertools
#   1. Use itertools.count() and islice() to generate the first 8
#      multiples of 7 (7, 14, 21, ..., 56).
#   2. Use itertools.cycle() and islice() to simulate dealing cards
#      to 4 players, 3 cards each (total 12 cards from an infinite deck).
#      Players are 'North', 'East', 'South', 'West'.
#   3. Use itertools.combinations() to find all 3-letter combinations
#      from 'ABCDE' and count them.
import itertools

### Your code starts here.



### Your code ends here.

[7, 14, 21, 28, 35, 42, 49, 56]
[('North', 1), ('East', 2), ('South', 3), ('West', 4), ('North', 5), ('East', 6), ('South', 7), ('West', 8), ('North', 9), ('East', 10), ('South', 11), ('West', 12)]
10 combinations
[('A', 'B', 'C'), ('A', 'B', 'D'), ('A', 'B', 'E'), ('A', 'C', 'D'), ('A', 'C', 'E'), ('A', 'D', 'E'), ('B', 'C', 'D'), ('B', 'C', 'E'), ('B', 'D', 'E'), ('C', 'D', 'E')]

11.2.7. Bridge: Async Iteration #

Synchronous generators pause at yield and resume when next() is called — all within a single thread. Async generators do the same thing, but they can also pause to wait for I/O (a network response, a database query, a file read) without blocking other work.

The pattern mirrors what you’ve seen, but uses async def with yield, and requires async for to consume:

import asyncio

async def async_count(n):
    """Yields 0..n-1, simulating a pause between each."""
    for i in range(n):
        await asyncio.sleep(0)   # yields control to the event loop
        yield i

async def main():
    async for value in async_count(5):
        print(value)

asyncio.run(main())   # 0 1 2 3 4

Key vocabulary for future study:

Term	Meaning
`async def`	Declares a coroutine (or async generator if it uses `yield`)
`await`	Pauses the coroutine until the awaited task is ready
`async for`	Iterates over an async iterable; releases control between items
Event loop	The scheduler that runs coroutines concurrently

You don’t need to master asyncio now. The takeaway: the iterator protocol you learned here — __iter__/__next__, generators, yield — is exactly the foundation Python’s async system is built on.

11.2.8. Summary #

Concept	Key idea
Generator function	Uses `yield`; returns a generator object; body runs lazily
`yield`	Pauses function, returns a value; resumes on next `next()` call
Generator expression	`(expr for item in seq if cond)`: lazy and memory-efficient
Generator pipeline	Chain of generators; each stage consumes the previous one lazily until a consumer pulls values
Iterator exhaustion	A finished generator yields nothing on a second pass; rebuild it or materialize it with `list()`
Infinite generator	Uses `while True: yield ...`; take values with `itertools.islice` or `next()`
Memory efficiency	Generators produce one value at a time, so memory stays constant regardless of sequence length
`itertools.islice`	Takes the first N values from any iterator
`itertools.chain`	Joins several iterables into one lazy stream; `chain.from_iterable` flattens one level

Generators

Contents

11.2. Generators#