Chapter 11

Iterators & Generators

11.0 Intro · 11.1 Iterators · 11.2 Generators

← → or Space to navigate · F for fullscreen

11.1 Iterators

The iteration protocol · iter() / next() · built-in lazy iterators · custom classes

The Iterator Protocol

An iterable can be looped over. An iterator remembers where it is.

# What a for loop does internally
nums = [1, 2, 3]
it = iter(nums)          # get iterator

print(next(it))          # 1
print(next(it))          # 2
print(next(it))          # 3
# next(it) → StopIteration

Built-in lazy iterators

Function What it produces
enumerate(seq) (index, value) pairs
zip(a, b) (a_i, b_i) pairs
map(f, seq) f(x) for each x
filter(pred, seq) elements where pred(x) is True
reversed(seq) elements in reverse

All are lazy — values produced on demand.

Custom Iterator Classes

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self           # the iterator IS the iterable here

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

for n in Countdown(5):
    print(n, end=" ")   # 5 4 3 2 1

11.2 Generators

yield · generator expressions · any/all · itertools

Generator Functions

A generator function uses yield instead of return. It produces values lazily — one at a time.

def count_up_to(n):
    i = 1
    while i <= n:
        yield i
        i += 1

gen = count_up_to(3)
print(next(gen))   # 1
print(next(gen))   # 2
for val in gen:
    print(val)     # 3

Memory efficiency

import sys

# List: builds everything in memory
lst = [x**2 for x in range(1_000_000)]
print(sys.getsizeof(lst))   # ~8 MB

# Generator: produces one value at a time
gen = (x**2 for x in range(1_000_000))
print(sys.getsizeof(gen))   # ~200 bytes

Use a generator when you iterate once and don't need to store all values.

any() and all() with Generator Expressions

Dropping the brackets avoids building an intermediate list and enables short-circuit evaluation:

data = [4, -1, 7, 0, 3, -2]

# any() stops as soon as it finds True
print(any(x < 0 for x in data))      # True  (stops at -1)

# all() stops as soon as it finds False
print(all(x > 0 for x in data))      # False (stops at -1)

# Common validation patterns
scores = [82, 91, 74, 88, 65]
print(all(s >= 60 for s in scores))  # True — all passed
print(any(s >= 90 for s in scores))  # True — at least one A

Generator Pipelines

Each stage consumes the previous one lazily. Nothing runs until the consumer (sum()) pulls values.

def read_rows(text):
    lines = iter(text.splitlines())
    next(lines)                  # skip header
    for line in lines:
        yield line

def parse(rows):
    for row in rows:
        region, product, amount = row.split(",")
        yield {"region": region, "amount": int(amount)}

def in_region(records, region):
    for rec in records:
        if rec["region"] == region:
            yield rec

# SALES: CSV text with a header, "region,product,amount"
east = in_region(parse(read_rows(SALES)), "East")
amounts = (rec["amount"] for rec in east)
print(sum(amounts))   # 425

Iterator exhaustion

print(sum(amounts))   # 0   (already exhausted)
print(list(amounts))  # []

# Fix: rebuild, or materialize once
def east_amounts(text):
    recs = in_region(parse(read_rows(text)), "East")
    return (rec["amount"] for rec in recs)

amounts = list(east_amounts(SALES))
print(sum(amounts), max(amounts))   # 425 210

One record travels through every stage before the next line is read. A finished generator cannot rewind.

itertools Highlights

Function Description
islice(it, n) Take first n items
chain(a, b) Concatenate iterables
count(start, step) Infinite counter
cycle(seq) Repeat sequence infinitely
combinations(seq, r) r-length combos
permutations(seq, r) r-length ordered arrangements 
import itertools

# Infinite Fibonacci
def fib():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

first_10 = list(itertools.islice(fib(), 10))
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

# Combinations
pairs = list(itertools.combinations("ABCD", 2))
# [('A','B'), ('A','C'), ...]

Chapter 11 — Quick Reference

Concept Key syntax / notes
Iterator protocol __iter__ returns self; __next__ returns next value
Manual iteration it = iter(seq) → next(it) → StopIteration
Built-in lazy enumerate, zip, map, filter, reversed
Generator function def f(): yield value
Generator expression (expr for x in seq) — lazy, O(1) memory
any / all Drop brackets for short-circuit: any(x > 0 for x in seq)
yield from Delegate to sub-generator
itertools islice, chain, count, cycle, combinations

End of Chapter 11

Next: Chapter 12 — APIs

HTTP · requests · JSON · authentication · pagination