Iterators, Generators, and Comprehensions

This section documents the current Python release line as published at the official Python documentation, which is the reference these pages are written and verified against. No specific patch version is pinned.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Every for loop in Python — and every comprehension — runs on top of the same small protocol: get an iterator from an iterable, then call it repeatedly until it signals it is done. This page walks through that protocol, the two constructs built directly on it (generator functions and generator expressions), and comprehensions as sugar over the same machinery.

The Iterator Protocol

An iterable is any object with an iter method returning an iterator; an iterator is any object with both iter (returning itself) and next, which returns the next value or raises StopIteration when exhausted. This is documented in the tutorial’s Iterators section and, at the data-model level, in Iterator Types.

A for loop is shorthand for calling iter() once and next() repeatedly, catching StopIteration to know when to stop:

numbers = [10, 20, 30]

iterator = iter(numbers)     # get an iterator from the list (an iterable)
print(next(iterator))        # 10
print(next(iterator))        # 20
print(next(iterator))        # 30
print(next(iterator))        # raises StopIteration

next() accepts an optional default, returned instead of raising:

iterator = iter([1, 2])
next(iterator)                # 1
next(iterator)                # 2
next(iterator, "done")        # 'done' -- no StopIteration raised

Writing a class that implements the protocol directly makes the two methods explicit. iter returns self (the object is its own iterator), and next raises StopIteration once there is nothing left:

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

for value in Countdown(3):
    print(value)          # 3, 2, 1

Because Countdown implements iter, it works anywhere an iterable is expected — for loops, list(…), sum(…), and comprehensions all drive it through the exact same iter()/next() calls shown above.

Generator Functions and Generator Expressions

Writing a full iterator class is verbose for anything beyond a toy example. A generator function — any function containing yield — lets Python build the iterator for you: calling it returns a generator object without running any of the function body, and each next() call resumes execution up to the next yield. See the tutorial’s Generators section and the formal yield expressions reference.

def countdown(start):
    current = start
    while current > 0:
        yield current
        current -= 1

gen = countdown(3)      # no code has run yet
print(next(gen))        # 3 -- runs up to the first `yield`
print(next(gen))        # 2 -- resumes after `yield`, runs to the next one
print(next(gen))        # 1
print(next(gen))        # raises StopIteration -- the function returned

countdown above replaces the entire Countdown class from the previous section in four lines, and iter/next/StopIteration are all generated automatically.

A generator expression is the same idea written inline, with the comprehension syntax but parentheses instead of brackets. It is documented under Generator expressions:

squares = (n ** 2 for n in range(5))
print(next(squares))    # 0
print(next(squares))    # 1
print(list(squares))    # [4, 9, 16] -- the rest, consuming the generator

Why Generators Are Memory-Efficient

A list comprehension builds the entire list in memory before you use any of it. A generator produces one value at a time, on demand — lazy evaluation — so it never holds more than the current value in memory, no matter how large the sequence is:

# Materializes 10 million ints in a list before summing -- a large, real allocation.
total = sum([n for n in range(10_000_000)])

# Produces one int at a time and discards it after summing -- effectively O(1) memory.
total = sum(n for n in range(10_000_000))

Both compute the same total, but only the generator expression scales to an unbounded or very large source (such as lines streamed from a file) without exhausting memory.

Comprehensions as Sugar over Iteration

List, set, and dict comprehensions all desugar to the same iter()/next() loop described above — documented together with for loops in the tutorial’s List Comprehensions section and, formally, under Displays for lists, sets and dictionaries:

squares = [n ** 2 for n in range(5)]        # list:  [0, 1, 4, 9, 16]
uniques = {n % 3 for n in range(6)}         # set:   {0, 1, 2}
by_n = {n: n ** 2 for n in range(5)}        # dict:  {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

Conceptually, squares above runs exactly like this explicit loop — the comprehension is syntactic sugar over the same iteration:

squares = []
for n in range(5):
    squares.append(n ** 2)

Nested Comprehensions

A comprehension can iterate over more than one for clause, evaluated left to right like nested loops — useful for flattening or for a Cartesian product:

matrix = [[1, 2, 3], [4, 5, 6]]
flattened = [value for row in matrix for value in row]
# [1, 2, 3, 4, 5, 6]

pairs = [(x, y) for x in range(3) for y in range(2) if x != y]
# [(0, 0), (0, 1), (1, 1), (2, 0), (2, 1)]

A comprehension nested inside another (one comprehension as the expression of the outer one) builds a list of lists instead of flattening them:

matrix = [[1, 2, 3], [4, 5, 6]]
transposed = [[row[i] for row in matrix] for i in range(3)]
# [[1, 4], [2, 5], [3, 6]]

Comprehension vs. Generator Expression

Prefer a generator expression over a list comprehension whenever the result will only be consumed once, in order — passed straight into sum(), any(), all(), max(), or a for loop — since nothing needs the intermediate list to exist:

lines = ["10", "20", "thirty", "40"]

# List comprehension: builds an intermediate list just to throw it away after summing.
total = sum([int(x) for x in lines if x.isdigit()])

# Generator expression: same result, no intermediate list ever materialized.
total = sum(int(x) for x in lines if x.isdigit())

Reach for the list (or set/dict) comprehension instead when you need the result more than once, need its length or indexing, or need it to be a real list/set/dict for an API that requires one.

From `iter()` to `StopIteration`

Every mechanism on this page — the explicit iterator = iter(obj) loop, a for loop, a generator function, a generator expression, and every comprehension — follows the same state transitions:

stateDiagram-v2 [*] --> Iterator: iter(obj) Iterator --> Iterator: next(iterator) returns a value Iterator --> Exhausted: next(iterator) raises StopIteration Exhausted --> [*]: loop ends