◀ Course contents Part 4 · Module 4-03

Iterators & Generators

Producing values one at a time, only when asked

Every for loop you have written has been quietly using a protocol you have never seen. This module shows it, and then shows the thing it makes possible: a function that produces a sequence lazily, one value at a time, without ever building the whole thing. That is how you process a file larger than memory, and how a pipeline of transformations costs one pass instead of four.

Ready?

What a `for` Loop Actually Does

Two words that sound alike and are not. An iterable is something that can produce an iterator — a list, a string, a dictionary, a file. An iterator is the thing that keeps the place.

numbers = [1, 2]

it = iter(numbers)   # ask the iterable for an iterator
next(it)             # 1
next(it)             # 2
next(it)             # StopIteration

A for loop is exactly that, with the exception caught: call iter(), call next() until StopIteration arrives, and stop. Nothing else is happening.

next() takes an optional second argument, which is returned instead of raising when there is nothing left:

next(iter([]), "nothing")   # "nothing"

The distinction matters because a list can be walked over and over — each for loop asks for a fresh iterator — while an iterator itself has one position and moves only forward.

A file is an iterator

Which is why looping over the same open file twice gives you every line and then nothing at all. It is not a list that happens to be lazy; it is a position that has reached the end.

Quick check

What ends a for loop over a list?

`yield`: a Function That Pauses

A function containing yield is a generator function. Calling it runs none of the body — it hands back a generator object and waits.

def counter(n):
    print("starting")
    for i in range(n):
        yield i

g = counter(3)        # prints nothing
next(g)               # NOW "starting" is printed, and 0 comes back

Each yield hands a value out and pauses, keeping every local variable exactly as it was. The next next() resumes on the line after the yield. When the function finally returns, StopIteration is raised and the loop ends.

That pausing is the whole trick. A generator holds one value and a position, whatever the length of what it is producing — so this is perfectly reasonable:

def naturals():
    n = 1
    while True:
        yield n
        n += 1

An infinite sequence, costing nothing until something asks. Taking a finite piece of it is itertools.islice, or a break.

Quick check

How much of the body runs when a generator function is called?

Chaining Them Into a Pipeline

Because a generator both consumes and produces lazily, generators plug into each other — and the whole chain costs one pass over the data, holding one item at a time.

def non_blank(lines):
    for line in lines:
        if line.strip():
            yield line.strip()

def errors_only(lines):
    for line in lines:
        if line.startswith("ERROR"):
            yield line

for line in errors_only(non_blank(raw_lines)):
    ...

With lists, each of those steps would build a complete new list. With generators nothing is built at all: each next() at the end pulls one item through the whole chain.

yield from hands over to another iterable, yielding everything it produces. It is how you flatten, and how one generator delegates to another:

def flatten(rows):
    for row in rows:
        yield from row     # instead of an inner for loop

And itertools has the pieces you would otherwise write: islice to take the first n, chain to run several iterables end to end, count for an endless sequence of numbers.

Quick check

Three generators chained, over a million-line file. How many lines are held at once?

Walked Once, and Then Empty

A generator is an iterator, so it has one position and it only moves forward. Once exhausted it stays exhausted — and, crucially, it does not complain:

g = counter(3)
list(g)     # [0, 1, 2]
list(g)     # []      — no error, no warning

This is the bug people actually hit. Code that works when it uses a result once starts producing empty output the day somebody adds a second use — a count and then a loop, say — and there is nothing in the traceback because there is no traceback.

Two fixes, and the choice is a real one:

Materialise it

items = list(gen) once, then use the list as often as you like. Costs the memory the generator was avoiding.

Call the function again

Keep the generator function around rather than one generator object, and get a fresh one per pass. Costs a second read of the source.

And the honest limits. A generator has no len(), no indexing, and no way back. If you need any of those, you need a list — and the memory argument only pays when the data is genuinely large or genuinely endless. For four hundred rows, build the list and stop thinking about it.

When it is worth it

A file bigger than memory. An endless stream. A pipeline where an early filter throws most of the data away. Those are the cases. A three-element list dressed up as a generator is a slower list with fewer features.

Laziness buys memory and costs re-use. Know which one you needed.

Quick check

You count a generator's items and then loop over it. What does the loop see?

0 of 9 completed

Real Python runs right here in your browser — nothing to install, nothing sent to a server. The interpreter downloads once the first time you press Run, then stays cached.

To do

iter() asks an iterable for an iterator. next() pulls the following value out of it, and raises StopIteration when there is nothing left.

That is the whole of what a for loop does: call iter(), call next() until the exception arrives, catch it, stop.

next() also takes a second argument, returned instead of raising when the iterator is finished.

Your task: walk the list by hand and print four lines — the two values, the word StopIteration when it runs out, and a default from an empty iterator:

a
b
StopIteration
nothing

your_code.py

letters = ["a", "b"]

it =

# Two values, then the exception, then a default.

Python

Hint

it = iter(letters), then print(next(it)) twice. Wrap the third call in try/except StopIteration and print the word. The last line is next(iter([]), "nothing").

Output

To do

Each for loop over a list asks for a fresh iterator, so a list can be walked as often as you like. An iterator has one position and moves only forward.

Your task: show both halves. Walk the list twice, then walk one iterator twice, collecting what each pass sees:

[1, 2, 3]
[1, 2, 3]
[1, 2, 3]
[]

The fourth line is empty because the third consumed the iterator — with no error to say so.

your_code.py

numbers = [1, 2, 3]
it = iter(numbers)

first_pass =
second_pass =
third_pass =
fourth_pass =

print(first_pass)
print(second_pass)
print(third_pass)
print(fourth_pass)

Python

Hint

The first two are list(numbers) each time — a list hands out a new iterator per pass. The last two are both list(it), on the same iterator.

Output

To do

A function containing yield is a generator function. Each yield hands a value out and pauses, keeping every local variable exactly where it was.

def steps():
    yield "build"
    yield "test"

Your task: write countdown(n) yielding n down to 1 and then the string "go". Print the whole thing as a list, then loop over a fresh one:

[3, 2, 1, 'go']
2
1
go

The loop starts at 2, so call it with a smaller number the second time.

your_code.py

def countdown(n):
    # Yield n down to 1, then "go".
    pass

print(list(countdown(3)))

for step in countdown(2):
    print(step)

Python

Hint

A while loop or a range walking downwards, with yield inside it, then one final yield "go" after the loop. Nothing is returned — the values leave through yield.

Output

To do

Calling a generator function runs no part of the body. It builds a generator object and waits. Even a print on the first line does not happen until the first value is asked for.

This surprises people, and it is the whole basis of laziness: work happens when a value is needed and not before.

Your task: demonstrate it. Build the generator, print a line to show nothing has happened, then consume it:

built, nothing has run
starting
0
1
2

your_code.py

def counter(n):
    print("starting")
    for i in range(n):
        yield i

g =

print("built, nothing has run")

# Now consume it.

Python

Hint

g = counter(3) builds it. Then a for loop over g prints each value — and "starting" appears at that moment, not before.

Output

To do

A generator holds one value and a position, so producing an endless sequence is perfectly reasonable — nothing is computed until it is asked for.

def naturals():
    n = 1
    while True:
        yield n
        n += 1

Taking a finite piece is itertools.islice(source, n), or a break. What you must never do is call list() on it.

Your task: write naturals(), then take the first five with islice, and the first three squares over 50 with a loop and a break:

[1, 2, 3, 4, 5]
[64, 81, 100]

your_code.py

from itertools import islice

def naturals():
    # An endless sequence of 1, 2, 3, ...
    pass

first_five =

big_squares = []
# Loop naturals(), collect squares over 50, stop at three.

print(first_five)
print(big_squares)

Python

Hint

naturals is a while True loop yielding n and incrementing it. first_five = list(islice(naturals(), 5)). For the squares, loop over naturals(), skip when n * n is 50 or less, append otherwise, and break once big_squares has three.

Output

To do

A generator both consumes and produces lazily, so they chain — and the whole chain costs one pass, holding one item at a time. With lists, each stage would build a complete new list.

for line in errors_only(non_blank(raw)):
    ...

Your task: write two generators — non_blank(lines), which yields the stripped non-empty lines, and errors_only(lines), which yields only those starting with ERROR. Chain them, then print the result and the count:

['ERROR db timeout', 'ERROR disk full']
2

Neither generator may build a list.

your_code.py

raw = ["INFO boot", "  ", "ERROR db timeout", "", "  ERROR disk full  ", "WARN slow"]

def non_blank(lines):
    # Yield the stripped, non-empty lines.
    pass

def errors_only(lines):
    # Yield only the ones starting with ERROR.
    pass

found = list(errors_only(non_blank(raw)))

print(found)
print(len(found))

Python

Hint

Each one is a for loop with an if and a yield — no list, no append, no return. non_blank strips first so errors_only sees a line with no leading spaces.

Output

To do

yield from yields everything another iterable produces. It replaces an inner loop, and it is how one generator delegates to another.

def flatten(rows):
    for row in rows:
        yield from row

Your task: write flatten(rows) using yield from, and numbered(rows) which delegates to it and yields (position, value) pairs starting at 1:

['a', 'b', 'c', 'd']
[(1, 'a'), (2, 'b'), (3, 'c'), (4, 'd')]

your_code.py

rows = [["a", "b"], ["c"], [], ["d"]]

def flatten(rows):
    # yield from each row.
    pass

def numbered(rows):
    # Delegate to flatten, and number what comes out.
    pass

print(list(flatten(rows)))
print(list(numbered(rows)))

Python

Hint

flatten is a for loop with yield from row inside. numbered can loop enumerate(flatten(rows), start=1) and yield each pair — the tuple is what enumerate already hands you.

Output

To do

This code counts the results and then reports them, and reports nothing at all. The count consumed the generator; the loop found an iterator that had already reached the end.

No error is raised. Code like this works for months, until somebody adds a second use of a result that was only ever used once.

Your task: fix it so both the count and the listing work. Keep squares a generator function — the fix is at the call site, not in the generator.

Count: 3
1
4
9

your_code.py

def squares(n):
    for i in range(1, n + 1):
        yield i * i

results = squares(3)

print("Count:", sum(1 for _ in results))

for value in results:
    print(value)

Python

Hint

Two honest options. Either materialise it once — results = list(squares(3)) — and use the list twice, or call squares(3) again for the second pass. The list is the right call here: three items, used twice.

Output

To do

Laziness buys memory and costs re-use. When the data is small and the code wants len(), indexing, or a second pass, a generator is a slower list with fewer features.

The code below wants all three, and works around the generator three times — a sum(1 for _ in ...) to count, a rebuild to index, and another rebuild to loop.

Your task: make load return a list, and simplify all three call sites:

3
whisk
['tea', 'whisk', 'cloth']

your_code.py

def load(raw):
    for name in raw.split(","):
        yield name.strip()

raw = "tea, whisk ,cloth"

print(sum(1 for _ in load(raw)))
print(list(load(raw))[1])
print(list(load(raw)))

Python

Hint

Make load return a list — a comprehension over raw.split(",") does it in one line. Then the three calls become len(names), names[1] and names, with load called once.

Output

The streaming pipeline

To do

A log arrives as a stream — in this exercise a list, in production a file too big to hold. Build a pipeline of generators that walks it once, throwing away what it does not need as it goes, and takes only as much as the report asks for.

Write four generator functions. None of them may build a list, call len(), or return anything — every value leaves through yield:

clean(lines) — yields the stripped lines, dropping blanks and anything starting with #
parse(lines) — yields (level, message), splitting each line with maxsplit=2 and dropping the timestamp
only(entries, level) — yields the entries matching one level
messages(entries) — yields just the message from each entry

Then chain all four and take the first two errors with itertools.islice. Also count every entry — with a second pass, since the first one is spent.

Print exactly five lines:

db timeout after 30s
disk full on /var
Errors shown: 2
Entries in total: 6
Levels: ERROR, INFO, WARN

The last line is every distinct level, sorted — which needs a third pass, because the other two are gone.

your_code.py

from itertools import islice

raw = [
    "2026-09-02 INFO   boot complete",
    "  ",
    "2026-09-02 ERROR  db timeout after 30s",
    "# rotated by logctl",
    "2026-09-02 WARN   slow query 1.4s",
    "2026-09-02 ERROR  disk full on /var",
    "2026-09-02 INFO   retry scheduled",
    "2026-09-02 ERROR  db timeout after 30s",
]

def clean(lines):
    # Yield stripped lines; drop blanks and comments.
    pass

def parse(lines):
    # Yield (level, message).
    pass

def only(entries, level):
    # Yield the entries at one level.
    pass

def messages(entries):
    # Yield the message from each entry.
    pass

# Chain them, take the first two errors, then count and list the levels.

# Five lines of output.

Python

Hint

Each generator is a for loop with a yield and no collecting. In parse, line.split(maxsplit=2) gives three pieces and you keep the last two. Chain with messages(only(parse(clean(raw)), "ERROR")) and wrap that in islice(..., 2). The count is a second full chain — sum(1 for _ in parse(clean(raw))) — and the levels are a set built from a third, sorted and joined with ", ".

Output

Sign in to open this module

Iterators & Generators

What a for Loop Actually Does

A file is an iterator

Quick check

yield: a Function That Pauses

Quick check

Chaining Them Into a Pipeline

Quick check

Walked Once, and Then Empty

Materialise it

Call the function again

When it is worth it

Quick check

Doing a for loop by hand

A list can be walked twice; an iterator cannot

A function that pauses

Calling it runs none of it

An infinite sequence that costs nothing

Generators plugged into each other

Handing over to another iterable

The empty output with no traceback

A generator that should have been a list

The streaming pipeline

Notification

What a `for` Loop Actually Does

`yield`: a Function That Pauses