Generators and yield

You will be able to write functions that hand back values one at a time, chain them into pipelines that read a stream without holding it in memory, and say what a for loop does underneath.

In Python in Practice, for walked lists, strings, dicts and sys.stdin, and map and filter gave back objects you could use only once. They all share one protocol, which you can now write yourself.

The iterator protocol

iter(x) asks an iterable for an iterator; next(it) asks the iterator for its next value and raises StopIteration when none is left. A for loop is those two calls inside a try/except StopIteration.

A list is iterable but not an iterator: each iter(lst) makes a fresh one. An iterator returns itself from iter(). A class can implement the protocol by hand:

Generator functions

A function containing yield is a generator function. The same countdown, without the bookkeeping:

Calling it runs nothing and returns a generator object. Each next() runs the body to the next yield, hands out that value and freezes, locals included:

next(g, default) returns the default instead of raising once the body finishes. return ends a generator early; raise StopIteration inside one becomes RuntimeError.

One pass, shared position

A generator is its own iterator, so it is spent after one pass, silently (sum(g) would now be 0):

Two loops over one iterator split the items between them, and break consumes the item it stopped on:

On purpose, header = next(rows) takes the first item and a loop over rows sees the rest.

Generators as __iter__, and yield from

A generator function can serve as __iter__, and yield from delegates to another iterable:

Each iter(s) starts a fresh generator, so s loops repeatedly; Countdown returned self and cannot.

Infinite streams and pipelines

A generator may never finish. Bound it with itertools.islice:

Generators that consume generators form a pipeline; each item travels the whole chain before the next is pulled:

Given lines INFO: boot, ERROR: disk full, INFO: ok, ERROR: no route, it prints disk full, no route. Memory use is one line, however long the input. (send() can also push values into a generator; that is coroutine territory.)

Common mistakes

• Reading state before the stream has run. A counter updated inside a generator is 0 until something pulls items through it. Print it after the loop.
• Looping twice over a spent generator. The second pass yields nothing and raises nothing, so a total comes out as 0.
• Peeking with for ... break. The item you broke on is gone; use next() to take exactly one.

Your exercise

Write two generators, one that yields parsed integers and one that consumes it, and let the starter's loop pull everything through. The classic slip is reading the integer count before the loop has finished.