Generators and yield

What problem are we solving?

Suppose you want the first ten square numbers: 0, 1, 4, 9, 16, and so on. The obvious approach is to build a list (an ordered collection of values held in memory) and fill it up:

squares = []
for n in range(10):
    squares.append(n * n)
print(squares)  # -> [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

This works. But it insists on computing and storing all ten values before you use even the first one. For ten numbers that is fine. For ten million numbers it means ten million values sitting in memory at once. And if you only end up looking at the first three, the other millions were computed for nothing.

A generator is a tool that produces values one at a time, only when asked, and forgets each one after you move past it. Nothing is stored ahead of time. This lesson explains what a generator is, how the keyword yield creates one, and when reaching for one is the right call.

Lazy evaluation: values produced on demand

Lazy evaluation means work is delayed until the result is actually needed, instead of done all up front. The opposite is eager evaluation, which is what the list above does: it computes everything immediately.

A generator is lazy. It hands you one value, then pauses and waits. Only when you ask for the next value does it do more work. This is the single idea behind everything that follows.

A generator function with yield

A function is a named block of reusable code. A normal function uses return to send back one result and finish. A generator function looks almost the same, but it uses the keyword yield instead of return. The moment a function contains yield, Python treats it as a generator.

def squares_up_to(limit):
    n = 0
    while n < limit:
        yield n * n   # hand back one value, then pause here
        n += 1

Calling this function does not run the body. It immediately returns a generator object (a value that knows how to produce the sequence on request):

gen = squares_up_to(4)
print(gen)  # -> <generator object squares_up_to at 0x...>

Nothing has been computed yet. To pull values out, iterate over it. Iterate means to visit each value in turn, which a for loop does automatically:

for value in squares_up_to(4):
    print(value)
# -> 0
# -> 1
# -> 4
# -> 9

Each time the loop needs a value, the generator runs forward until it hits yield, produces that value, and pauses. The loop ends when the function body finishes.

State is paused and resumed at each yield

The unusual part is what happens at yield. A normal function, once it returns, is gone; its local variables disappear. A generator instead freezes at the yield line. Its local variables (here n) keep their values, and execution resumes on the next line the next time you ask for a value.

You can drive a generator by hand with the built-in next function, which asks for exactly one value:

gen = squares_up_to(3)
print(next(gen))  # -> 0   (runs to first yield, pauses)
print(next(gen))  # -> 1   (resumes, n is remembered as 1, runs to yield)
print(next(gen))  # -> 4   (resumes again, n is 2)

When the generator has no more values, next raises StopIteration, a signal that the sequence is finished. A for loop catches this signal for you, which is why you rarely call next directly:

print(next(gen))  # -> StopIteration (nothing left to produce)

Picture the back-and-forth between your loop and the generator: the loop asks for a value, the generator hands one over and freezes, and this repeats until there is nothing left to give.

sequenceDiagram
    participant Loop as for loop
    participant Gen as generator
    Loop->>Gen: next()
    Gen-->>Loop: yield 0 (pause)
    Loop->>Gen: next()
    Gen-->>Loop: yield 1 (pause, n remembered)
    Loop->>Gen: next()
    Gen-->>Loop: yield 4 (pause)
    Loop->>Gen: next()
    Gen-->>Loop: StopIteration (done)

Why this saves memory

Compare the memory used by the two approaches. Building a full list of n values needs room for all n at once. In Big-O notation, which describes how cost grows as input grows, that is O(n) space (space proportional to the number of values).

A generator holds only its current position and local variables, no matter how many values it will eventually produce. That is O(1) space (constant space, independent of n). The total work to produce all values is the same either way, O(n) time, but the memory footprint is dramatically smaller.

# Eager: a real list of 10 million ints lives in memory.
total = sum([n * n for n in range(10_000_000)])   # square brackets build a list

# Lazy: values are produced and discarded one at a time.
total = sum(n * n for n in range(10_000_000))      # no brackets: a generator

Both compute the same sum. The second version never holds more than one square at a time, so its memory use stays flat while the first allocates the whole list.

Generator expressions

The second line above uses a generator expression: a compact way to write a generator inline, without defining a function. It looks like a list comprehension (a short syntax for building a list) but with round parentheses instead of square brackets.

list_version = [n * n for n in range(5)]   # a list, built now
gen_version  = (n * n for n in range(5))   # a generator, produces on demand

print(list_version)  # -> [0, 1, 4, 9, 16]
print(gen_version)   # -> <generator object <genexpr> at 0x...>
print(list(gen_version))  # -> [0, 1, 4, 9, 16]   (list() pulls every value out)

When a generator expression is the only argument to a function, you can drop the extra parentheses, as in sum(n * n for n in range(5)).

Infinite generators with a stop condition

Because a generator only produces values on demand, it can describe an endless sequence without ever trying to build it all. Here is a generator that counts up forever:

def count_from(start):
    n = start
    while True:        # never stops on its own
        yield n
        n += 1

while True would hang if you tried to collect every value, but with a generator you simply stop asking. The caller controls when to quit. break exits a loop early:

for n in count_from(100):
    if n > 103:
        break          # the caller decides where to stop
    print(n)
# -> 100
# -> 101
# -> 102
# -> 103

The generator itself has no end; the stop condition lives in the code that consumes it. This pattern is common: an unbounded source paired with a consumer that takes only what it needs.

When to use generators

Reach for a generator when:

• The full sequence would be large and you process items one at a time (log lines, rows from a file, records from a network response). You save O(n) memory.
• The sequence is infinite or of unknown length.
• You might stop early and want to avoid computing values you will never look at.
• You are chaining steps in a pipeline, where each stage transforms items as they flow through.

Prefer a plain list when:

• The data is small and simplicity matters more than memory.
• You need to use the values more than once. A generator is single-use: once you iterate to the end, it is exhausted and produces nothing further. To reuse the values, store them in a list.

gen = squares_up_to(3)
print(list(gen))  # -> [0, 1, 4]
print(list(gen))  # -> []   (already exhausted; nothing left)

• You need random access, such as data[5] or len(data). A generator has no index and no length; it only moves forward.

Common pitfalls

• A generator is consumed once. After a for loop or list() runs it to the end, it is empty. Looping over the same generator object again yields nothing, as shown above. Recreate it (call the function again) if you need a fresh pass.
• Round vs square brackets. (x for x in items) is a lazy generator; [x for x in items] builds a full list immediately. It is an easy typo with very different memory behavior.
• len() and indexing do not work. len(gen) raises TypeError and gen[0] raises TypeError. If you need those, convert to a list first with list(gen) (which of course brings back the O(n) memory cost).
• Printing a generator shows the object, not the values. print(gen) prints something like <generator object ...>. Use list(gen) or a loop to see the contents.
• An infinite generator with no stop condition hangs. list(count_from(0)) never finishes. Always pair an endless generator with a break or a bounded consumer.

Practice

1. Write a generator function evens(limit) that yields the even numbers 0, 2, 4, … up to but not including limit. Confirm list(evens(10)) gives [0, 2, 4, 6, 8].
2. Rewrite this list comprehension as a generator expression and use it inside sum(...): [len(word) for word in ["hi", "there", "you"]]. It should total 10.
3. Write an infinite generator powers_of_two() that yields 1, 2, 4, 8, 16, … Then use it with a loop and break to print every power of two below 1000.