Every time you write a for loop over a list, a string, or a dictionary, you're relying on something called the iterator protocol behind the scenes. Understanding it unlocks one of Python's most useful features for working with large or even infinite sequences of data: generators.

The Iterator Protocol

An iterable is anything you can loop over (a list, a string, a range). An iterator is the object that actually does the work of producing one value at a time — it implements two special methods: __iter__(), which returns the iterator itself, and __next__(), which returns the next value or raises StopIteration when there's nothing left.

numbers = [10, 20, 30]
it = iter(numbers)

print(next(it))
print(next(it))
print(next(it))
print(next(it))  # raises StopIteration

Output:

10
20
30
Traceback (most recent call last):
StopIteration

A for loop is really just repeatedly calling next() on an iterator and catching StopIteration for you automatically — you never see that mechanism unless you use iter()/next() directly, like above.

Generator Functions

Writing a full class with __iter__ and __next__ just to produce a sequence is a lot of boilerplate. A generator function does the same job with far less code — any function containing a yield statement automatically becomes a generator:

def count_up_to(n):
    i = 1
    while i <= n:
        yield i
        i += 1

for num in count_up_to(5):
    print(num, end=" ")

Output:

1 2 3 4 5

Calling count_up_to(5) doesn't run the function's body immediately — it returns a generator object. Each time the loop asks for the next value, the function resumes exactly where it left off at the last yield, runs until the next yield (or the function ends), and pauses again. This is fundamentally different from a normal function, which runs fully to completion and returns a single value.

Why Generators Save Memory

The real payoff is memory. A function that builds and returns a full list holds every value in memory at once. A generator produces values one at a time, on demand, and never holds the whole sequence in memory simultaneously:

def build_list(n):
    return [i * i for i in range(n)]   # all n values exist in memory at once

def build_generator(n):
    for i in range(n):
        yield i * i                    # one value exists at a time

# build_list(10_000_000) uses roughly 400+ MB of memory
# build_generator(10_000_000) uses only a few hundred bytes

This is exactly why you'd reach for a generator over a list when processing huge files line by line, streaming data from an API, or — as shown with full working code in our generator Fibonacci example — producing a sequence like Fibonacci numbers that could in principle run forever.

Generator Expressions

Just like list comprehensions have a shorthand syntax, generators do too — swap the square brackets for parentheses:

squares = (i * i for i in range(5))
print(sum(squares))  # consumes the generator immediately

Output:

30

A good rule of thumb: if you're going to iterate over a sequence only once and don't need to keep it all in memory, prefer a generator. If you need to index into it, loop over it multiple times, or know its length up front, a list is the right choice.