2.6. Generator Yield From

  • Since Python 3.3: PEP 380 -- Syntax for Delegating to a Subgenerator

  • Helps with refactoring generators

  • Useful for large generators which can be split into smaller ones

  • Delegation call

  • yield from terminates on GeneratorExit from other function

  • The value of the yield from expression is the first argument to the StopIteration exception raised by the iterator when it terminates

  • Return expr in a generator causes StopIteration(expr) to be raised upon exit from the generator

>>> def run():
...     yield from generator1()
...     yield from generator2()

2.6.1. Example

>>> def run():
...     for x in range(0, 3):
...         yield x
...     for x in range(10, 13):
...         yield x
>>> def generator1():
...     for x in range(0, 3):
...         yield x
>>>
>>> def generator2():
...     for x in range(10, 13):
...         yield x
>>>
>>> def run():
...     yield from generator1()
...     yield from generator2()

2.6.2. Why?

This will not work at all! Mind that no code is executed by a function after the return keyword.

>>> def generator1():
...     for x in range(0, 3):
...         yield x
>>>
>>> def generator2():
...     for x in range(10, 13):
...         yield x
>>>
>>> def run():
...     return generator1()
...     return generator2()

This will yield generators (not their values):

>>> def generator1():
...     for x in range(0, 3):
...         yield x
>>>
>>> def generator2():
...     for x in range(10, 13):
...         yield x
>>>
>>> def run():
...     yield generator1()
...     yield generator2()

2.6.3. Execute

>>> def generator1():
...     for x in range(0, 3):
...         yield x
>>>
>>> def generator2():
...     for x in range(10, 13):
...         yield x
>>>
>>> def run():
...     yield from generator1()
...     yield from generator2()
>>>
>>>
>>> result = run()
>>>
>>> type(result)
<class 'generator'>
>>>
>>> next(result)
0
>>> next(result)
1
>>> next(result)
2
>>> next(result)
10
>>> next(result)
11
>>> next(result)
12
>>> next(result)
Traceback (most recent call last):
StopIteration

2.6.4. Itertools Chain

The code is equivalent to itertools.chain():

>>> from itertools import chain
>>>
>>>
>>> def generator1():
...     for x in range(0, 3):
...         yield x
>>>
>>> def generator2():
...     for x in range(10, 13):
...         yield x
>>>
>>> def run():
...     for x in chain(generator1(), generator2()):
...         yield x
>>>
>>>
>>> result = run()
>>>
>>> type(result)
<class 'generator'>
>>>
>>> list(result)
[0, 1, 2, 10, 11, 12]

2.6.5. Delegation call

yield from turns ordinary function, into a delegation call:

>>> def worker():
...     return [1, 2, 3]
>>>
>>> def run():
...     yield from worker()
>>>
>>>
>>> result = run()
>>>
>>> next(result)
1
>>> next(result)
2
>>> next(result)
3
>>> next(result)
Traceback (most recent call last):
StopIteration
>>> def worker():
...     return [x for x in range(0,3)]
>>>
>>> def run():
...     yield from worker()
>>>
>>>
>>> result = run()
>>>
>>> next(result)
0
>>> next(result)
1
>>> next(result)
2
>>> next(result)
Traceback (most recent call last):
StopIteration

2.6.6. Yield From Sequences

>>> def run():
...     yield from [0, 1, 2]
>>>
>>>
>>> result = run()
>>>
>>> type(result)
<class 'generator'>
>>>
>>> next(result)
0
>>> next(result)
1
>>> next(result)
2
>>> next(result)
Traceback (most recent call last):
StopIteration

2.6.7. Yield From Comprehensions

>>> def run():
...     yield from [x for x in range(0,3)]
>>>
>>>
>>> result = run()
>>>
>>> type(result)
<class 'generator'>
>>>
>>> next(result)
0
>>> next(result)
1
>>> next(result)
2
>>> next(result)
Traceback (most recent call last):
StopIteration

2.6.8. Yield From Generator Expression

>>> def run():
...     yield from (x for x in range(0,3))
>>>
>>>
>>> result = run()
>>>
>>> type(result)
<class 'generator'>
>>>
>>> next(result)
0
>>> next(result)
1
>>> next(result)
2
>>> next(result)
Traceback (most recent call last):
StopIteration

2.6.9. Conclusion

  • Python yield keyword creates a generator function.

  • It's useful when the function returns a large amount of data by splitting it into multiple chunks.

  • We can also send values to the generator using its send() function.

  • The yield from statement is used to create a sub-iterator from the generator function.

2.6.10. Use Case - 0x01

>>> def get_files(directories):
...     for directory in directories:
...         yield from Path(directory).rglob('*.py')
>>>
>>>
>>> paths = [
...     'ChapterA/_assignments',
...     'ChapterA/_solutions',
...     'ChapterB/_solutions',
...     'ChapterB/_solutions',
... ]
>>>
>>> for file in get_files(paths):  
...     print(file)

2.6.11. Assignments

Code 2.29. Solution
"""
* Assignment: Generator YieldFrom Path
* Complexity: easy
* Lines of code: 2 lines
* Time: 3 min

English:
    1. Create function `get_files()`
    2. Using `Pathlib.glob()` return files in `path: Path`:
       a. Python files (extension: `*.py`)
       b. ReST files (extension: `*.rst`)
    3. Return result as `Iterator[Path]` using `yield from`

Polish:
    1. Stwórz funkcję `get_files()`
    2. Używając `Pathlib.glob()` zwróć pliki w katalogu `path: Path`:
       a. Pliki Python (rozszerzenie: `*.py`)
       b. Pliki ReST (rozszerzenie: `*.rst`)
    3. Zwróć wyniki jako `Iterator[Path]` używając `yield from`

Tests:
    >>> import sys; sys.tracebacklimit = 0
    >>> from inspect import isgenerator, isgeneratorfunction

    >>> path = Path.cwd()
    >>> file = get_files(path)

    >>> assert isgeneratorfunction(get_files)
    >>> assert isgenerator(file)
    >>> assert isinstance(next(file), Path)
"""

from pathlib import Path
from typing import Iterator


def get_files(path: Path) -> Iterator[Path]:
    ...