Classes to run producers (iterators) and consumers (coroutines) in a background thread/process.
There are many libraries to create pipelines with stages running in separate processes, a nice one is parallelpipe, but this library does something different. It will lift the entire pipeline up to the point of the Producer into a separate process or thread. It's a more coarse library but easier to integrate since things keep looking as normal generators.
There are currently 3 implementations:
dummy.Producer: non-concurrent implementationthread.Producer: uses a background thread to run the generatorprocess.Producer: uses a background process to run the generator
dummy.Producer is useless in practice.
thread.Producer is useful for IO bound generators.
process.Producer is useful for CPU or IO bound generators.
It has the complications of dealing with processes (different memory spaces,
logging, etc).
For logging, module multiprocessing-logging can be used.
Basic example:
from concurrent_iterator.thread import Producer
...
items = Producer(slow_generator, maxsize=5)
for item in items:
[Do some time consuming task]
In the previous example, while doing some time consuming task, the
slow_generator will continue running in a background thread and will
pre-calculate up to 5 values.