Flow - Flow-based framework

ctapipe.flow is a Python implementation of the flow-based programming paradigm for ctapipe framework. In flow-based programming, applications are defined as networks of black-box components that exchange data across predefined connections. These components can be reconnected to form different applications. ctapipe-flow executes ctapipe processing modules in a sequantial or multiprocess environment. User implements steps in Python class.

The multiprocess mode is based on ZeroMQ library (http:queue//zeromq.org) for messages passing between process. ZMQ library allows to stay away from class concurrency mechanisms like mutexes, critical sections semaphores, while being thread safe. Passing data between steps is managed by the router thanks to Pickle serialization. A Step can be executed by several process, in this case the router uses LRU pattern (least recently used ) to choose the step that will receive next data.

The router also manage Queue for each step.