Execution of pipeline based on condition

I wanted to know if it’s possible to create conditional pipelines in kedro where the condition is evaluated on the basis of outputs of one of the pipelines.
As it can be seen in the above diagram, ‘pipeline 2’ is run for a given page only if desired output is not obtained using ‘pipeline 1’

So this kind of conditional pipeline needs to be executed for each page of the document and then outputs for individual pages should be merged together to get document level info.

Any thoughts on how this can be implemented using kedro would be really helpful!

2 Likes

I think the easiest way is to include an extra “should run” value that can be passed along with the other data, in parallel.

The downstream nodes will take the should_run value and emit a new result or emit the previous result, based on the should_run.

2 Likes

I think you can do this with a python script. I think you can run one pipeline, then decide what to do. If you can do it in a python script it can also be added to your projects cli. What version of kedro are you on? I think I would put it in run.py in 0.16.x versions, but I am not familar with the session object of 0.17.x versions.

1 Like