How to retry pipeline runs

Hello,

Is there a way in kedro to apply a rerun policy? To run the whole pipeline or just the failed node after failing waiting 30 seconds for example?

1 Like

I don’t think kedro has this built-in in any way. What is a use case example? What runner are you using?

You could do this in a python or shell script that runs your pipeline, then decides to rerun or not.

It might also be possible to do with a shell script.

Hello Waylon,

We are running the pipeline with “kedro run --pipeline <name_of_pipeline>”. the functionality that I was looking for it was to add the tenacity library when declaring the pipeline. Something like adding the wrapper to the pipepline function could it be implemented?

1 Like

I am not familiar with the tenacity, but it looks like you can decorate a node with retry. Does that work? Otherwise a on_node_error hook should be able to do this as well.

Hello,

The kedro doc actually indicates how to do it: Hooks — Kedro 0.17.3 documentation

This works:

from tenacity import retry

from kedro.framework.hooks import hook_impl
from kedro.pipeline.node import Node


class ProjectHooks:
    @hook_impl
    def before_node_run(self, node: Node):
        # adding retrying behaviour to nodes tagged as flaky
        if "flaky" in node.tags:
            node.func = retry(node.func)

Something similar could be done at pipeline level

1 Like

That is quite an interesting use of tags, I haven’t even notice that in the docs before. Glad you were able to work.

I have never looks into tenacity before. I just took a look at their docs and the composability of various retry methods looks fantastic.