Kedro's new deployment documentation

Kedro docs just released a new stack of docs to help deployment of Kedro projects using different platforms. In addition to the single machine deployment, now there are sections to help if you are using:

We hope this helps our users to find out the best option to deploy their Kedro pipelines and the most suitable way to do it.

We hope you enjoy (and in case you don’t, let us know why).

Happy pipelining, folks!

4 Likes

Very interested in diving into this new BatchRunner. We have been using batch with Docker!

3 Likes

Oh what, tell me more about your setup

1 Like

Azure Pipelines ci/cd -> push images to ecr -> cloudwatch (schedule/trigger) -> batch (run containers)

batch can run kedro pipelines all the various ways right out of the box, from nodes, to nodes, certain tags, certain pipelines. Currently the whole pipeline gets ran on one instance.

If I understand correctly the BatchRunner will spin up a series of individual batch jobs that better leverage the infrastructure and parallelization.

2 Likes