Pipeline based on command line parameters

Hi everyone! I’m working on a pipeline that produces a monthly report from a bunch of data sources. Some sources contain data for many months, others make a month specific API request so in order to specify which month to get data for I supply a month parameter through command line when running the pipeline. This works reasonably well.

An extra complication comes from the fact that some of the processing that needs to happen changes depending on the year (i.e., reports for any month in 2020 follow one format and for 2021 follow a slightly different format). I’ve addressed this with getting the command line month parameter from session context in register_pipelines.

I’ve got a feeling that this is not really the “kedro” way to go about it. Also, kedro-viz doesn’t work with this approach. Are there better options for constructing a pipeline based on command line parameters? Should I just make separate pipelines for each year and specify that in the kedro run command?

1 Like