How to deploy Kedro pipeline with other python applications or How can we pass python object instead of string into Kedro pipeline?

I have a use case that my pipeline did a few steps.

  1. Get data (For development use)
  2. Train a Model
  3. Save the output

Let say if I package up this pipeline and want to pass it to another team. I would like to pass in an actual python object to the pipeline instead of relying on catalog only (Since it would mean they must first write the data to some kind of file)


data = release_team_get_data()
context.run(pipeline="my_publish_pipeline", input=data)
# I want to pass in data that is already loaded in memory

AFIK, kedro inputs and outputs can only be catalog entries. You might be able to pass an object in by storing creating a dataset for it, then running save on that dataset. Another less appealing option might be storing data in globals and pulling from globals in your function, this would be a last ditch effort.

data = release_team_get_data()
context.catalog.datasets.your_pickle_data_set.save(data)
context.run(pipeline="my_publish_pipeline", input=data)
# Now your pipeline can load 'your_pickle_data_set'