How to save a node/pipeline output as a parameter?

Questions
#1

Hi! I’m building a training pipeline and I would like to save one of the values from it as a parameter so I can pass it into other nodes in other pipelines. My only idea is to open and write to the parameters.yml file but is there a better way to do this?
Thanks!

#2

I don’t think you need to do anything special, something like this will work fine. Kedro handles passing these “transient” (technical term: MemoryDataSet) through nodes automatically. Like this:

def my_func():
  return 1 # this is the parameter you want to pass from one node to another

def my_other_func(stuff):
  assert stuff == 1

def create_pipeline():
  return Pipeline([
    node(my_func, inputs=None, outputs="my_param"),
    node(my_other_func, inputs="my_param", outputs=None)
  ])
1 Like
#3

Thanks zain! Really appreciate the code chunk, too. What if I also want to save the parameter into my catalog? What file type should a single number be held as?

#4

I believe that isn’t something that is typically supported (a genuine “parameter” in the sense of a Kedro parameter). But you can absolutely save a single number as a TextDataSet or JSONDataSet - though it might require some serialisation.

#5

I would lean on PickleDataSet as it will load right back in as you saved it without needing to fuss with types or lookups.