Custom dataset reading parameters

Hi all,

Is it possible for custom dataset classes to be able to read the parameters.yml or used when kedro run --params is used? The goal for the custom dataset is to set a date string corresponding to date set as a parameter or in CLI. I could read the yml file directly but I won’t be able to read those specified with the CLI. Any ideas?

Thanks all

Hi @munchmuch !

I think you will find your answer in this thread:

Let me know if this is not the answer you are after :slight_smile:

Hi @ljam thanks for your reply.

We have written our custom dataset class inheriting from the SparkHiveDataSet class. Say I have a parameter in parameters.yml or given by kedro run --params, called global_date: “2021-01-01”. How can I access global_date from the custom dataset class?

Thanks again

Hi @ljam,

Finally understood your reply and got it to work!

The templated loader reads in the globals.yml and fills in ${…} variables in catalog.yml which also takes in CLI parameters. I trying so hard to think of the name of global variable which I can access the global parameter from the dataset class file.

Thanks again!

Glad to hear everything worked out!

Fyi, it doesnt necessary have to be in a file named globals.yml. It can be in any file in your conf folder. For instance here I see the globals to be in parameters.yml.

    def register_config_loader(
        self, conf_paths: Iterable[str], env: str, extra_params: Dict[str, Any],
    ) -> ConfigLoader:
        loader = TemplatedConfigLoader(
            conf_paths,
            globals_pattern="*parameters.yml",
            globals_dict=extra_params,
        )
        return loader