Configs load in Kedro, but break in jupyter notebook

Hey there, we have a set of experiments where each experiment will depend on a different set of datasets. We want to be able to configure the experiment we run via config, so we’ve set-up our nodes to load their input datasets from config.

Our config.yml is structured as follows:

experiment_to_run: v0

experiments:
  demand:
    v0:
      - 
        identifier: "XXX_ZZZ"
        class: "XXX_AAA"
        data: data_set_in_catalog_one
        zones: 
          - "A"
      -
        identifier: "XXX_YYY"
        class: "XXX_BBB"
        data: data_set_in_catalog_two
        zones: null

In pipeline.py we load the config as follows:

conf_paths = ["conf/base", "conf/local"]
conf_loader = ConfigLoader(conf_paths)
try:
    conf_parameters = conf_loader.get("parameters*", "parameters*/**")
except MissingConfigException:
    conf_parameters = {}

experiment_to_run = conf_parameters["experiment_to_run"]

def create_pipeline(**kwargs):
    datasets_to_ingest = {
        i["identifier"]: i["data"]
        for i in conf_parameters["experiments"]["demand"][experiment_to_run]
    }
    return Pipeline(
        [
            node(
                func=train_demand_functions,
                inputs={
                    **{
                        "model_params": "params:experiments.demand.{}".format(
                            experiment_to_run
                        )
                    },
                    **datasets_to_ingest,
                },
                outputs="func_output",
                name="process_input",
            )
        ]

This works fine when we run the pipeline via kedro run --pipeline model_train. However, when we try to initialise kedro juptyer notebook we get the following error:

ValueError: Given configuration path either does not exist or is not a valid directory: conf/base

After some digging, it looks like the notebook is trying to load root_project_dir/notebooks/conf/base as opposed to root_project_dir/conf/base. How do I go about ensuring that I can use kedro jupyter notebook while also being able to run my pipeline?

Would appreciate any support here, if possible :slight_smile:

FYI, I wasn’t able to resolve the file stub issue. However, I managed to solve the problem by assigning a params property to ProjectHooks as follows:

@hook_impl
def register_config_loader(self, conf_paths: Iterable[str]) -> ConfigLoader:
    c_config = ConfigLoader(conf_paths)
    self.config_params = c_config.get("parameters*", "parameters*/**")
    return c_config

With this, I could call create_pipeline and pass-in the parameters directly as follows:

def register_pipelines(self) -> Dict[str, Pipeline]:
    """Register the project's pipeline.

    Returns:
        A mapping from a pipeline name to a ``Pipeline`` object.

    """
    model_training_pipeline = mt.create_pipeline(params=self.config_params)
    model_testing_pipeline = sco.create_pipeline(params=self.config_params)
    return {
        "__default__": model_training_pipeline,
        "model_train": model_training_pipeline,
        "model_run_offline": model_testing_pipeline,
    }

This means I could access params from create_pipeline() without manually creating conf_loader in my pipeline.py script. This avoids any file path issues, as these are all handled implicitly through the Kedro package :slight_smile:

Would be great to document this as a way of making parameters accessible from create_pipeline