The problem is that we have a circular dependency:
mypackage.cli.run calls catalog.py on the
with KedroSession.create(...) as session: line, but catalog.py requires an initalized KedroSession to read parameters & modify dataset paths/parameters.
I tried to break this circular dependency by directly reading the config yamls, but then if I override variables from the CLI (e.g.
--params "env:prod") the catalog will only ever read the default
env: dev setting from the yaml.
As a terrible hack, I store some variables in my OS environement variables inside
mypackage.cli.run, but this is ugly and only supports saving strings. Is there a better way?
# inside cli.py run() # TODO: make atexit register to clean these env vars up after finished running global_params = ["zoom", "env", "export"] for param in global_params: if param in params: param_val = params[param] else: param_val = get_kedro_param(param) # manually creates ConfigLoader(conf_paths) and fetches param os.environ[param] = param_val