Since Kedro comes with a local/ folder to keep credentials.yml, how does one test the pipeline as we will not include this file in the repository? For example, if I need to run this pipeline in CI, what’s the preferred way to achieve this?
Does this help?
You can then add your credentials to your CI/CD env variables.
I think this could work.
However, I found that increasingly I have to make changes and the whole experience with Kedro is inconsistent.
By doing environment variable injection, all of a sudden I have to put that local config into version control in order to get it running in CI or I have to move that out from the local folder.
Why not doing this in the first place if Kedro aim to support production?
CI/CD services come with tools to store secrets such that they don’t end up in your git history. This allows you to have people work on the project without needing to give them credentials to things they aren’t on a need to know basis, this might be the ability to read directly out of a sensitive database that other sections of the pipeline run and sanitize sensitive information from, or the keys to editing production tables. Some CI services only let you have secret variables while others let you have entire secret files.
Overall its just bad practice to keep secrets in your git history as it is almost impossible to ensure that you have completely removed them without blowing away your git history. Even worse on public repos bad actors are likely to sniff them out from the github apis almost instantly.