Spotify has just open sourced Klio, an ecosystem for processing audio files – or any binary files – easily and at scale. It wraps around Beam. Might be interesting to take a look and see if we can leverage the same approach to enable real-time processing for Kedro.
I like the split up of functionality. I noticed that the klio cli has its own library. I like that I can extend kedros so easily, but I don’t like updating the base cli module in every project or suffer from inconsistent behavior.
Whoooaa, that’s cool.