In order to get the best help, it is suggested to answer the following questions:
What is the goal you are trying to achieve?
I have a dataset ~30m rows in a MySQL table and I would like to call subsets of the data based on a key then iterate the data through a node and consolidate the results from each subset at the end.
What have you tried, in order to accomplish the goal?
I have made an attempt that seems to work using a generator object for the database results then passing the subsets into the node and appending the results to and existing table.(feels very filthy).
What version of Kedro are you using? (Use kedro -V)
0.16.6
Do you have any custom plugins?
Nope
Perhaps I am overthinking this and there is a simpler way to do this…