Thanks @Alex Cano - This still means that the unit of work are still docker scripts, but I was wondering if it could still be more fine grained? I'd like to explain with a real example. We have a script which
1. Creates a config file
2. hits two different of data bases
3. runs three levels of transformations
4. one validation check
5. Pushes the output to a database
6. Pushes the state to another database
We currently have the entire thing dockerized in a single container. We could potentally dockerize each step independently, that would mean 9 different docker containainers, but each time the state of the world has to be recreated and I think the overhead would make it defeat the purpose of modularity.
I'm really fascinated by the Prefect approach of having each function as a separate task which can be monitored independently, and I was wondering if there was a solution for that. I'm looking for something like an R Task, similar to other tasks,
Maybe I'm trying to have the cake and eat it too, but I was wondering if there were bettter solutions