Hi All, I have this ELT scenario and appreciate that you could give me suggestions.
1: ET runs as first step that pulls data from a SQL server, saves data into zipped csv files and then uploads them into snowflake by PowerShell scripts.
2: After EL is done, T runs through a dbt project
3: Then we need to process tabular cubes and Power BI datasets through PowerShell scripts and API.
I'm thinking a dedicated image for each step and then use prefect to create containers in a workflow, run each step as a task insider this container one by one. and then shutdown containers in the flow's last step. Is that possible?
Another way is that creating flow for each step and another "parent" flow to mange the workflow.