Hey everyone!
I'm working on migration of some of data pipelines to prefect.
So i've made few flows, but it doesn't look like the best way to build it.
The flow is deployed in kubernetes (currently as prefect agent).
And one of the steps now is just a docker image that is being executred as a kubernetes job. To run it and transfer data, persistent volume is being created (to map input and output from the main flow).
So here's the way it works now:
3 step flow
1. fetch data (large binaries) and some data preparation
2. trigger k8s job directly using kubernetes api (with created persistent volume to transfer data from step 1)
3. wait for results and generate output.
So I want to understand what is the best practice around building solutions like this and how to build this step 2 better? I know that there's a kubernetes job integration in prefect but there's no clarity on how it could be triggered and then results are fetched.
One small thing to mention: This 2nd step job should still be a different job as it requires GPU and different configuration. Also code-wise it should be separated, so looks like separate image is perfect
@Marvin