https://prefect.io logo
b

Brian Mesick

12/07/2020, 4:27 PM
Hi folks, we’ve got some older flows that I’d like to get current with how Prefect works now, but I’m not entirely sure I understand how to fit our old paradigm into the new world. I’ll thread in because this might be long…
We have ~8 flows all using Docker storage and KubernetesJobEnvironment with yaml files.
We’d like, I think, to use run_configs to get rid of the yaml files, github storage to be able to update our flows without a giant image build step, and LocalDaskExecutor to parallelize our mapped tasks.
run_configs seems like a pretty straight forward change, but I’m not sure I understand the relationship between the containers, executor, and file based storage. I’ve spent a bunch of time in the docs, but can’t seem to put the pieces together. It seems like we would still need to build a container per flow, then tell the container to pull the flow file from GitHub. There also seem to be some changes that would require a rebuild of the image, but it’s not entirely clear what they would be outside of changes in Python requirements. I’m also not sure how a LocalDaskExecutor plays with a K8s deployment. I’m guessing it just runs the local cluster on the container?
k

Kyle Moon-Wright

12/07/2020, 5:02 PM
Hey @Brian Mesick, Regarding your last few questions here - I would recommend rebuilding your images for the new
run_config
pattern as a means to clean up your flow from the KubernetesJobEnvironment. In which case, you’ll likely maintain your Docker storage, utilize a KubernetesRun, and use a LocalDaskExecutor. I think the most significant change you will notice will be a lack of an intermediary Kubernetes Job to submit your DaskCluster onto your deployment. And yes, it just runs the local cluster on the container.
b

Brian Mesick

12/07/2020, 5:56 PM
Cool, thanks. I’m still trying to wrap my head around file based docker storage, but I think I’ll just have to try it and see what happens.
k

Kyle Moon-Wright

12/07/2020, 6:52 PM
No problem. Overall, we hope the
run_configs
pattern is simpler and more semantic for users - but feel free to let us know how your transition from legacy environments goes and we’re happy to help with any snags. 👍
👍 1