Looking for any resources regarding development lifecycles between local, dev, and prod environments for data engineering projects - specifically using Dask/Prefect, but that’s probably too niche so other resources are welcome!

I would love to see more, but the only thing in this topic I’ve seen so far is this

Awesome - thank you! Yes - we are planning on Kubernetes deployments of Prefect Server, and Dask Gateway, with Gitlab storage. Initially thought we would have multiple deployments of Prefect Server rather than projects - one for development environments and one for production, and using different IAM roles for each to control what is accessed and where things are run. Something like running prefect flows locally via python (

python myflow.py

) with a dedicated dask cluster for this purpose, then running in development, then finally pushing to prod. Things are a little tricky with the local step, because we may have a use case for users playing with sensitive data at that stage, which is a whole other animal.

Ah i see, but even if you had multiple servers, it still seems the sensitive data being exposed would still be an issue right? The setup to push to prod looks good though

See https://github.com/PrefectHQ/prefect/discussions/4042 as well