https://prefect.io logo
Title
t

Thomas La Piana

05/04/2020, 6:41 AM
Has anyone gotten flows executing in a Dask Gateway remote environment? based on the docs it looks like i'll be writing my own, but wanted to double check there isn't a way to do it currently without writing a new environment
j

Joe Schmid

05/04/2020, 1:16 PM
@Thomas La Piana I haven't done it yet, but (1) I don't think it would be difficult and (2) you should be able to follow the pattern I used for
DaskCloudProviderEnvironment
in this PR #2360
(FWIW, we've been testing
DaskCloudProviderEnvironment
in a Flow scheduled to run every 15 minutes for the last week and it's been very reliable.)
t

Thomas La Piana

05/04/2020, 2:40 PM
thanks!
exactly what i was looking for 😄
👍 1
@Joe Schmid thanks a ton! rewriting that to work for DaskGateway was a piece of cake thanks to all of the heavy lifting you'd done already. got it up and running
j

Joe Schmid

05/11/2020, 1:35 PM
@Thomas La Piana, glad to hear it. Also keep an eye on this issue: https://github.com/PrefectHQ/prefect/issues/2508 @Jim Crist-Harif had a great idea around decoupling Environments from Dask which I think makes a ton of sense. It's probably a bit longer-term, i.e. no need to stop your work on an Environment that uses Dask Gateway, but I suspect it'll be a better approach in the long run.
t

Thomas La Piana

05/11/2020, 3:02 PM
huge thanks for that link 🙂