https://prefect.io logo
Title
r

Richard Pelgrim

09/24/2021, 2:14 PM
Just published a blog about connecting Prefect to a Dask cluster using Coiled for processing larger-than-memory Tasks and/or Flows. https://towardsdatascience.com/scaling-your-prefect-workflow-to-the-cloud-2dec4e0b213b
๐Ÿ‘ 7
๐Ÿš€ 15
:marvin: 8
i

itay livni

09/27/2021, 2:39 PM
@Richard Pelgrim Nice. I like the resource manager example...
a

Andrew Black

09/27/2021, 4:49 PM
Excellent, Richard! I really like how you describe what Prefect does, how our technologies work together, and the TLDR at the top.
r

Richard Pelgrim

09/28/2021, 6:47 AM
Thanks @itay livni - lmk if you end up implementing it with Coiled anywhere. Would love to have a look!
Thanks @Andrew Black, happy to read you enjoyed the post!
t

Todd de Quincey

11/06/2021, 7:53 AM
Great example gist. Exceptionally clear, concise and covers all the bases. Awesome work @Richard Pelgrim
r

Richard Pelgrim

11/06/2021, 1:04 PM
Thanks @Todd de Quincey! ๐Ÿ™ would love to hear about anything (good or bad) you're running into with Dask clusters!
t

Todd de Quincey

11/06/2021, 2:48 PM
Will do. Just looking at coiled. Whatโ€™s included in the free tier? 1000 CPU hours per month if I am reading it correctly
r

Richard Pelgrim

12/17/2021, 2:52 PM
Slack just sent me a notification for this message... sorry to have missed this earlier! How was your Coiled onboarding?
t

Todd de Quincey

12/17/2021, 2:54 PM
Didn't use it lol
r

Richard Pelgrim

12/17/2021, 3:29 PM
haha, ok, fair enough ๐Ÿ™‚
because it was tricky onboarding or because you didn't end up needing it?