set the channel description: Place for contributors to Prefect can hang out and arrange to work together!

Welcome!! 🙂

I’m getting my gcal to honor my desire to have a publically joinable event, but as mentioned on the announcement I’m hosting a hangout this Friday for anyone who wants to synchronously work on Prefect stuff (or just ask questions/hang out). It will be at 2pm EST

Howdy...I have potential side project I may start to work on to enable non-kubernetes based ephemeral dask clusters ala dask-kubernetes....would it be better gauge interest here or in main community channel?

for AWS

Update on tomorrow’s hangout! I will be voice/video chatting out of https://meet.google.com/oyj-phjj-tya starting at 2pm EST; open to questions about current work, future work, or just chatting. Hope to make this a regular thing if it is helpful to people! (slash I just want to hang out with you all 🙂 ). I also made a meetup link for the convenience of letting people outside this Slack world know (https://www.meetup.com/Prefect-Community/events/269535485/) or easily getting it onto your gcals if you want to subscribe there for that reason. Hope to see some of you then!

set the channel topic: Core Contributors Cantina 3/20 at 2 pm EST: https://meet.google.com/oyj-phjj-tya

<!here> myself and Marvin with be in the hangout in 30 mins for anyone who wants to synchronously chat about Prefect stuff this Friday, here’s a reminder link for convenience: https://meet.google.com/oyj-phjj-tya I have queued up the Cantina Band theme from star wars too, in case anyone was worried 🙂

thanks for stopping by everyone! Since we all could use a little more facetime these days, I’m gonna go ahead and put one on the books for next Friday at 4pm EST :marvin: And, as mentioned today, we are going to go digital for the sprints we were going to do at PyCon on April 20th, so clear your schedule 😁

set the channel topic: Core Contributors Cantina 3/27 at 4 pm EST: https://meet.google.com/dok-gvpz-wxn

Hi Team, I'm looking for guidance on the correct approach towards modifying a result_handler. Specifically, the

S3ResultHandler

doesn't expose all of the

boto3.client

arguments, which prevents us from re-using this to connect to non-AWS S3 compliant Object Stores (such as MinIO). I created a wrapper around

S3ResultHandler

which simply allows the user to specify more of the parameters at initialization, and then forwards those parameters off to

boto3.client

when it's initialized. Rather than creating another Result Handler as a separate Python package, would it make more sense to modify the

S3ResultHandler

in Prefect Core? Or does it make more sense to have a dedicated Result Handler specific to MinIO, and use their Python wrapper around boto3?

Just a reminder I’ll be at the Cantina (aka on a Google Hangout) in an hour if you want to come chat about Core roadmap/PRs/issues/etc! The hangout link is: https://meet.google.com/dok-gvpz-wxn

set the channel topic: Project Earth livestream 4/3 at 4 PM EDT on Youtube Live

set the channel topic: Project Earth livestream 4/3 at 4 PM EDT on Youtube Live:

Hey! Are there any docs on how to run tests? I wrote

tasks.docker.container.RemoveContainer(Task)

(based on

StopContainer

) which I thought I’d contribute

Alright, here it is: https://github.com/PrefectHQ/prefect/pull/2263

Awesome!! Thanks for joining the team @Manuel Aristarán

set the channel topic: Join us to chat PRs/issues/roadmap at the Core Contributors’ Cantina 4/10 at 4 PM EDT: meet.google.com/zow-qvhj-ibp

Hey all, wanted to chat about DFE for a hot second to get a better understanding of the Prefect engine and the decisions that went into the current design and things to watch out for in the future design. So in theory, I 100% understand the difference between the DFE execution model and the BFE execution model, but I’m wanting to know if I’m understanding how Prefect is currently handling BFE. I would also like to know, big picture, why the idea I’m having for DFE wouldn’t work. So current understanding of BFE is that when the FlowRunner goes to submit each task, they gets initialized and prepare to submit for running, it does a quick check where it sees if the its a parent of a mapped task (going to submit a bunch of tasks) or the child of that parent (one of the spawned mapped tasks). The FlowRunner looks to wait for the Task (or all the children of that Task) before continuing to walk the DAG. In the context of Prefect, this means that each Task (or parent of a group of Mapped Tasks) represents the computation, and the reason that the BFE works so “easily” is that you’re only ever going down 1 level of execution. In addition to this, the Task graph assumes that a task can have either the Task or the group of Mapped Tasks as a parent. So my understanding of how DFE would be executed would be to add a recursion option onto both the 1 level of execution (now can be 1 to N levels), and the parent where can it expect either a Task or 1 to N levels of mapped parents. But in actuality (I think) all that would matter is that it sees is the upstream dependency on M tasks, and would just need to wait until those are done. I’m trying to wrap my head around the change of semantics (with specific regards to what would need to change for Prefect). I think Chris mentioned on the github issue that task running logic would need to move to the FlowRunner. I would think that the only portion that needs to change on that would be how gathering futures works, and whether we needed to wait on the completion of the futures or just passing around the futures themselves to any potential downstream tasks. Things I’m not sure about: • When tasks are getting prepared to run, is the assumption that since the Task was told to run, it has the go ahead (outside of the light validation that gets done) • Would DFE execution require some form of weighted futures to stay performant (which sub-trees to prioritize?) • Should the determination of which upstream tasks are ready fall onto the TaskRunner of the downstream task? Or is that in the FlowRunner by a design choice?

https://aws.amazon.com/about-aws/whats-new/2020/04/aws-fargate-launches-platform-version-14/ https://aws.amazon.com/about-aws/whats-new/2020/04/amazon-ecs-aws-fargate-support-amazon-efs-filesystems-generally-available/

big deal for fargate based flows that need more storage space...you now get 20 GB per task and can leverage efs mount

Reminder to drop by the Cantina today if you want! Besides normal chatting and syncing up on open PRs/issues, hoping to get a little conversation going on subflows and streaming/event driven flows, and in particular see if anyone wants to work on the latter with me. 4pm EDT at meet.google.com/zow-qvhj-ibp

grid view, i like it a lot: https://chrome.google.com/webstore/detail/google-meet-grid-view/bjkegbgpfgpikgkfidhcihhiflbjgfic?hl=en-US haven’t used yet but has a couple of cool features: https://chrome.google.com/webstore/detail/google-meet-enhancement-s/ljojmlmdapmnibgflmmminacbjebjpno

<!here> based on the convo from today’s Cantina I’m going to go ahead and set something up, targeting next week, on streaming/event driven flows for any contributors interested in a kick off/sync up. We will review some base assumptions/knowledge transfer, debate PIN 14, and figure out how to work together going forward. @Alex Cano and @Alexander Hirner in particular I’d love to make sure you both can attend since you’ve expressed interested before, and of course anyone else who wants to join. Hit me with 🔴 for next Tuesday at 4 EDT 🔵 for next Thursday at 4 EDT 🕶️ if you hate your choices

set the channel topic: PyCon 2020 was cancelled, but we’re hosting digital versions of our events next week. Join me Friday 4/17 at 4 EDT for a users’ Open Space and Monday 4/20 for a full day of sprints! More info in #pycon2020

Hey all. Here’s a snippet which I thought might be useful. It’s a version of

ShellTask

with support for sending a string to the child process’

STDIN

https://gist.github.com/jazzido/4c34fa2e56c3b403a6851791b8c752dc

Unfortunately I don’t have time this week to submit a proper PR :/

I think the proper way to do this would have been to extend

ShellTask

with a

stdin

kwarg, but I lost the battle against

Popen

. My version uses the (IMHO) much nicer

subprocess.run()