https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-contributors
  • l

    Laura Lorenz (she/her)

    03/17/2020, 8:50 PM
    set the channel description: Place for contributors to Prefect can hang out and arrange to work together!
  • l

    Laura Lorenz (she/her)

    03/17/2020, 10:24 PM
    Welcome!! šŸ™‚
    šŸ‘‹ 3
  • l

    Laura Lorenz (she/her)

    03/17/2020, 10:28 PM
    I’m getting my gcal to honor my desire to have a publically joinable event, but as mentioned on the announcement I’m hosting a hangout this Friday for anyone who wants to synchronously work on Prefect stuff (or just ask questions/hang out). It will be at 2pm EST
  • b

    Braun Reyes

    03/18/2020, 8:33 PM
    Howdy...I have potential side project I may start to work on to enable non-kubernetes based ephemeral dask clusters ala dask-kubernetes....would it be better gauge interest here or in main community channel?
    a
    l
    l
    • 4
    • 4
  • b

    Braun Reyes

    03/18/2020, 8:37 PM
    for AWS
  • l

    Laura Lorenz (she/her)

    03/19/2020, 11:44 PM
    Update on tomorrow’s hangout! I will be voice/video chatting out of https://meet.google.com/oyj-phjj-tya starting at 2pm EST; open to questions about current work, future work, or just chatting. Hope to make this a regular thing if it is helpful to people! (slash I just want to hang out with you all šŸ™‚ ). I also made a meetup link for the convenience of letting people outside this Slack world know (https://www.meetup.com/Prefect-Community/events/269535485/) or easily getting it onto your gcals if you want to subscribe there for that reason. Hope to see some of you then!
    :marvin: 2
    šŸŽ‰ 1
  • l

    Laura Lorenz (she/her)

    03/19/2020, 11:44 PM
    set the channel topic: Core Contributors Cantina 3/20 at 2 pm EST: https://meet.google.com/oyj-phjj-tya
  • l

    Laura Lorenz (she/her)

    03/20/2020, 5:32 PM
    <!here> myself and Marvin with be in the hangout in 30 mins for anyone who wants to synchronously chat about Prefect stuff this Friday, here’s a reminder link for convenience: https://meet.google.com/oyj-phjj-tya I have queued up the Cantina Band theme from star wars too, in case anyone was worried šŸ™‚
    :marvin: 2
  • l

    Laura Lorenz (she/her)

    03/20/2020, 7:36 PM
    thanks for stopping by everyone! Since we all could use a little more facetime these days, I’m gonna go ahead and put one on the books for next Friday at 4pm EST :marvin: And, as mentioned today, we are going to go digital for the sprints we were going to do at PyCon on April 20th, so clear your schedule 😁
    :upvote: 5
    šŸ’Æ 4
    :prefect: 4
  • l

    Laura Lorenz (she/her)

    03/20/2020, 7:36 PM
    set the channel topic: Core Contributors Cantina 3/27 at 4 pm EST: https://meet.google.com/dok-gvpz-wxn
  • s

    Scott Zelenka

    03/27/2020, 2:05 PM
    Hi Team, I'm looking for guidance on the correct approach towards modifying a result_handler. Specifically, the
    S3ResultHandler
    doesn't expose all of the
    boto3.client
    arguments, which prevents us from re-using this to connect to non-AWS S3 compliant Object Stores (such as MinIO). I created a wrapper around
    S3ResultHandler
    which simply allows the user to specify more of the parameters at initialization, and then forwards those parameters off to
    boto3.client
    when it's initialized. Rather than creating another Result Handler as a separate Python package, would it make more sense to modify the
    S3ResultHandler
    in Prefect Core? Or does it make more sense to have a dedicated Result Handler specific to MinIO, and use their Python wrapper around boto3?
    j
    a
    • 3
    • 8
  • l

    Laura Lorenz (she/her)

    03/27/2020, 7:02 PM
    Just a reminder I’ll be at the Cantina (aka on a Google Hangout) in an hour if you want to come chat about Core roadmap/PRs/issues/etc! The hangout link is: https://meet.google.com/dok-gvpz-wxn
    m
    a
    a
    • 4
    • 11
  • l

    Laura Lorenz (she/her)

    03/27/2020, 9:14 PM
    set the channel topic: Project Earth livestream 4/3 at 4 PM EDT on Youtube Live
  • l

    Laura Lorenz (she/her)

    04/03/2020, 2:43 PM
    set the channel topic: Project Earth livestream 4/3 at 4 PM EDT on Youtube Live:

    https://youtu.be/k9arDAr1GeAā–¾

  • m

    Manuel AristarƔn

    04/03/2020, 3:20 PM
    Hey! Are there any docs on how to run tests? I wrote
    tasks.docker.container.RemoveContainer(Task)
    (based on
    StopContainer
    ) which I thought I’d contribute
    l
    • 2
    • 1
  • m

    Manuel AristarƔn

    04/03/2020, 3:59 PM
    Alright, here it is: https://github.com/PrefectHQ/prefect/pull/2263
    šŸš€ 5
    :marvin: 3
  • j

    Jeremiah

    04/03/2020, 4:00 PM
    Awesome!! Thanks for joining the team @Manuel AristarƔn
    šŸ™‚ 1
  • l

    Laura Lorenz (she/her)

    04/03/2020, 9:02 PM
    set the channel topic: Join us to chat PRs/issues/roadmap at the Core Contributors’ Cantina 4/10 at 4 PM EDT: meet.google.com/zow-qvhj-ibp
  • a

    Alex Cano

    04/07/2020, 1:37 AM
    Hey all, wanted to chat about DFE for a hot second to get a better understanding of the Prefect engine and the decisions that went into the current design and things to watch out for in the future design. So in theory, I 100% understand the difference between the DFE execution model and the BFE execution model, but I’m wanting to know if I’m understanding how Prefect is currently handling BFE. I would also like to know, big picture, why the idea I’m having for DFE wouldn’t work. So current understanding of BFE is that when the FlowRunner goes to submit each task, they gets initialized and prepare to submit for running, it does a quick check where it sees if the its a parent of a mapped task (going to submit a bunch of tasks) or the child of that parent (one of the spawned mapped tasks). The FlowRunner looks to wait for the Task (or all the children of that Task) before continuing to walk the DAG. In the context of Prefect, this means that each Task (or parent of a group of Mapped Tasks) represents the computation, and the reason that the BFE works so ā€œeasilyā€ is that you’re only ever going down 1 level of execution. In addition to this, the Task graph assumes that a task can have either the Task or the group of Mapped Tasks as a parent. So my understanding of how DFE would be executed would be to add a recursion option onto both the 1 level of execution (now can be 1 to N levels), and the parent where can it expect either a Task or 1 to N levels of mapped parents. But in actuality (I think) all that would matter is that it sees is the upstream dependency on M tasks, and would just need to wait until those are done. I’m trying to wrap my head around the change of semantics (with specific regards to what would need to change for Prefect). I think Chris mentioned on the github issue that task running logic would need to move to the FlowRunner. I would think that the only portion that needs to change on that would be how gathering futures works, and whether we needed to wait on the completion of the futures or just passing around the futures themselves to any potential downstream tasks. Things I’m not sure about: • When tasks are getting prepared to run, is the assumption that since the Task was told to run, it has the go ahead (outside of the light validation that gets done) • Would DFE execution require some form of weighted futures to stay performant (which sub-trees to prioritize?) • Should the determination of which upstream tasks are ready fall onto the TaskRunner of the downstream task? Or is that in the FlowRunner by a design choice?
    :upvote: 1
    :marvin: 1
    šŸ‘€ 3
    c
    • 2
    • 16
  • b

    Braun Reyes

    04/08/2020, 9:01 PM
    https://aws.amazon.com/about-aws/whats-new/2020/04/aws-fargate-launches-platform-version-14/ https://aws.amazon.com/about-aws/whats-new/2020/04/amazon-ecs-aws-fargate-support-amazon-efs-filesystems-generally-available/
  • b

    Braun Reyes

    04/08/2020, 9:03 PM
    big deal for fargate based flows that need more storage space...you now get 20 GB per task and can leverage efs mount
    šŸŽ‰ 1
    šŸ’Æ 1
  • l

    Laura Lorenz (she/her)

    04/10/2020, 6:20 PM
    Reminder to drop by the Cantina today if you want! Besides normal chatting and syncing up on open PRs/issues, hoping to get a little conversation going on subflows and streaming/event driven flows, and in particular see if anyone wants to work on the latter with me. 4pm EDT at meet.google.com/zow-qvhj-ibp
    šŸ’Æ 2
  • l

    Laura Lorenz (she/her)

    04/10/2020, 8:05 PM
    grid view, i like it a lot: https://chrome.google.com/webstore/detail/google-meet-grid-view/bjkegbgpfgpikgkfidhcihhiflbjgfic?hl=en-US haven’t used yet but has a couple of cool features: https://chrome.google.com/webstore/detail/google-meet-enhancement-s/ljojmlmdapmnibgflmmminacbjebjpno
  • j

    Jim Crist-Harif

    04/10/2020, 8:06 PM
    https://addons.mozilla.org/en-US/firefox/addon/google-meet-grid-view/
  • l

    Laura Lorenz (she/her)

    04/10/2020, 9:44 PM
    <!here> based on the convo from today’s Cantina I’m going to go ahead and set something up, targeting next week, on streaming/event driven flows for any contributors interested in a kick off/sync up. We will review some base assumptions/knowledge transfer, debate PIN 14, and figure out how to work together going forward. @Alex Cano and @Alexander Hirner in particular I’d love to make sure you both can attend since you’ve expressed interested before, and of course anyone else who wants to join. Hit me with šŸ”“ for next Tuesday at 4 EDT šŸ”µ for next Thursday at 4 EDT šŸ•¶ļø if you hate your choices
    šŸ”“ 3
    šŸ•¶ļø 1
    šŸ”µ 4
    :marvin: 1
    a
    • 2
    • 3
  • l

    Laura Lorenz (she/her)

    04/10/2020, 9:49 PM
    set the channel topic: PyCon 2020 was cancelled, but we’re hosting digital versions of our events next week. Join me Friday 4/17 at 4 EDT for a users’ Open Space and Monday 4/20 for a full day of sprints! More info in #pycon2020
  • m

    Manuel AristarƔn

    04/13/2020, 5:15 PM
    Hey all. Here’s a snippet which I thought might be useful. It’s a version of
    ShellTask
    with support for sending a string to the child process’
    STDIN
    https://gist.github.com/jazzido/4c34fa2e56c3b403a6851791b8c752dc
    šŸ˜ 2
  • m

    Manuel AristarƔn

    04/13/2020, 5:16 PM
    Unfortunately I don’t have time this week to submit a proper PR :/
  • m

    Manuel AristarƔn

    04/13/2020, 5:24 PM
    I think the proper way to do this would have been to extend
    ShellTask
    with a
    stdin
    kwarg, but I lost the battle against
    Popen
    . My version uses the (IMHO) much nicer
    subprocess.run()
  • l

    Laura Lorenz (she/her)

    04/17/2020, 9:08 PM
    @Edidiong Etuk hi! welcome to the contributors channel šŸ™‚
    šŸ‘‹ 5
    e
    • 2
    • 1
Powered by Linen
Title
l

Laura Lorenz (she/her)

04/17/2020, 9:08 PM
@Edidiong Etuk hi! welcome to the contributors channel šŸ™‚
šŸ‘‹ 5
e

Edidiong Etuk

04/17/2020, 9:15 PM
Thank youšŸ™‚
View count: 4