• Scott Zelenka

    Scott Zelenka

    2 years ago
    Hi Team, I'm looking for guidance on the correct approach towards modifying a result_handler. Specifically, the
    S3ResultHandler
    doesn't expose all of the
    boto3.client
    arguments, which prevents us from re-using this to connect to non-AWS S3 compliant Object Stores (such as MinIO). I created a wrapper around
    S3ResultHandler
    which simply allows the user to specify more of the parameters at initialization, and then forwards those parameters off to
    boto3.client
    when it's initialized. Rather than creating another Result Handler as a separate Python package, would it make more sense to modify the
    S3ResultHandler
    in Prefect Core? Or does it make more sense to have a dedicated Result Handler specific to MinIO, and use their Python wrapper around boto3?
    Scott Zelenka
    j
    +1
    8 replies
    Copy to Clipboard
  • Laura Lorenz (she/her)

    Laura Lorenz (she/her)

    2 years ago
    Just a reminder I’ll be at the Cantina (aka on a Google Hangout) in an hour if you want to come chat about Core roadmap/PRs/issues/etc! The hangout link is: https://meet.google.com/dok-gvpz-wxn
    Laura Lorenz (she/her)
    manesioz
    +2
    11 replies
    Copy to Clipboard
  • Laura Lorenz (she/her)

    Laura Lorenz (she/her)

    2 years ago
    set the channel topic: Project Earth livestream 4/3 at 4 PM EDT on Youtube Live
  • Laura Lorenz (she/her)

    Laura Lorenz (she/her)

    2 years ago
    set the channel topic: Project Earth livestream 4/3 at 4 PM EDT on Youtube Live:

    https://youtu.be/k9arDAr1GeA

  • m

    Manuel Aristarán

    2 years ago
    Hey! Are there any docs on how to run tests? I wrote
    tasks.docker.container.RemoveContainer(Task)
    (based on
    StopContainer
    ) which I thought I’d contribute
    m
    1 replies
    Copy to Clipboard
  • m

    Manuel Aristarán

    2 years ago
  • Jeremiah

    Jeremiah

    2 years ago
    Awesome!! Thanks for joining the team @Manuel Aristarán
  • Laura Lorenz (she/her)

    Laura Lorenz (she/her)

    2 years ago
    set the channel topic: Join us to chat PRs/issues/roadmap at the Core Contributors’ Cantina 4/10 at 4 PM EDT: meet.google.com/zow-qvhj-ibp
  • a

    Alex Cano

    2 years ago
    Hey all, wanted to chat about DFE for a hot second to get a better understanding of the Prefect engine and the decisions that went into the current design and things to watch out for in the future design. So in theory, I 100% understand the difference between the DFE execution model and the BFE execution model, but I’m wanting to know if I’m understanding how Prefect is currently handling BFE. I would also like to know, big picture, why the idea I’m having for DFE wouldn’t work. So current understanding of BFE is that when the FlowRunner goes to submit each task, they gets initialized and prepare to submit for running, it does a quick check where it sees if the its a parent of a mapped task (going to submit a bunch of tasks) or the child of that parent (one of the spawned mapped tasks). The FlowRunner looks to wait for the Task (or all the children of that Task) before continuing to walk the DAG. In the context of Prefect, this means that each Task (or parent of a group of Mapped Tasks) represents the computation, and the reason that the BFE works so “easily” is that you’re only ever going down 1 level of execution. In addition to this, the Task graph assumes that a task can have either the Task or the group of Mapped Tasks as a parent. So my understanding of how DFE would be executed would be to add a recursion option onto both the 1 level of execution (now can be 1 to N levels), and the parent where can it expect either a Task or 1 to N levels of mapped parents. But in actuality (I think) all that would matter is that it sees is the upstream dependency on M tasks, and would just need to wait until those are done. I’m trying to wrap my head around the change of semantics (with specific regards to what would need to change for Prefect). I think Chris mentioned on the github issue that task running logic would need to move to the FlowRunner. I would think that the only portion that needs to change on that would be how gathering futures works, and whether we needed to wait on the completion of the futures or just passing around the futures themselves to any potential downstream tasks. Things I’m not sure about: • When tasks are getting prepared to run, is the assumption that since the Task was told to run, it has the go ahead (outside of the light validation that gets done) • Would DFE execution require some form of weighted futures to stay performant (which sub-trees to prioritize?) • Should the determination of which upstream tasks are ready fall onto the TaskRunner of the downstream task? Or is that in the FlowRunner by a design choice?
    a
    Chris White
    16 replies
    Copy to Clipboard