https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-contributors
  • l

    Laura Lorenz (she/her)

    04/20/2020, 1:42 PM
    Cross posting on ALL the channels haha but as mentioned in #pycon2020 I’m hosting an all-day Zoom call for Prefect contributors, feel free to stop by before 7 pm EDT! 🙂 https://zoom.us/j/92984979179
    🚀 1
    :marvin: 1
    💯 1
  • s

    Scott Zelenka

    04/20/2020, 6:07 PM
    how would I go about searching for active work on a particular topic? Specifically, I'd like to add a basic async wrapper around some of the GraphQL queries from the client. The use case I'm working on is wrapping the Prefect queries behind a web interface for users in our company which don't have (and don't need) credentials to Prefect Cloud, but we want to expose the ability for them to trigger a Flow Run through a web UI. The web UI is written in an ASGI framework.
    l
    • 2
    • 1
  • l

    Laura Lorenz (she/her)

    04/20/2020, 11:32 PM
    set the channel topic: Core Contributors Cantina Fri 4/24 at 4pm EDT meet.google.com/quz-zuea-kix
  • l

    Laura Lorenz (she/her)

    04/23/2020, 8:10 PM
    Hey all! Hoping to see you tomorrow at the cantina, I’m especially hoping to get a discussion going about code organization regarding contrib / plugins / related PyPI packages (especially for prefect server). We threw some
    contrib
    subdirectories into prefect last week to make it easier to get code into core and I am interested in everyone’s opinions 🙂
    🙌 3
    :upvote: 5
  • j

    Joe Schmid

    04/24/2020, 6:16 PM
    Kinda low-level question on Environments. If I want to access the parameters for a flow run in an Environment, it looks like I could do the following in the Environment's `execute()`: (taken from
    CloudFlowRunner
    ):
    def execute(  # type: ignore
            self, storage: "Storage", flow_location: str, **kwargs: Any  # type: ignore
        ) -> None:
            flow_run_id = prefect.context.get("flow_run_id")
    
            try:
                flow_run_info = self.client.get_flow_run_info(flow_run_id)
                except Exception as exc:
                self.logger.debug(
                    "Failed to retrieve flow state with error: {}".format(repr(exc))
                )
                if state is None:
                    state = Failed(
                        message="Could not retrieve state from Prefect Cloud", result=exc
                    )
                raise ENDRUN(state=state)
            updated_parameters = flow_run_info.parameters or {}  # type: ignore
    Is there a better way to do this than fetching the flow run info?
    j
    b
    • 3
    • 9
  • j

    Joe Schmid

    04/24/2020, 6:20 PM
    (And @Laura Lorenz (she/her) I would totally come to the cantina to discuss, but I've got an internal call at that time at every week... 😞)
    👍 1
    l
    • 2
    • 3
  • l

    Laura Lorenz (she/her)

    04/24/2020, 8:39 PM
    As discussed at cantina, if you want to be involved with event driven flows, we are discussing over in #contributors-event-driven-flows
  • s

    sk4la

    04/28/2020, 3:27 PM
    Hi all! Just a quick word to tell you that I absolutely love the project and what it's becoming! 💯
    🚀 8
    ✨ 5
    💯 6
  • l

    Laura Lorenz (she/her)

    04/28/2020, 6:51 PM
    set the channel topic: Task library livestream Fri 5/1; Core Contributors Cantina Fri 5/8
  • l

    Laura Lorenz (she/her)

    04/30/2020, 6:25 PM
    Hey all! I’m noodling on an integration with Great Expectations, so if you have an interest in that library or in data validation/data assertions generally, I have a discussion issue open with some questions and value your thoughts! : https://github.com/PrefectHQ/prefect/issues/2436
    💯 3
    :upvote: 7
    🚀 5
    j
    s
    j
    • 4
    • 5
  • j

    Joe Schmid

    04/30/2020, 8:45 PM
    Another pretty low-level question: I'm debugging a custom Environment and would like to: 1. Run with local agent (that part works fine, can register a local flow & have it run, etc.) 2. See debug logging during the Flow run / Environment execute(). I can enable & see debug output for the local agent, but I think since the flow run happens in a subprocess all I get is:
    [2020-04-30 20:35:05,397] INFO - agent | Found 1 flow run(s) to submit for execution.
    [2020-04-30 20:35:05,397] DEBUG - agent | Updating states for flow run f7912fd5-8ce3-4048-8cfe-965a15d46845
    [2020-04-30 20:35:05,400] DEBUG - agent | Flow run f7912fd5-8ce3-4048-8cfe-965a15d46845 is in a Scheduled state, updating to Submitted
    [2020-04-30 20:35:05,505] INFO - agent | Deploying flow run f7912fd5-8ce3-4048-8cfe-965a15d46845
    [2020-04-30 20:35:05,519] DEBUG - agent | Submitted flow run f7912fd5-8ce3-4048-8cfe-965a15d46845 to process PID 46552
    [2020-04-30 20:35:05,608] DEBUG - agent | Completed flow run submission (id: f7912fd5-8ce3-4048-8cfe-965a15d46845)
    [2020-04-30 20:36:22,277] INFO - agent | Process PID 46552 returned non-zero exit code
    Is there a good way to (a) have the flow runner in process or (b) see log output from the subprocess?
    c
    • 2
    • 11
  • t

    Troy Köhler

    05/01/2020, 6:07 PM
    Hi there! Would love to ask are there issues you (experienced developers and team of prefect) prefer me to get? I need something for person who just start getting familiar with the project, but probably not just one line issues.
    j
    l
    • 3
    • 9
  • l

    Laura Lorenz (she/her)

    05/01/2020, 9:29 PM
    set the channel topic: Core Contributors Cantina Fri 5/8 at 4pm EDT. Hangout link: meet.google.com/quz-zuea-kix
  • l

    Laura Lorenz (she/her)

    05/08/2020, 4:47 PM
    Friendly reminder about Cantina today at 4pm EDT at meet.google.com/quz-zuea-kix — and friendly reminder to wear a floral pattern if you want to match Alex 😉
    🌸 2
  • a

    Alex Cano

    05/11/2020, 2:17 PM
    @Jeremiah @Laura Lorenz (she/her) I wanted to touch base again on the flow concurrency PR (https://github.com/PrefectHQ/prefect/pull/2382)… specifically on where to put the concurrency check in the flow state setting pipeline. I thought of 2 points that I wanted to either make sure were non-issues from past experiences or figure out how to deal with. I’m going to attach 2 images in separate posts, one for each scenario. They attempt to show what I’m going for, but I’ve never been great at diagramming! I also wanted to make sure I understood the interaction once a flow is marked as
    Submitted
    by the Agent. After this, from what I can tell each flow handles the transition from
    Submitted
    to
    Running
    by means of the
    CloudFlowRunner
    , so each
    CloudFlowRunner
    is calling the
    set_flow_run_states
    with its own flow run, right? So maybe the conditions below don’t matter? First is since the server is written using async, if we put the concurrency check in the
    api.states.set_flow_run_state
    call, and an API call comes in with N flow runs that are trying to transition into a
    Running
    state and there’s M concurrency slots where M < N, how can we guarantee only M flows will transition into the
    Running
    state? Second, when we’re creating the “Run Queue” in the
    api.runs.get_runs_in_queue
    , we’re specifically returning flow runs in the order of first scheduled. Related to the above, if we’re submitting flow runs to a
    Running
    state, are we guaranteeing that state changes in the flow runs submitted in the
    set_flow_runs_states
    mutation are occurring in order? If there are 3 flow runs and only one concurrency slot, are we just guaranteeing that one of the flow runs in the payload will succeed? Or are we guaranteeing that the first flow run in the payload will succeed?
    j
    • 2
    • 3
  • a

    Alex Cano

    05/11/2020, 2:18 PM
    First Scenario: Async concurrency checks
  • a

    Alex Cano

    05/11/2020, 2:19 PM
    Second Scenario: Out of Order
  • l

    Laura Lorenz (she/her)

    05/13/2020, 6:36 PM
    Hey contributors! FYI I am hosting as Prefect this Saturday 12-4 EDT for Pycon’s Hatchery program “Mentored Sprints for Diverse Beginners”. If you want to stop by to sprint or to help mentor, check out the details at https://us.pycon.org/2020/hatchery/mentoredsprints/. It is designed to be super beginner friendly, and they are also looking for independent and/or git/github mentors if that is at all your jam. It is going to be hosted over discord which should be fun! If you’re so gung ho you want to be an ‘official’ mentor for Prefect for the Sunday event, LMK and I will add you to our roster, I think it gets you some extra discord role so people know they can reach out to you ✌️
    ❤️ 4
    :marvin: 4
    e
    • 2
    • 3
  • l

    Laura Lorenz (she/her)

    05/13/2020, 6:38 PM
    set the channel topic: Livestream on Vue.js Server dev 5/15 (

    https://youtu.be/YHqfJwFvTFY▾

    ); Pycon Mentored sprints 5/17 on Pycon’s discord; Cantina 5/22 (meet.google.com/quz-zuea-kix)
    ❤️ 3
  • i

    itay livni

    05/22/2020, 2:53 PM
    What time is the Cantina today?
  • l

    Laura Lorenz (she/her)

    05/22/2020, 2:53 PM
    4pm EDT!
    ✔️ 1
  • l

    Laura Lorenz (she/her)

    05/22/2020, 10:05 PM
    set the channel topic: Prefect User’s Meetup 5/29 (https://zoom.us/j/92072258198); Core Cantina 6/5 (meet.google.com/quz-zuea-kix)
  • n

    Nate Atkins

    05/22/2020, 10:09 PM
    Sorry I missed Cantina today. Hope everyone with a holiday on Monday has a great long weekend!
    🙂 1
    l
    • 2
    • 1
  • a

    Alex Cano

    05/26/2020, 8:18 PM
    Does the prefect client located at
    prefect.client.client.Client
    only work for Cloud?
    c
    z
    • 3
    • 7
  • i

    itay livni

    05/29/2020, 2:11 PM
    Hi - Looking for feedback and collaboration. DRAFT PIN X:  Update Flow update https://docs.google.com/document/d/1JmMxolHxwQfWdwqzTpOCk_qhc4-YqU_bZSKgfvBN1QY/edit?usp=sharing
    a
    l
    • 3
    • 5
  • l

    Laura Lorenz (she/her)

    05/29/2020, 9:36 PM
    set the channel topic: Core Cantina 6/5 (meet.google.com/quz-zuea-kix)
  • i

    itay livni

    06/04/2020, 5:06 PM
    Hi - I'd like some opinions and direction on where methods to join multiple flows should be written. https://github.com/PrefectHQ/prefect/pull/2719 . file:///home/ilivni/Downloads/disjoint_union.png @Laura Lorenz (she/her) summed up everything nicely here. https://github.com/PrefectHQ/prefect/issues/1745#issuecomment-637698235 Currently
    Flow
    has one method,
    update
    . My concern to continue this approach is twofold: (1) Adding more code to a core piece of prefect and thus adding complexity -- That are not really essential to
    Flow
    (2) The idea of doing direct operations on a
    Flow
    can lead to undesirable effects (???). On the flip side is it's convenient:
    node_resources_fl.disjoint(fl)
    flow_lst = [get_secondary_sources_fl, get_primary_sources_fl, choose_defintions_fl]
    
    # Test disjoint
    node_resources_fl = Flow("node_resources")
    for fl in flow_lst:
        node_resources_fl.disjoint(fl)
    node_resources_fl.visualize()
    The other way would be something along the lines of a module in
    core.utilities
    . And would be structured like networkx operators.
    from prefect.utilities.flow_operators import disjoint_union
    new_flow = disjoint_union(get_primary_sources_fl, get_secondary_sources_fl)
    c
    • 2
    • 9
  • i

    itay livni

    06/04/2020, 5:11 PM
    c
    • 2
    • 1
  • s

    Scott Zelenka

    06/04/2020, 5:14 PM
    when specifying a Docker storage environment, I'd love to just point it to a directory and have it copy over the whole directory to a source directory inside the container. Is anyone working on something similar to accomplish this?
    d
    c
    • 3
    • 3
  • a

    Alex Cano

    06/04/2020, 11:22 PM
    For the
    Flow
    state lifecycle, is there any specific reason the
    Client.set_flow_run_state(flow_run_id, state)
    call doesn’t return the new state? Or is it just that, unlike the task version, the flow state lifecycle is much more streamlined and it just hasn’t been needed up until now?
    j
    • 2
    • 3
Powered by Linen
Title
a

Alex Cano

06/04/2020, 11:22 PM
For the
Flow
state lifecycle, is there any specific reason the
Client.set_flow_run_state(flow_run_id, state)
call doesn’t return the new state? Or is it just that, unlike the task version, the flow state lifecycle is much more streamlined and it just hasn’t been needed up until now?
j

Jeremiah

06/04/2020, 11:24 PM
I think that’s exactly it - was only required for tasks up til now
a

Alex Cano

06/04/2020, 11:25 PM
Gotcha! Okay, was realizing the PR for flow concurrency limiting would probably require flows to be able to be set to
Queued
as a state, so wanted to make sure I understood why it was the way it was before starting to hit it with a wrench 🙂
j

Jeremiah

06/04/2020, 11:26 PM
:yes:
View count: 4