Cross posting on ALL the channels haha but as mentioned in #pycon2020 I’m hosting an all-day Zoom call for Prefect contributors, feel free to stop by before 7 pm EDT! 🙂 https://zoom.us/j/92984979179

how would I go about searching for active work on a particular topic? Specifically, I'd like to add a basic async wrapper around some of the GraphQL queries from the client. The use case I'm working on is wrapping the Prefect queries behind a web interface for users in our company which don't have (and don't need) credentials to Prefect Cloud, but we want to expose the ability for them to trigger a Flow Run through a web UI. The web UI is written in an ASGI framework.

set the channel topic: Core Contributors Cantina Fri 4/24 at 4pm EDT meet.google.com/quz-zuea-kix

Hey all! Hoping to see you tomorrow at the cantina, I’m especially hoping to get a discussion going about code organization regarding contrib / plugins / related PyPI packages (especially for prefect server). We threw some

contrib

subdirectories into prefect last week to make it easier to get code into core and I am interested in everyone’s opinions 🙂

Kinda low-level question on Environments. If I want to access the parameters for a flow run in an Environment, it looks like I could do the following in the Environment's `execute()`: (taken from

CloudFlowRunner

):

def execute(  # type: ignore
        self, storage: "Storage", flow_location: str, **kwargs: Any  # type: ignore
    ) -> None:
        flow_run_id = prefect.context.get("flow_run_id")

        try:
            flow_run_info = self.client.get_flow_run_info(flow_run_id)
            except Exception as exc:
            self.logger.debug(
                "Failed to retrieve flow state with error: {}".format(repr(exc))
            )
            if state is None:
                state = Failed(
                    message="Could not retrieve state from Prefect Cloud", result=exc
                )
            raise ENDRUN(state=state)
        updated_parameters = flow_run_info.parameters or {}  # type: ignore

Is there a better way to do this than fetching the flow run info?

(And @Laura Lorenz (she/her) I would totally come to the cantina to discuss, but I've got an internal call at that time at every week... 😞)

As discussed at cantina, if you want to be involved with event driven flows, we are discussing over in #contributors-event-driven-flows

Hi all! Just a quick word to tell you that I absolutely love the project and what it's becoming! 💯

set the channel topic: Task library livestream Fri 5/1; Core Contributors Cantina Fri 5/8

Hey all! I’m noodling on an integration with Great Expectations, so if you have an interest in that library or in data validation/data assertions generally, I have a discussion issue open with some questions and value your thoughts! : https://github.com/PrefectHQ/prefect/issues/2436

Another pretty low-level question: I'm debugging a custom Environment and would like to: 1. Run with local agent (that part works fine, can register a local flow & have it run, etc.) 2. See debug logging during the Flow run / Environment execute(). I can enable & see debug output for the local agent, but I think since the flow run happens in a subprocess all I get is:

[2020-04-30 20:35:05,397] INFO - agent | Found 1 flow run(s) to submit for execution.
[2020-04-30 20:35:05,397] DEBUG - agent | Updating states for flow run f7912fd5-8ce3-4048-8cfe-965a15d46845
[2020-04-30 20:35:05,400] DEBUG - agent | Flow run f7912fd5-8ce3-4048-8cfe-965a15d46845 is in a Scheduled state, updating to Submitted
[2020-04-30 20:35:05,505] INFO - agent | Deploying flow run f7912fd5-8ce3-4048-8cfe-965a15d46845
[2020-04-30 20:35:05,519] DEBUG - agent | Submitted flow run f7912fd5-8ce3-4048-8cfe-965a15d46845 to process PID 46552
[2020-04-30 20:35:05,608] DEBUG - agent | Completed flow run submission (id: f7912fd5-8ce3-4048-8cfe-965a15d46845)
[2020-04-30 20:36:22,277] INFO - agent | Process PID 46552 returned non-zero exit code

Is there a good way to (a) have the flow runner in process or (b) see log output from the subprocess?

Hi there! Would love to ask are there issues you (experienced developers and team of prefect) prefer me to get? I need something for person who just start getting familiar with the project, but probably not just one line issues.

set the channel topic: Core Contributors Cantina Fri 5/8 at 4pm EDT. Hangout link: meet.google.com/quz-zuea-kix

Friendly reminder about Cantina today at 4pm EDT at meet.google.com/quz-zuea-kix — and friendly reminder to wear a floral pattern if you want to match Alex 😉

@Jeremiah @Laura Lorenz (she/her) I wanted to touch base again on the flow concurrency PR (https://github.com/PrefectHQ/prefect/pull/2382)… specifically on where to put the concurrency check in the flow state setting pipeline. I thought of 2 points that I wanted to either make sure were non-issues from past experiences or figure out how to deal with. I’m going to attach 2 images in separate posts, one for each scenario. They attempt to show what I’m going for, but I’ve never been great at diagramming! I also wanted to make sure I understood the interaction once a flow is marked as

Submitted

by the Agent. After this, from what I can tell each flow handles the transition from

Submitted

to

Running

by means of the

CloudFlowRunner

, so each

CloudFlowRunner

is calling the

set_flow_run_states

with its own flow run, right? So maybe the conditions below don’t matter? First is since the server is written using async, if we put the concurrency check in the

api.states.set_flow_run_state

call, and an API call comes in with N flow runs that are trying to transition into a

Running

state and there’s M concurrency slots where M < N, how can we guarantee only M flows will transition into the

Running

state? Second, when we’re creating the “Run Queue” in the

api.runs.get_runs_in_queue

, we’re specifically returning flow runs in the order of first scheduled. Related to the above, if we’re submitting flow runs to a

Running

state, are we guaranteeing that state changes in the flow runs submitted in the

set_flow_runs_states

mutation are occurring in order? If there are 3 flow runs and only one concurrency slot, are we just guaranteeing that one of the flow runs in the payload will succeed? Or are we guaranteeing that the first flow run in the payload will succeed?

First Scenario: Async concurrency checks

Second Scenario: Out of Order

Hey contributors! FYI I am hosting as Prefect this Saturday 12-4 EDT for Pycon’s Hatchery program “Mentored Sprints for Diverse Beginners”. If you want to stop by to sprint or to help mentor, check out the details at https://us.pycon.org/2020/hatchery/mentoredsprints/. It is designed to be super beginner friendly, and they are also looking for independent and/or git/github mentors if that is at all your jam. It is going to be hosted over discord which should be fun! If you’re so gung ho you want to be an ‘official’ mentor for Prefect for the Sunday event, LMK and I will add you to our roster, I think it gets you some extra discord role so people know they can reach out to you ✌️

set the channel topic: Livestream on Vue.js Server dev 5/15 (); Pycon Mentored sprints 5/17 on Pycon’s discord; Cantina 5/22 (meet.google.com/quz-zuea-kix)

What time is the Cantina today?

4pm EDT!

set the channel topic: Prefect User’s Meetup 5/29 (https://zoom.us/j/92072258198); Core Cantina 6/5 (meet.google.com/quz-zuea-kix)

Sorry I missed Cantina today. Hope everyone with a holiday on Monday has a great long weekend!

Does the prefect client located at

prefect.client.client.Client

only work for Cloud?

Hi - Looking for feedback and collaboration. DRAFT PIN X:  Update Flow update https://docs.google.com/document/d/1JmMxolHxwQfWdwqzTpOCk_qhc4-YqU_bZSKgfvBN1QY/edit?usp=sharing

set the channel topic: Core Cantina 6/5 (meet.google.com/quz-zuea-kix)

Hi - I'd like some opinions and direction on where methods to join multiple flows should be written. https://github.com/PrefectHQ/prefect/pull/2719 . file:///home/ilivni/Downloads/disjoint_union.png @Laura Lorenz (she/her) summed up everything nicely here. https://github.com/PrefectHQ/prefect/issues/1745#issuecomment-637698235 Currently

Flow

has one method,

update

. My concern to continue this approach is twofold: (1) Adding more code to a core piece of prefect and thus adding complexity -- That are not really essential to

Flow

(2) The idea of doing direct operations on a

Flow

can lead to undesirable effects (???). On the flip side is it's convenient:

node_resources_fl.disjoint(fl)

flow_lst = [get_secondary_sources_fl, get_primary_sources_fl, choose_defintions_fl]

# Test disjoint
node_resources_fl = Flow("node_resources")
for fl in flow_lst:
    node_resources_fl.disjoint(fl)
node_resources_fl.visualize()

The other way would be something along the lines of a module in

core.utilities

. And would be structured like networkx operators.

from prefect.utilities.flow_operators import disjoint_union
new_flow = disjoint_union(get_primary_sources_fl, get_secondary_sources_fl)

when specifying a Docker storage environment, I'd love to just point it to a directory and have it copy over the whole directory to a source directory inside the container. Is anyone working on something similar to accomplish this?