Hi everyone, I’ve tried registering a prefect flow using docker storage and ECS run config today and received a

pip

and

python 2.7

error? (pls see screenshots) The dockerfile I’m using for the docker storage (

flow.storage = Docker(dockerfile="Dockerfile", …)

looks like this

FROM python:3.9
WORKDIR /app
COPY . .
ENV PYTHONPATH=$PYTHONPATH:`pwd`
FROM <http://gcr.io/google-appengine/python|gcr.io/google-appengine/python>
RUN apt-get update
RUN apt-get install poppler-utils -y

I’ve set the python version to 3.9 in the dockerfile, why doesn’t it use that?

Hey Prefect peeps, Do you have an example of how we can use the

prefect.context.get("logger")

to log throught our entire project ? eg. when sub modules are called by calling their classes in that flow?

I got the following error when I try to invite a new user in Prefect Cloud. (excuse if this is not the place for this).

Hi, I’m having a DAG with basic fan-out, fan-in idea. Some of the fanned out tasks may fail, but I still want to continue the fan-in task for the succesful tasks. Thus if I have upstream: [x, y, z, FAILURE], the downstream still should process for input [x, y, z]. What I did; • Added trigger=all_finished to downstream task • Iterate over the input, filter out exception Although this worked fine locally while developing, I now got an edge case that one of the upstream tasks is TriggerFailed. How can I mitigate this? And, still process the succesfull upstreams?

Hi I am looking forward to try prefect cloud for better understanding of its advantages over prefect core server. Can anyone please share where can i get the access to free developer account for testing and trying prefect cloud.

Hi Prefect, I'm trying to work out how to re run a task in the UI. I have a flow that is very straight forward and the third task failed with the other two passing. Is there a way to re run this individual task and not just re run the entire flow? 🙏

Is there a way to edit the Automations Actions once they are created?

Hi everyone — I have a flow that is ending with “Flow run SUCCESS: all reference tasks succeeded” even though one of the tasks failed! Am I missing something? The task that fails is a DBTShellTask.. Problem is that I have a flow state handler that sends slack alerts if the overall flow state is FAILED, but that’s no happening so we’re not getting any alert..

Hi everyone, I have perhaps a more generic question, on AWS it is possible to run flows serverless using ECS (there is an appropriate agent/run_config for it). Are there alternatives on Azure/GCP to do the same (using e.g. cloud run or container instances)? And if so, how would you do that?

Hi, everyone! I have a question regarding performance of database operations: • Is every log entry stored on the database? • Is every task execution registered on the database? • Is there any built-in automatic history clean-up of old data? • Wouldn't it be better to use redis to avoid high frequency operations and thus high IO on the database? My concern is that if I proceed to break my flows in as many tasks as it might make sense I might cause a performance bottleneck on the DB, which seems counterproductive. Ideally (and it might not be possivle), I should be able to break any flow in as many tasks as I want without this decision affecting performance

Hi - I'm adding an automation to some of my flows using your api, immediately after I've deployed the flow. I'm calling the "create_flow_run_state_changed_hook_input" mutation to create a new hook each time with the complete list of flow_group_ids. Is this the best way to do it? In other words, I'm looking to confirm that there is no way to update the hook with a new list of flow_group_ids and that I have to create a new hook each time with the updated list of flow_ids and then delete the old hook.

Hi there! Sorry if it's a dumb question but I can't figure out what's wrong, I'm new to Prefect. My setup: • Prefect Cloud to orchestrate • a docker container with prefect installed In the container I registered a flow with

prefect register --project nike_dev --path flows/test.py

, it's sucessfuly registered, I can see it in my dashboard. I also have an agent running in the container

[2021-06-24 15:27:12,585] INFO - agent | Starting LocalAgent with labels ['93d6256207bd']
[2021-06-24 15:27:12,586] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2021-06-24 15:27:12,586] INFO - agent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2021-06-24 15:27:13,124] INFO - agent | Waiting for flow runs...

In the dashboard I manually scheduled the flow to run immediately but the agent doesn't pick it up https://d.pr/i/j55IrO. How can I debug it further? What can be wrong?

Hi all I followed this guide to set up my prefect enviornment with ecs fargate, and now I’m trying to use great expectations as part of the flow runs. As far as I know, the files for GE have to be on the same filesystem that’s running the flow, so does anyone know how I can get these files onto every ecs instance in a safe manner?

Hi there folks, I need a little hand from the

Prefect Cloud

team. I want to upgrade my Legacy account but I need to add the CTO as admin because he owns the credit card. Long story short: I demoted my user (I know, it's dumb 😖*) and I have no way of inviting the CTO so he can pay for the upgrade.* If someone can help me with this, please reach me via DM! Thanks :)

Greetings again - I have another question about automations and the API mutations for it. For automations related to state changes, you have a mutation called "create_flow_run_state_changed_hook" in which you provide a list of flow_group_ids to assign the hook to an action. It seems sla automations are handled differently. For these, I have to create flow group sla per flow (you can't provide a list of flows) using the "create_flow_group_sla" mutation. Is this understanding correct? What do I call to assign this sla to an action?

Hi there. I have a similar question as Gabriel. I am now testing the cloud (proof of concept) on my work mail, with my card. When we create a company account and add my mail to it, will I have problems with accounts?

Hi Prefect Team , I want to do POC for prefect cloud and wanted to know difference between standard and starter features. Can someone please elaborate on 1st and 4th point of standard pack in below shown picture.

Hi all, We use the Prefect cloud standard subscription and need invoices for the amounts charged to our credit card. I haven't been able to find this in the web UI. Can someone here help me get my hands on invoices? Thanks!

Hi - I have another question about automations, specifically state change automations. When creating the hook through the API using the mutation "create_flow_run_state_changed_hook_input", it says that if you leave the states argument empty in the mutation, all states will trigger the hook. However, I've done this and then looked at the hook from the API and I don't see any states. When I create the automation through the UI for all states, and then I look at the hook in the API, all the states are populated.

Is there a way to register flows recursively from a folder?

prefect register --project "my project" --path /api/flows/

registers only flows that are in

api/flows

but not in

api/flows/subfolder

Hello ! I've found a possible bug, can't investigate a lot more so I'll be posting here. I've been using Gitlab storage, and after a package update I started getting

GitlabGetError 404: Project Not Found

errors when fetching flows. Turns out

python-gitlab

broke something from 2.7.0 onwards. I assume it's not an issue for prefect to solve but maybe it can help someone.

Hi all, Using the Standard cloud subscription, somehow, we're now in a situation where some team members have the role Administrator others Read Only, and I'm unable to change these roles. Is it expected to not be able to change roles (as Administrator) on this plan? If so, how do we get all users to the same role (and what role would that be?). Thanks!

Hi guys, I am looking at the possibility to have loops in a flow. The existing LOOP feature that I found in the doc is interesting, but would it be possible to do the same with several tasks ? Like for example Task A produces a result, then Task B uses this result to produce another one, then Task C uses this second result, does its own thing with it and, if a condition is not met, calls A again. I know it sort of contradicts the meaning of a DAG, but it would be really useful for us... Is it an existing feature?

Hi - sorry to keep peppering yall with questions about the automations api. I'm trying to add/configure automations into my flow deployment process. I've gotten it figured out with state change automations, but I'm struggling with SLA automations. Basically, I'm trying to write a query to see if a SLA automation already exists for a given flow_group_id and if it does not, I will create the flow_sla_config, and then the flow_sla_failed_hook. So my question is, how can I query if a flow_sla_config exists for a given flow (using the flow_group_id)?

Hey all! Is there a way to see all flows as a graph? Having a lot of dependencies between them (the dependencies are managed with StartFlowRun) I'd like to visually see them and trigger specific flows manually. I'm using Prefect Cloud.

Double-checking - I need all the dependencies for a Flow installed on the machine that Registers it, right? Even if it runs elsewhere?

Seeing this.

You have no currently running Agents configured to pick up flow runs without labels

How do I set a Local Agent to pick up flow runs without labels? And/or see what labels my agents are looking for? I don't see where my Agent's labels are.

Hrm, just tried to use a flow with a

StartFlowRun

task, with the storage for both being in

git

. The underlying flow works, but the new flow fails with

GitProtocolError('Repository not found.')

Hello, has anyone run into Prefect registering the same script twice when calling

prefect register

from the CLI? Storage: S3 Execution: ECS Prefect version:

0.14.22

vscode ➜ /workspaces/flowstate/flows (main ✗) $ prefect register --project prefect_project_name -p project_path

Collecting flows...

Processing 'project_path/flow_one.py':
  Building `S3` storage...
[2021-06-26 10:10:44+0000] INFO - prefect.S3 | Uploading script /workspaces/github_project/flows/prefect_project_name/flow_one.py to flow_one/2021-06-26t10-10-44-047892-00-00 in aws-ecs-flows
  Registering 'flow_one'... Done
  └── ID: 41b7ea82-6cee-47c9-871c-0155267e6373
  └── Version: 1

Processing 'project_path/flow_two.py':
  Building `S3` storage...
[2021-06-26 10:10:45+0000] INFO - prefect.S3 | Uploading script /workspaces/github_project/flows/prefect_project_name/flow_one.py to flow_one/2021-06-26t10-10-44-047892-00-00 in aws-ecs-flows
-------------------------------------------------------------------------------------------------------------------- * SAME FILE * -------------------------------------------------------------
[2021-06-26 10:10:45+0000] INFO - prefect.S3 | Uploading script /workspaces/github_project/flows/prefect_project_name/flow_one.py to flow_two/2021-06-26t10-10-45-180059-00-00 in aws-ecs-flows
  Registering 'flow_two'... Done
  └── ID: f03469a7-c463-486f-a032-53d0fd4f265c
  └── Version: 1

======================== 2 registered ========================

This results in an error when attempting to execute `flow_two`:

Failed to load and execute Flow's environment: ValueError("Flow 'flow_two' not found in file. Found flows:\n- 'flow_one'")