prefect-community #prefect-community

prefect-community

Anna Geller

04/26/2022, 6:42 PM

Hi, Prefectionists! 👋 Want to chat about best practices in workflow orchestration? We've just created a new channel #best-practices-orchestration. Especially as we continue to iterate on Prefect 2.0 to be the best workflow orchestration engine everyone will love working with, we welcome a dialogue about best practices and your expectations! 😛refect2: :cloud2: Also, if you want to chat about various trends in the data industry or share some tricks to work with data more efficiently, the new channel #data-tricks-and-tips is the best place to start a conversation. Lastly, to celebrate how :marvin-duck: conquers the world, we started a channel #marvin-in-the-wild posting pictures from places Marvin has visited so far. Make sure to join the channels - we are excited about all data and workflow orchestration chats. Happy Engineering!

:upvote: 2

Slackbot

04/26/2022, 6:55 PM

This message was deleted.

✅ 1

Dylan

04/26/2022, 7:20 PM

Have a few questions on flow boilerplate and k8's templates

Michael Moscater

04/26/2022, 7:52 PM

Hello, is it possible to have a parent flow run a mapped create_flow_run task with a flow and multiple parameters, so the the created flows would be able to use the different parameters passed?

Josephine Douglas

04/26/2022, 11:33 PM

Hello! I am trying to create a flow that runs a task, sends the data resulting from that task to a secondary flow, waits for the secondary flow to run successfully, then runs another task. [code in thread] The primary and secondary flows are both registered in the same tenant and project but are set to use different images. When I try to run the primary flow, I get this error:

Failed to load and execute Flow's environment: TypeError("got an unexpected keyword argument 'raise_final_state'")

What am I doing incorrectly here?

:discourse: 1

Slackbot

04/27/2022, 1:32 AM

This message was deleted.

:discourse: 1

Raviraja Ganta

04/27/2022, 6:28 AM

I am new to Prefect and want to understand couple of things. • I want to run flows in GPU instance. This means that I have to run the prefect agent in GPU. Does this instance needs to be switched on all the time? How can we scale down/up based on the requests? • I want some other system to trigger the flows (backend systems in my org). For that I am thinking to write APIs for all the flows and trigger them when an API is called. If flows are invoked like this, will this still be shown in the UI? • How exactly prefect agent knows what code to run? Does prefect core code needs to be running all the time? • What are some best practices used for building applications using prefect? - I am getting confused regarding how code is managed Thank you for the help provided 🤗

Florian Guily

04/27/2022, 10:08 AM

Hello guys, quick questions about retries behaviour. I have a terminal mapped task that has retry policy. The retries seem to work well but i'm a bit worried about the logs. The delay between each retries is 10 secondes and for testing purpose, it is a tiny mapping (tried with 6 children tasks) and the flow is launched locally from terminal. So, between each retries, there is time where there is no task to execute so prefect enters in a "waiting" state according to the log (

Waiting for next available Task run at 2022-04-27T10:00:35.571259+00:00

). After a few seconds, a new log message states:

Beginning Flow run for 'flow_name'

and i have all of the previous log message of the previous tasks, as if the flow was re-executed. Is it normal ? Is the flow really reexecuted ? When executing this flow from prefect cloud, i don't see those logs so i'm a bit confused.

Praveen Chaudhary

04/27/2022, 10:25 AM

i am able to see projects under command prefect get projects

Praveen Chaudhary

04/27/2022, 10:25 AM

but now able to see on UI

Praveen Chaudhary

04/27/2022, 10:25 AM

how to fix that ?

Sang Young Noh

04/27/2022, 11:40 AM

Hello, I'm very new here so apologies for the question if it sounds newbie-like - I'm currently reviewing some prefect 1.0 code with mind to refactor into prefect 2.0, and one of the functions for prefect that is currently being used is prefect.core.parameter.DateTimeParameter, which doesnt seem to be supported in prefect 2.0. Is that the correct interpretation? If so, what would be prefect 2.0 alternative for this? Thanks in advance

👋 1

Baris Cekic

04/27/2022, 12:30 PM

Hello here, I am new to Prefect and run my first flow in my local K8S cluster and use

KubernetesRun

. The flow is trying to download the

py

file from a local git repo in the same local K8S cluster. But I got an

SSL_VERIFICATION

error even though the

url/host

for

git storage

HTTP

. Is there anyone having similar issue?

Vivek Kaushal

04/27/2022, 12:33 PM

Hello folks, I am trying to run Prefect server on an EC2 instance using the instructions in Prefect Docs and am facing the following error while trying to get it up and running. Has anyone faced a similar issue in hosting Prefect server?

Could not upgrade the database!
graphql_1   | Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f25746773d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
towel_1     | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 37, in run_once\n    return await self.reschedule_flow_runs()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 89, in reschedule_flow_runs\n    limit=500,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n    raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}]"}

Jonathan Mathews

04/27/2022, 1:24 PM

Hi folks, wondering if it’s possible to get IP addresses of Prefect Cloud. I’d like to lock down the network rules for my agent running on AWS ECS, so restricting access to a fixed set of IP addresses would be useful. I’ve been googling but couldn’t find anything. Thanks

Sash Stasyk

04/27/2022, 1:42 PM

Hey, is there a way to limit concurrency on a

map

call?

Marc Lipoff

04/27/2022, 1:52 PM

I'm trying to get a DaskExecutor on ECS/Fargate to work. And having issues. The flow hangs on

Creating a new Dask cluster with None.fargate_cluster...

. I'm running on prefect==0.15.6. My code is attached. The trouble is that it just hangs and I have no clue what it's "trying" to do.

Billy McMonagle

04/27/2022, 3:27 PM

Hello! Do you have any tips for monitoring my Prefect flows that fail before they can even begin?

Matthew Roeschke

04/27/2022, 3:42 PM

When using

Task.map

, is there a way to customize the repr of the child task besides seeing

Mapped Child n

where n is a number? For example

Task.map(x=list("abc"), ...)

, is there a way to see

Mapped Child a

instead?

Jonathan Mathews

04/27/2022, 3:43 PM

Another question on ECS deployment.. what’s the best way to get environment variables set from my task definition? Can I use

os.environ.get

? I have also tried

EnvVarSecret

but that only seems to work within a flow?

Rio McMahon

04/27/2022, 4:10 PM

Question regarding automations: is there a way to assign an automation at the project level? E.g. I would like to setup a slack notification anytime any flow within a project fails to start at its scheduled time. Currently (I think) I need to manually select all relevant flows; this is fine in the short term but I would like to avoid having to do this each time I add a new flow to the project.

FuETL

04/27/2022, 4:35 PM

Hey guys do we have a ETA when the 2.0 will be released to the broader audience?

Ben Fisher

04/27/2022, 4:56 PM

Hi All. Quick question on how to create a Flow that references external files. In my case, I’m going to run a set of perhaps a dozen queries against a database, and I obviously don’t want to embed the SQL. I’d like to store the SQL external in files. Is that something I can do?

Frederick Thomas

04/27/2022, 5:39 PM

I have some weirdness going on with a flow that pulls from Azure secrets into the prefect.config object(?) Anyways, the main issue is that the code gives this error

File "/mnt/data/prefect/venv/lib/python3.8/site-packages/box/box.py", line 490, in __getattr__

raise BoxKeyError(str(err)) from None

box.exceptions.BoxKeyError: "'Config' object has no attribute 'staging_db_constring'"

However if I attempt to print out the config object the code works as expected. I haven't touched the config.toml file and Googling hasn't helped. Any help would be appreciated. Thanks

Alex Cannon

04/27/2022, 5:41 PM

Greetings all 👋 Is there a way to create a WebhookAction that includes a header in the payload via the Automations UI? If not would anyone be able to share a GraphQL query I could use to do this?

Milly gupta

04/27/2022, 5:58 PM

Hi All, Is there way to start a new flow using automations? I am looking to trigger a flow B when flow A has finished.

Jai P

04/27/2022, 6:29 PM

👋 in a flow that contains subflows in prefect 2.0, is there a way to specify

wait_for

between the subflows? i'll drop a trivial example in the thread

👋 1

Deepak Pilligundla

04/27/2022, 6:31 PM

Hi Team , We are trying to automate registering the prefect flows with Jenkins but not able to register it , can you please share a documentation which helps to register the flows

Chris Reuter

04/27/2022, 6:45 PM

👋 Hey all! PrefectLive is starting in 15 minutes - join DevOps king @Tyler Wanner as he works on a Kubernetes deployment in Cloud 2. You can find us on Twitch, and the replay will be available on our PrefectLive YouTube playlist.

:upvote: 1

🔥 2

🎉 1

Matt Delacour

04/27/2022, 7:52 PM

hi everyone 👋 We are interesting in using ECS Agent with Prefect 2.0. Is this something already available ? Cannot find anything related to that in the Orion doc so far. cc @Julian Mehnle

Title

Matt Delacour

04/27/2022, 7:52 PM

hi everyone 👋 We are interesting in using ECS Agent with Prefect 2.0. Is this something already available ? Cannot find anything related to that in the Orion doc so far. cc @Julian Mehnle

Kevin Kho

04/27/2022, 7:54 PM

I don’t think that has been implemented yet. Just the Kubernetes/Docker Flow Runners at the moment

Michael Adkins

04/27/2022, 8:04 PM

There have been a lot of difficulties integrating with AWS’s API in 1.0 so we’re intending to implement it very carefully this time 🙂

👍 2

💯 1

Julian Mehnle

04/27/2022, 8:20 PM

What would you recommend for a greenfield project that'll run on AWS, with a strong hesitance to adopt Kubernetes at this time? Create your own fleet of EC2 instances that run Docker?

(I'm working with @Matt Delacour on this.)

Kevin Kho

04/27/2022, 8:28 PM

Unless Michael corrects me, I think that is right. Agent on EC2 with Flows that use DockerFlowRunner

Anna Geller

04/27/2022, 8:46 PM

@Julian Mehnle in Prefect 2.0, this is even easier than that! Given that the same agent process can simultaneously deploy flow runs as Docker containers, subprocesses, or Kubernetes jobs, depending on which

flow_runner

type you attach to your

DeploymentSpec

. I have a rough draft of a tutorial for AWS EC2 agent and was planning to polish it more before publishing, but given you need it now, I will share it already

https://discourse.prefect.io/t/getting-started-with-cloud-2-0-using-ec2-instance-as-the-execution-layer/551/1

Julian Mehnle

04/27/2022, 8:53 PM

Thanks, that's useful. When would you say that ECS (not EC2) support is likely to be available in 2.0? Some time in 2022?

Anna Geller

04/27/2022, 8:54 PM

Would you be open to use EKS on Fargate instead of ECS Fargate?

Julian Mehnle

04/27/2022, 8:54 PM

Open, but less likely.

K8s introduces a whole other virtualization layer on top of a virtual data center that we're skeptical of dealing with for the foreseeable future.

FWIW, we can absolutely run a small EC2 instance fleet for now.

🙌 1

Anna Geller

04/27/2022, 9:01 PM

From the perspective of cost and convenience benefits, Fargate is definitely attractive, but in general ECS has a really convoluted API and it's hard to build an interface that will match user's expectations since everyone seems to want to use ECS differently 😄 Some want to use it with EC2, some with ECS data plane, then there are different capacity providers, complex networking setup that many users would want Prefect to infer automatically from a default VPC, while more advanced users want to build it via a custom VPC deployed with Terraform - a really tough problem tbh. That's why we need more time to think about the right way of exposing that API to users. In contrast, if you would use EKS on Fargate, it's "just" Kubernetes in the end which has a standardized API we can support out of the box with

KubernetesFlowRunner

. And for that, we already have this awesome tutorial from Laura:

https://youtu.be/bJtnx34YIYg▾

You could create an EKS on Fargate cluster using eksctl in a single command (LMK if you need help with that) and then follow the tutorial from Laura to deploy the execution layer

Julian Mehnle

04/27/2022, 9:04 PM

Understood. We will likely start with a small fleet of EC2 instances running the Prefect agent. We'll cross the K8s bridge if and when that becomes necessary.

👍 1

(And, yes, we're using Terraform to manage the underlying infrastructure.)

👍 1

View count: 4