https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • a

    Anna Geller

    04/26/2022, 6:42 PM
    Hi, Prefectionists! 👋 Want to chat about best practices in workflow orchestration? We've just created a new channel #best-practices-orchestration. Especially as we continue to iterate on Prefect 2.0 to be the best workflow orchestration engine everyone will love working with, we welcome a dialogue about best practices and your expectations! 😛refect2: :cloud2: Also, if you want to chat about various trends in the data industry or share some tricks to work with data more efficiently, the new channel #data-tricks-and-tips is the best place to start a conversation. Lastly, to celebrate how :marvin-duck: conquers the world, we started a channel #marvin-in-the-wild posting pictures from places Marvin has visited so far. Make sure to join the channels - we are excited about all data and workflow orchestration chats. Happy Engineering!
    :upvote: 2
  • s

    Slackbot

    04/26/2022, 6:55 PM
    This message was deleted.
    ✅ 1
    k
    • 2
    • 2
  • d

    Dylan

    04/26/2022, 7:20 PM
    Have a few questions on flow boilerplate and k8's templates
    k
    • 2
    • 6
  • m

    Michael Moscater

    04/26/2022, 7:52 PM
    Hello, is it possible to have a parent flow run a mapped create_flow_run task with a flow and multiple parameters, so the the created flows would be able to use the different parameters passed?
    k
    • 2
    • 7
  • j

    Josephine Douglas

    04/26/2022, 11:33 PM
    Hello! I am trying to create a flow that runs a task, sends the data resulting from that task to a secondary flow, waits for the secondary flow to run successfully, then runs another task. [code in thread] The primary and secondary flows are both registered in the same tenant and project but are set to use different images. When I try to run the primary flow, I get this error:
    Failed to load and execute Flow's environment: TypeError("got an unexpected keyword argument 'raise_final_state'")
    What am I doing incorrectly here?
    :discourse: 1
    k
    a
    • 3
    • 7
  • s

    Slackbot

    04/27/2022, 1:32 AM
    This message was deleted.
    :discourse: 1
    k
    a
    • 3
    • 6
  • r

    Raviraja Ganta

    04/27/2022, 6:28 AM
    I am new to Prefect and want to understand couple of things. • I want to run flows in GPU instance. This means that I have to run the prefect agent in GPU. Does this instance needs to be switched on all the time? How can we scale down/up based on the requests? • I want some other system to trigger the flows (backend systems in my org). For that I am thinking to write APIs for all the flows and trigger them when an API is called. If flows are invoked like this, will this still be shown in the UI? • How exactly prefect agent knows what code to run? Does prefect core code needs to be running all the time? • What are some best practices used for building applications using prefect? - I am getting confused regarding how code is managed Thank you for the help provided 🤗
    a
    • 2
    • 1
  • f

    Florian Guily

    04/27/2022, 10:08 AM
    Hello guys, quick questions about retries behaviour. I have a terminal mapped task that has retry policy. The retries seem to work well but i'm a bit worried about the logs. The delay between each retries is 10 secondes and for testing purpose, it is a tiny mapping (tried with 6 children tasks) and the flow is launched locally from terminal. So, between each retries, there is time where there is no task to execute so prefect enters in a "waiting" state according to the log (
    Waiting for next available Task run at 2022-04-27T10:00:35.571259+00:00
    ). After a few seconds, a new log message states:
    Beginning Flow run for 'flow_name'
    and i have all of the previous log message of the previous tasks, as if the flow was re-executed. Is it normal ? Is the flow really reexecuted ? When executing this flow from prefect cloud, i don't see those logs so i'm a bit confused.
    a
    k
    • 3
    • 36
  • p

    Praveen Chaudhary

    04/27/2022, 10:25 AM
    i am able to see projects under command prefect get projects
    a
    • 2
    • 19
  • p

    Praveen Chaudhary

    04/27/2022, 10:25 AM
    but now able to see on UI
  • p

    Praveen Chaudhary

    04/27/2022, 10:25 AM
    how to fix that ?
  • s

    Sang Young Noh

    04/27/2022, 11:40 AM
    Hello, I'm very new here so apologies for the question if it sounds newbie-like - I'm currently reviewing some prefect 1.0 code with mind to refactor into prefect 2.0, and one of the functions for prefect that is currently being used is prefect.core.parameter.DateTimeParameter, which doesnt seem to be supported in prefect 2.0. Is that the correct interpretation? If so, what would be prefect 2.0 alternative for this? Thanks in advance
    👋 1
    a
    • 2
    • 52
  • b

    Baris Cekic

    04/27/2022, 12:30 PM
    Hello here, I am new to Prefect and run my first flow in my local K8S cluster and use
    KubernetesRun
    . The flow is trying to download the
    py
    file from a local git repo in the same local K8S cluster. But I got an
    SSL_VERIFICATION
    error even though the
    url/host
    for
    git storage
    is
    HTTP
    . Is there anyone having similar issue?
    a
    k
    • 3
    • 75
  • v

    Vivek Kaushal

    04/27/2022, 12:33 PM
    Hello folks, I am trying to run Prefect server on an EC2 instance using the instructions in Prefect Docs and am facing the following error while trying to get it up and running. Has anyone faced a similar issue in hosting Prefect server?
    Could not upgrade the database!
    graphql_1   | Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f25746773d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
    apollo_1    | Checking GraphQL service at <http://graphql:4201/health> ...
    towel_1     | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n  File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n    await self.run_once()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 37, in run_once\n    return await self.reschedule_flow_runs()\n  File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 89, in reschedule_flow_runs\n    limit=500,\n  File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n    as_box=not apply_schema,\n  File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n    as_box=as_box,\n  File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n    raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}]"}
    a
    k
    • 3
    • 11
  • j

    Jonathan Mathews

    04/27/2022, 1:24 PM
    Hi folks, wondering if it’s possible to get IP addresses of Prefect Cloud. I’d like to lock down the network rules for my agent running on AWS ECS, so restricting access to a fixed set of IP addresses would be useful. I’ve been googling but couldn’t find anything. Thanks
    a
    k
    • 3
    • 6
  • s

    Sash Stasyk

    04/27/2022, 1:42 PM
    Hey, is there a way to limit concurrency on a
    map
    call?
    k
    • 2
    • 1
  • m

    Marc Lipoff

    04/27/2022, 1:52 PM
    I'm trying to get a DaskExecutor on ECS/Fargate to work. And having issues. The flow hangs on
    Creating a new Dask cluster with None.fargate_cluster...
    . I'm running on prefect==0.15.6. My code is attached. The trouble is that it just hangs and I have no clue what it's "trying" to do.
    k
    • 2
    • 25
  • b

    Billy McMonagle

    04/27/2022, 3:27 PM
    Hello! Do you have any tips for monitoring my Prefect flows that fail before they can even begin?
    k
    a
    j
    • 4
    • 20
  • m

    Matthew Roeschke

    04/27/2022, 3:42 PM
    When using
    Task.map
    , is there a way to customize the repr of the child task besides seeing
    Mapped Child n
    where n is a number? For example
    Task.map(x=list("abc"), ...)
    , is there a way to see
    Mapped Child a
    instead?
    k
    • 2
    • 2
  • j

    Jonathan Mathews

    04/27/2022, 3:43 PM
    Another question on ECS deployment.. what’s the best way to get environment variables set from my task definition? Can I use
    os.environ.get
    ? I have also tried
    EnvVarSecret
    but that only seems to work within a flow?
    k
    • 2
    • 22
  • r

    Rio McMahon

    04/27/2022, 4:10 PM
    Question regarding automations: is there a way to assign an automation at the project level? E.g. I would like to setup a slack notification anytime any flow within a project fails to start at its scheduled time. Currently (I think) I need to manually select all relevant flows; this is fine in the short term but I would like to avoid having to do this each time I add a new flow to the project.
    a
    j
    • 3
    • 16
  • f

    FuETL

    04/27/2022, 4:35 PM
    Hey guys do we have a ETA when the 2.0 will be released to the broader audience?
    k
    a
    • 3
    • 3
  • b

    Ben Fisher

    04/27/2022, 4:56 PM
    Hi All. Quick question on how to create a Flow that references external files. In my case, I’m going to run a set of perhaps a dozen queries against a database, and I obviously don’t want to embed the SQL. I’d like to store the SQL external in files. Is that something I can do?
    k
    • 2
    • 6
  • f

    Frederick Thomas

    04/27/2022, 5:39 PM
    I have some weirdness going on with a flow that pulls from Azure secrets into the prefect.config object(?) Anyways, the main issue is that the code gives this error
    File "/mnt/data/prefect/venv/lib/python3.8/site-packages/box/box.py", line 490, in __getattr__
    raise BoxKeyError(str(err)) from None
    box.exceptions.BoxKeyError: "'Config' object has no attribute 'staging_db_constring'"
    However if I attempt to print out the config object the code works as expected. I haven't touched the config.toml file and Googling hasn't helped. Any help would be appreciated. Thanks
    k
    • 2
    • 6
  • a

    Alex Cannon

    04/27/2022, 5:41 PM
    Greetings all 👋 Is there a way to create a WebhookAction that includes a header in the payload via the Automations UI? If not would anyone be able to share a GraphQL query I could use to do this?
    k
    a
    m
    • 4
    • 16
  • m

    Milly gupta

    04/27/2022, 5:58 PM
    Hi All, Is there way to start a new flow using automations? I am looking to trigger a flow B when flow A has finished.
    k
    a
    • 3
    • 12
  • j

    Jai P

    04/27/2022, 6:29 PM
    👋 in a flow that contains subflows in prefect 2.0, is there a way to specify
    wait_for
    between the subflows? i'll drop a trivial example in the thread
    👋 1
    a
    • 2
    • 9
  • d

    Deepak Pilligundla

    04/27/2022, 6:31 PM
    Hi Team , We are trying to automate registering the prefect flows with Jenkins but not able to register it , can you please share a documentation which helps to register the flows
    a
    • 2
    • 2
  • c

    Chris Reuter

    04/27/2022, 6:45 PM
    👋 Hey all! PrefectLive is starting in 15 minutes - join DevOps king @Tyler Wanner as he works on a Kubernetes deployment in Cloud 2. You can find us on Twitch, and the replay will be available on our PrefectLive YouTube playlist.
    :upvote: 1
    🔥 2
    🎉 1
    • 1
    • 1
  • m

    Matt Delacour

    04/27/2022, 7:52 PM
    hi everyone 👋 We are interesting in using ECS Agent with Prefect 2.0. Is this something already available ? Cannot find anything related to that in the Orion doc so far. cc @Julian Mehnle
    k
    m
    +2
    • 5
    • 15
Powered by Linen
Title
m

Matt Delacour

04/27/2022, 7:52 PM
hi everyone 👋 We are interesting in using ECS Agent with Prefect 2.0. Is this something already available ? Cannot find anything related to that in the Orion doc so far. cc @Julian Mehnle
k

Kevin Kho

04/27/2022, 7:54 PM
I don’t think that has been implemented yet. Just the Kubernetes/Docker Flow Runners at the moment
m

Michael Adkins

04/27/2022, 8:04 PM
There have been a lot of difficulties integrating with AWS’s API in 1.0 so we’re intending to implement it very carefully this time 🙂
👍 2
💯 1
j

Julian Mehnle

04/27/2022, 8:20 PM
What would you recommend for a greenfield project that'll run on AWS, with a strong hesitance to adopt Kubernetes at this time? Create your own fleet of EC2 instances that run Docker?
(I'm working with @Matt Delacour on this.)
k

Kevin Kho

04/27/2022, 8:28 PM
Unless Michael corrects me, I think that is right. Agent on EC2 with Flows that use DockerFlowRunner
a

Anna Geller

04/27/2022, 8:46 PM
@Julian Mehnle in Prefect 2.0, this is even easier than that! Given that the same agent process can simultaneously deploy flow runs as Docker containers, subprocesses, or Kubernetes jobs, depending on which
flow_runner
type you attach to your
DeploymentSpec
. I have a rough draft of a tutorial for AWS EC2 agent and was planning to polish it more before publishing, but given you need it now, I will share it already
https://discourse.prefect.io/t/getting-started-with-cloud-2-0-using-ec2-instance-as-the-execution-layer/551/1
j

Julian Mehnle

04/27/2022, 8:53 PM
Thanks, that's useful. When would you say that ECS (not EC2) support is likely to be available in 2.0? Some time in 2022?
a

Anna Geller

04/27/2022, 8:54 PM
Would you be open to use EKS on Fargate instead of ECS Fargate?
j

Julian Mehnle

04/27/2022, 8:54 PM
Open, but less likely.
K8s introduces a whole other virtualization layer on top of a virtual data center that we're skeptical of dealing with for the foreseeable future.
FWIW, we can absolutely run a small EC2 instance fleet for now.
🙌 1
a

Anna Geller

04/27/2022, 9:01 PM
From the perspective of cost and convenience benefits, Fargate is definitely attractive, but in general ECS has a really convoluted API and it's hard to build an interface that will match user's expectations since everyone seems to want to use ECS differently 😄 Some want to use it with EC2, some with ECS data plane, then there are different capacity providers, complex networking setup that many users would want Prefect to infer automatically from a default VPC, while more advanced users want to build it via a custom VPC deployed with Terraform - a really tough problem tbh. That's why we need more time to think about the right way of exposing that API to users. In contrast, if you would use EKS on Fargate, it's "just" Kubernetes in the end which has a standardized API we can support out of the box with
KubernetesFlowRunner
. And for that, we already have this awesome tutorial from Laura:

https://youtu.be/bJtnx34YIYg▾

You could create an EKS on Fargate cluster using eksctl in a single command (LMK if you need help with that) and then follow the tutorial from Laura to deploy the execution layer
j

Julian Mehnle

04/27/2022, 9:04 PM
Understood. We will likely start with a small fleet of EC2 instances running the Prefect agent. We'll cross the K8s bridge if and when that becomes necessary.
👍 1
(And, yes, we're using Terraform to manage the underlying infrastructure.)
👍 1
View count: 4