https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • f

    Fina Silva-Santisteban

    04/05/2022, 8:27 PM
    Hi everyone! Is there a limit to how many service accounts you can have?? The button to add a new one is greyed out for us 🤔
    k
    a
    • 3
    • 6
  • j

    Jonathan Mathews

    04/05/2022, 8:36 PM
    Hi, if I’m deploying using ECS, do I have to store secrets (such as Snowflake credentials) within Prefect Cloud, or can I somehow keep them within my infrastructure? I’m guessing I could access them using AWS secrets manager, or within the docker image (prob not a great idea), but I’m wondering what the best practice is. Thanks
    💡 1
    k
    f
    • 3
    • 2
  • r

    Rhys Mansal

    04/05/2022, 9:29 PM
    Sorry to ask multiple questions but I have a flow that is running on ECR which fails with the following message
    An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified.
    I have set memory (and cpu) in the ECSRun task_definition (both quoted and unquoted) and still get this error. Does anyone know what I might try?
    :discourse: 1
    k
    a
    • 3
    • 8
  • m

    Matt Alhonte

    04/05/2022, 11:17 PM
    Upgrading to 1.0 (using the nightly version that has the Capacity Provider fix), digging a lot! Trying to test out the Dask Performance Reports - is there a way to dynamically generate a filename for it based on the Run's arguments? (not totally sure what Event Handlers have access to 😅 )
    k
    a
    • 3
    • 5
  • s

    Shiyu Gan

    04/06/2022, 1:29 AM
    is label the "primitive" to use for achieving some sort of tenancy isolation?
    k
    • 2
    • 6
  • s

    Shiyu Gan

    04/06/2022, 5:54 AM
    How different is Orion from 11.0? We recently began researching Prefect as a solution, shortly before Orion came out. Is it necessary to also evaluate Orion (in addition to 11.0) to form the most comprehensive view of prefect? Or will Orion co-exist with existing 11.0 because they solve different problems and neither will replace the other? Thanks!
    a
    • 2
    • 5
  • s

    Shiyu Gan

    04/06/2022, 6:32 AM
    What does
    *prefect.core.flow.Flow.run_agent*
    do? It's unclear from documentation and src code, which seems to say it runs an Agent from inside a Flow, which might be confusing since I was under the impression that Agent runs Flows, not the other way around.
    :discourse: 1
    a
    k
    m
    • 4
    • 7
  • s

    Shiyu Gan

    04/06/2022, 6:46 AM
    Are the
    services
    enumerated on this page still update to date and complete?
    a
    • 2
    • 1
  • s

    Stephen Lloyd

    04/06/2022, 7:44 AM
    What is the standard way to set up prefect code to handle promotion through various environments? (dev -> stage -> prod). I’m using Prefect Cloud and would like to use the Secrets manager.
    a
    • 2
    • 8
  • t

    Thomas Mignon

    04/06/2022, 8:22 AM
    Hi 🙂 I'm facing an error on an HPC cluster with my hello_world only with prefect 2:
    a
    • 2
    • 36
  • n

    Nicolaas Negron

    04/06/2022, 12:00 PM
    Hi all 😎 I'm going through the Orion docs and noticed that the "states" link under the FAQ > Deployment > How is Orion organized? section leads to a 404 page (w/ URL = "https://orion-docs.prefect.io/faq/concepts/states") Not sure if this is the appropriate place to post this (please redirect me if not!), I believe the appropriate link is to the "states" page is "https://orion-docs.prefect.io/concepts/states/#overview"
    👋 1
    a
    m
    • 3
    • 2
  • e

    Ewerton Henrique Marschalk

    04/06/2022, 12:42 PM
    Hi there, noob question here, there is an http api in prefect? I wan't to get the prefect task status on my zabbix, with an http request to prefect, so I can create trigger when something goes wrong, didnt find anything about in the docs
    👋 1
    a
    • 2
    • 4
  • а

    Андрій Демиденко

    04/06/2022, 1:18 PM
    Hi everybody, It seems like our team is going to have tons of flows. Is there any ideas how to create a hierarchy of the flows or to arrange them somehow? Thank you.
    :discourse: 1
    a
    • 2
    • 12
  • c

    Christoph Mayer

    04/06/2022, 1:56 PM
    Hi there, is it possible to use https://docs.prefect.io/api/latest/tasks/files.html file operators with files hosted on SFTP with user/pass? Or another to use prefect to fetch files from SFTP?
    k
    • 2
    • 2
  • h

    Haseeb Ahmad

    04/06/2022, 3:01 PM
    Hi guys. We're trying to set our log format to log everything JSON so we can consume it in Datadog and search the logs. We'd targetting the following format:
    {
      "flow_name": "",
      "flow_id": "",
      "task_name": "",
      "task_id": "",
      "message": "",
      "customer_id": ""
    }
    The questions we have are: 1. Is using
    PREFECT__LOGGING__FORMAT
    the correct place to specify the JSON format? 2. How can we handle this additional attribute
    customer_id
    . Our intent here is for most tasks that are related to our customers to do something like
    <http://logger.info|logger.info>('Something', customer_id)
    3. To the previous point, how can we extend this further such that we can add arbitrary parameters or objects to the log,
    <http://logger.info|logger.info>('Something', my_object,my_second_object)
    and have them show up nsted in the json?
    k
    • 2
    • 1
  • n

    Noosymer

    04/06/2022, 3:01 PM
    Hello all Is Prefect can be entirely installed on Kubernetes? If yes, is there a guide somewhere? To what I've understood, only some parts of the architecture can run in K8S. Am I wrong? Thank you
    k
    • 2
    • 2
  • c

    Chris Reuter

    04/06/2022, 3:35 PM
    See you at 3:15p Eastern today! https://prefect-community.slack.com/archives/C036FRC4KMW/p1649259298561779
    🚀 2
    :marvin: 2
  • f

    Fina Silva-Santisteban

    04/06/2022, 4:14 PM
    Hi everyone! I’m trying to use an S3List task https://docs.prefect.io/api/latest/tasks/aws.html#s3list but it seems like the bucket prefix is not an optional value even though it should be? 🤔 Boto3 docs lists it as optional (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects), but the task implementation doesn’t (see screenshot). This is what I try to do:
    s3_list_task = S3List()
    
            flow.set_dependencies(
                task=s3_list_task,
                keyword_tasks=dict(
                    bucket='my-bucket-name'
                )
            )
    And this is the error message:
    TypeError: run() missing 1 required positional argument: 'prefix'
    Does this mean I can’t use buckets without prefixes together with the S3 prefect tasks?
    k
    f
    • 3
    • 6
  • n

    Nicolaas Negron

    04/06/2022, 5:10 PM
    Total rookie question here -- I'm attempting to install and play around with Prefect 2.0 in PyCharm, having some trouble verifying the install was successful. When I attempt to check the version in PyCharm's terminal I receive the following error messages. I receive similar error messages when attempting to run simple python scripts listed in the First Steps tutorial. I created a python project in PyCharm and installed Prefect 2.0b2 using their settings interface. Any idea where I went wrong? Ive attached screenshots of the error messages and my project's Prefect directory.
    k
    • 2
    • 6
  • m

    Matthew Seligson

    04/06/2022, 5:19 PM
    We need alerting when a flow is submitted for execution but is never executed. Can we create an automation for this? Basically for every flow, send an email if a flow is submitted but not started after 60 seconds. Is this possible?
    :discourse: 1
    k
    m
    a
    • 4
    • 13
  • e

    Ewerton Henrique Marschalk

    04/06/2022, 6:02 PM
    hey, someone can help me with a query in graphql api? I wan't to collect all created flows in one query, but I wan't ONLY the LASTEST version of each, how can I do so?
    :discourse: 1
    k
    • 2
    • 3
  • f

    FuETL

    04/06/2022, 6:48 PM
    Is possible to debug flows like celery? Example: I have my flows registered and i want to be the one that will pull the flow to run (with debug) in pycharm
    k
    • 2
    • 3
  • c

    Chris Reuter

    04/06/2022, 7:00 PM
    PrefectLive is starting in 15 minutes! See you in chat! https://prefect-community.slack.com/archives/C036FRC4KMW/p1649259298561779
    :marvin: 1
    :themoreyouknow: 1
    l
    h
    • 3
    • 3
  • m

    Matt Yamkowy

    04/06/2022, 8:06 PM
    Hi. I'm just starting to look at Prefect 2.0. Is there a suggested way of handling a waiting period mid flow? For example if I start a flow, and after an initial task I want to wait for 5 minutes before continuing, do I just sleep for 5 minutes in a task? What about if I wanted to wait for days?
    :discourse: 1
    k
    a
    • 3
    • 15
  • r

    Rajan Subramanian

    04/06/2022, 8:09 PM
    is there a way programatically to get a list of deployments in prefect? i know we can do
    prefect deployment ls
    but i want to get all my created deployments...1000 of them and run them all. so im assuming i have to programatically execute prefect deployment run deploymenet_name for each deployment in my deployments
    :discourse: 1
    k
    • 2
    • 5
  • h

    Henrietta Salonen

    04/06/2022, 9:24 PM
    Hi, I have two projects in Prefect Cloud: one for dev & one for prod. Each project has its own a Docker agent using DockerRun for running flows in containers (in EC2) built based on images from Dockerhub. These images are being re-built with a CI/CD flow (using a Dockerfile) whenever we push code to our Git repository either in dev or prod (one image tag for dev and one for prod) The flow source code is stored in Github and this is configured as the storage for our flows. For some reason my flow runs in the
    prod
    project suddenly fail with
    Failed to load and execute flow run: ModuleNotFoundError("No module named 'project_name'")
    however, all the flows with exactly the same source code in Prefect
    dev
    project run just fine. When I pull these images and start containers locally the project structure and packages are the same for both dev and prod. What could be the issue here?
    k
    a
    • 3
    • 9
  • s

    Shiyu Gan

    04/06/2022, 9:42 PM
    Strictly speaking, do Tasks need to run in an environment that has the original "Task" function source code? Cloudpickle/pickle would lead me to think that the "bare minimum" necessary would be just an environment with task's py dependencies, but not the task source code themselves.
    a
    k
    • 3
    • 2
  • s

    Shiyu Gan

    04/06/2022, 9:49 PM
    What is the recipe for configuring a scheduled Flow's parameters based on the "scheduled time"? I found
    parameter_defaults
    for Clocks, but they seem to be static w.r.t each clock. For example, I want the parameter be "day of the week",i.e. Mon, Tue....
    a
    k
    • 3
    • 3
  • b

    Brad

    04/06/2022, 10:18 PM
    Hey team - having some troubles running docker tasks in a docker agent on windows (dind) ..
    a
    k
    • 3
    • 19
  • s

    Shiyu Gan

    04/06/2022, 10:25 PM
    What is the difference between lazarus and zombie killer?
    k
    • 2
    • 8
Powered by Linen
Title
s

Shiyu Gan

04/06/2022, 10:25 PM
What is the difference between lazarus and zombie killer?
k

Kevin Kho

04/06/2022, 10:30 PM
They are just separate services. Lazarus being responsible for restarting, and zombie killer responsible for marking stuff as failed. I don’t think you can look at the services and think why they are split or together because these were open sourced from Cloud but the Cloud versions may still have added functionality
s

Shiyu Gan

04/06/2022, 10:33 PM
If zombie killer marks a task as failed, will the "failed" status also be communicated/sent to the client flow/task?
k

Kevin Kho

04/06/2022, 10:34 PM
No because Zombie Killer marks things as Failed precisely because communication was lost, so it’s a direct change of state on the backend. Does that make sense?
s

Shiyu Gan

04/06/2022, 11:46 PM
It does. So what happens if the "failed" task actually recovers and completes its unit of work when ZK already marked it failed? Will the existence of a "Done" (on client side) and "Failed" (on cloud) cause inconsistency/conflict?
k

Kevin Kho

04/06/2022, 11:48 PM
When that task completes and tries to update state it will run into an error saying the state cant be modified
s

Shiyu Gan

04/07/2022, 12:00 AM
u rock!
Does ZombieKiller only handle Tasks or does it handle Flow as well?
k

Kevin Kho

04/07/2022, 12:44 AM
Ah that’s a good thought because I believe it’s Task but Lazarus can also trigger if the Flow never starts (gets stuck waiting for resources)
View count: 8