https://prefect.io logo
Join Slack
Channels
ask-community
announcements
ask-marvin
best-practices
data-ecosystem
data-tricks-and-tips
events
feedback-deployment-concurrency
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
intros-test
livestream-chatter
marvin-ai
marvin-in-the-wild
pacc-apr-10-11-2024
pacc-apr-30-may-1-2024
pacc-apr-9-10-2025
pacc-aug-16-2023
pacc-aug-22-23-2023
pacc-aug-28-29-2024
pacc-aug-29-30-2023
pacc-clearcover-june-12-2023
pacc-dec-17-18-2024
pacc-feb-13-14-2024
pacc-feb-26-27-2025
pacc-jan-28-29-2025
pacc-july-11-12-2023
pacc-july-17-18-2023
pacc-july-30-31-2024
pacc-july-6-2023
pacc-june-14-2023
pacc-june-20-21-2024
pacc-london-2023
pacc-london-sept-2024
pacc-mar-12-13-2024
pacc-may-31-2023
pacc-nov-19-20-2024
pacc-nov-8-2023
pacc-nyc-may-2024
pacc-oct-11-12-2023
pacc-oct-1-2-2024
pacc-oct-16-17-2024
pacc-sept-13-14-2023
pacc-sept-20-21-2023
pacc-sept-26-27-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-contributors-archived
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-and-tell
Powered by
# ask-community
  • b

    Bruno Murino

    02/07/2022, 2:14 PM
    Hi everyone — I'm have an ECS Agent kicking off some ECS tasks one the underlying EC2 instances (we are not using Fargate), and sometimes the agent tries to start the ecs task on an ec2 that doesn't have enough memory available, so the flow fails to start. Is there any way to tell the ECS agent to try again or something?
    k
    a
    • 3
    • 9
  • f

    Fabrice Toussaint

    02/07/2022, 2:33 PM
    Hi all, I got a question regarding the `apply_map`function of Prefect. Is it possible to run this from within a task as well, as I am getting the following error:
    Copy code
    ValueError("Couldn't infer a flow in the current context")
    k
    • 2
    • 3
  • r

    Ryan Brennan

    02/07/2022, 2:39 PM
    Hi everyone - I’m curious to hear how teams are organizing their
    Projects
    . Right now we just have one big project for all of our flows. Is there any benefit to breaking them out further? Is it possible to do things like “run all flows in project X” or are projects only for organizational aesthetics?
    k
    a
    • 3
    • 3
  • k

    Ken Nguyen

    02/07/2022, 5:17 PM
    Hi everyone, This may seem a bit strange but I’m planning to use a Google Sheet as a front end where users can type in inputs. Then, users can click the submit button to trigger a Prefect flow that will read the Gsheet’s inputs, transform it, then write it back onto the Gsheet. I’m wondering if anyone has tried something similar and can offer feedback on my approach: 1. We have a button in Gsheets that will activate a Google Apps Script to do an API call 2. The API call (hosted on AWS API Gateway) will trigger a Lambda function, which will trigger a Prefect flow 3. Prefect flow will read data in Gsheets, transform the data, then write data in Gsheets Any suggestions or input is appreciated!
    a
    • 2
    • 6
  • h

    Heeje Cho

    02/07/2022, 5:20 PM
    Does task decorator currently support python generators? Trying to see if tasks play nice with yield.
    Copy code
    @task
    def generator(limit: int):
        logger = prefect.context.get("logger")
        for i in range(limit):
            <http://logger.info|logger.info>(i)
            yield i*i
    k
    • 2
    • 5
  • n

    Nick Hart

    02/07/2022, 6:17 PM
    Hi, I’m trying to create a conditional Flow of Flows that runs a flow and then based on that flow’s final state either runs
    flow_if_success
    or
    flow_if_failure
    . Basically, I want to know how I can take the final state signal from
    conditional-flow
    and decide whether I want to run
    flow_if_success
    or
    flow_if_failure
    . I was having trouble matching the task signal with a value so that the case works properly. I was hoping you would be able to help me out. Would I need to use the
    get_task_run_result
    or is there a better way I can just grab the task result? Here is a snippet of my current test code:
    Copy code
    var1 = Parameter("var1", default = 4)
    var2 = Parameter("var2", default = 12)
    
    conditional_id = StartFlowRun(flow_name="Conditional-Flow", project_name="Test", wait=True)
    flow_if_success_id = StartFlowRun(flow_name="Flow_if_success", project_name="Test", wait=True)
    flow_if_failure_id = StartFlowRun(flow_name="Flow_if_failure", project_name="Test", wait=True)
    
    
    # Idea: Conditional flow runs first. If it succeeds with Success signal, run flow_if_success, if it fails with failure signal, run flow_if_failure
    with Flow("Conditional FoF") as parent_flow:
        conditional_run = conditional_id()
    
        with case (conditional_run, Success): #Conditional_run never matches Success even if successful task run
            flow_if_success_run = flow_if_success_id(parameters=dict(num=var1))
        with case (conditional_run, Failed): 
            flow_if_failure_run = flow_if_failure_id(parameters=dict(number=var2))
    a
    • 2
    • 13
  • a

    Andrea Haessly

    02/07/2022, 6:23 PM
    I'm still trying to understand when I can set task attributes during the task constructor (vs when a copy of the task is made and the values are not carried over). Will post code example in 🧵
    k
    • 2
    • 10
  • b

    Brian Phillips

    02/07/2022, 7:55 PM
    Is anyone aware of any CI vendors that follow a similar execution model to Prefect? i.e. one where the source code is never on servers owned by the vendor?
    t
    • 2
    • 2
  • t

    Tim Enders

    02/07/2022, 8:40 PM
    Getting this error on Prefect Cloud, but not locally.
    Clients have non-trivial state that is local and unpickleable.
    The load still seems to run OK, but it is marked as failure. How can I get rid of this error?
    k
    • 2
    • 2
  • d

    David Yang

    02/07/2022, 9:20 PM
    Hi All, How to run local or docker agent processes backend on a windows system? similar to Supervisor.
    k
    a
    • 3
    • 8
  • v

    Vamsi Reddy

    02/08/2022, 12:08 AM
    Hi all, is it possible to know the status of a flow with a particular run_name ? I want to query flow runs using python….we want to check if a flow is currently running else we will be creating a run for it.
    k
    • 2
    • 16
  • d

    Daniel Saxton

    02/08/2022, 1:19 AM
    What's the easiest way to cancel a flow based on a condition (like say you have a pipeline that checks for a file and only continues if it's found)?
    k
    • 2
    • 1
  • o

    Ovo Ojameruaye

    02/08/2022, 7:58 AM
    I have been running prefect server for the last month and my containers and services have stayed up and healthy. I am not sure what changed in the last few hours but my agents can no longer connect to the server and the UI no longer renders. I am not sure how to debug this. From what I can see, the services are still up. For the agents, Timeout exception, the UI tries to load but doesn't render. I can shut down and restart all containers but I would like to keep the metadata stored in the database. Can anyone help?
    a
    k
    • 3
    • 49
  • a

    Amichai Ben Ami

    02/08/2022, 12:16 PM
    Hi, I am trying to run prefect on azure AKS. we are using Helm, but not using Postgres subchart, but external postgres. it used to work until few days ago. when installing it we get in graghql
    Copy code
    db-upgrade] Error applying Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml
    I could not find anything that changed from our side. any idea? Thanks 🙏
    k
    • 2
    • 2
  • a

    Antonio Manuel BR

    02/08/2022, 12:58 PM
    Hello, I am executing one flow in a remote distributed dask cluster. Previously, I executed my code locally and gathered the flow results in the local easy way (e.g.
    subflow_res.result[deployed_model].result
    ). When I wrote the code, I read the docs, knowing this way was not valid for remote executions. I would like to know the proper way to gather results when working in a remote distributed environment. Can anyone help me?
    k
    • 2
    • 6
  • b

    Brett Naul

    02/08/2022, 1:19 PM
    is there (going to be) any way to run prefect 1.0 and 2.0 in the same installation for backwards compatibility? it'd be nice to have a way to experiment with one or two flows without mixing environments/branches
    k
    • 2
    • 5
  • t

    Thomas Pedersen

    02/08/2022, 2:37 PM
    Docker image prefecthq/prefect:0.15.13-python3.9 comes with google-cloud-aiplatform==1.9.0 and google-cloud-storage=2.1.0 preinstalled. But google-cloud-aiplatform==1.9.0 requires google-cloud-storage >=1.32.0,<2.0.0dev ...
    k
    m
    • 3
    • 3
  • b

    Bruno Murino

    02/08/2022, 2:54 PM
    Hi everyone — I need to send some Prefect logs to Elasticsearch, is there a way to modify the prefect logger to do that in addition to what it already does?
    k
    • 2
    • 2
  • a

    Archie Kennedy-Dyson

    02/08/2022, 3:09 PM
    Hey everyone, im looking to add a timeout to a task that has a max_retry on it. Does anyone know if the timeout is per retry or for all the retries combined?
    👍 1
    k
    • 2
    • 2
  • a

    Andrew Lawlor

    02/08/2022, 3:24 PM
    does anyone know how to optimize the memory and cpu request values for a KubernetesRun? I ran a flow and would like to see how much of each is required for it, so I can request the correct values next time
    k
    • 2
    • 1
  • m

    Matthew Seligson

    02/08/2022, 5:42 PM
    Are flow and task UUIDs unique even across tenants and projects? How are they seeded and could we expect collisions of task ID’s between tenants or projects?
    k
    • 2
    • 3
  • m

    Madison Schott

    02/08/2022, 8:19 PM
    Hi all, I'm looking to run a command within my Docker container in production in order to generate a slack notification for a dbt open-source package. Is there a specific type of task I would need to write for this within my Python file? The command is as follows:
    Copy code
    re_data notify slack \
    --start-date 2021-01-01 \
    --end-date 2021-01-31 \
    --webhook-url <https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX> \
    --subtitle="[Optional] Markdown text to be added as a subtitle in the slack message generated"
    k
    • 2
    • 27
  • t

    Tom Shaffner

    02/08/2022, 8:42 PM
    I have a case in which a LocalDaskExecutor task pulling data from a db gets the data and completes, but then never actually returns. It seems to stop without returning. I had a problem like this a release or three back in Prefect that resulted in a bug fix, but I haven't seen such an instance since. Is the issue back to our knowledge?
    k
    z
    • 3
    • 116
  • l

    Leon Kozlowski

    02/08/2022, 9:47 PM
    Does anyone have agent deployments in multiple cloud providers pointing to the same prefect cloud account? I have a use case where I plan to access some data in GCP, while my agents are deployed on AWS EKS clusters - I’ve considered going with some sort of peering approach, or simply standing up an agent on GCP GKE
    k
    • 2
    • 8
  • t

    Tony Yun

    02/08/2022, 9:51 PM
    Hi, do you may know how to add Context from the UI? I have variable defined in
    [context]
    local
    /.prefect/config.yaml
    but when released to Cloud, it says the CONTEXT not exists. But I don’t find where can create that on UI.
    k
    • 2
    • 5
  • g

    Gaurang Katre

    02/08/2022, 10:25 PM
    Hi, I am looking for a Prefect task for doing an BigQuery export to GCS. Something similar to BigQueryTask but instead of writing to a
    table_dest
    , to write to a GCS bucket. Is there a Task or a way to do this?
    a
    • 2
    • 7
  • i

    Ifeanyi Okwuchi

    02/08/2022, 10:39 PM
    I'm running the following flow using
    create_flow_run
    to run a flow multiple times with different parameters. Here is my code and the error i'm getting.
    k
    • 2
    • 28
  • v

    vinoth paari

    02/09/2022, 5:00 AM
    Hi How to call outside flow from task. I am getting error
    k
    • 2
    • 17
  • k

    Kevin

    02/09/2022, 5:07 AM
    is it possible to get prefect to take in a boolean parameter? Or do all parameters need to be strings?
    k
    m
    a
    • 4
    • 8
  • r

    Romain

    02/09/2022, 7:10 AM
    Hello, It might be a silly question but I can't really figure out the answer. One of the tasks of our flow is retried while it was previously already successful (see screenshot). And because this is a task calling a DELETE API endpoint, the retried task failed (because the object has already been removed). Although I could easily handle such thing, I would like to understand a bit more what's going on. I suspect something failed somewhere in the flow, and the flow is restarted from scratch, and all the tasks are restarted? I could not really find anything in the doc about that. What would cause a flow to retry a task that has been previously successful? Some precision: prefect core 0.15.12 prefect server 2022.01.12
    k
    • 2
    • 5
1...250251252...509Latest