Hi everyone — I'm have an ECS Agent kicking off some ECS tasks one the underlying EC2 instances (we are not using Fargate), and sometimes the agent tries to start the ecs task on an ec2 that doesn't have enough memory available, so the flow fails to start. Is there any way to tell the ECS agent to try again or something?

Hi all, I got a question regarding the `apply_map`function of Prefect. Is it possible to run this from within a task as well, as I am getting the following error:

ValueError("Couldn't infer a flow in the current context")

Hi everyone - I’m curious to hear how teams are organizing their

Projects

. Right now we just have one big project for all of our flows. Is there any benefit to breaking them out further? Is it possible to do things like “run all flows in project X” or are projects only for organizational aesthetics?

Hi everyone, This may seem a bit strange but I’m planning to use a Google Sheet as a front end where users can type in inputs. Then, users can click the submit button to trigger a Prefect flow that will read the Gsheet’s inputs, transform it, then write it back onto the Gsheet. I’m wondering if anyone has tried something similar and can offer feedback on my approach: 1. We have a button in Gsheets that will activate a Google Apps Script to do an API call 2. The API call (hosted on AWS API Gateway) will trigger a Lambda function, which will trigger a Prefect flow 3. Prefect flow will read data in Gsheets, transform the data, then write data in Gsheets Any suggestions or input is appreciated!

Does task decorator currently support python generators? Trying to see if tasks play nice with yield.

@task
def generator(limit: int):
    logger = prefect.context.get("logger")
    for i in range(limit):
        <http://logger.info|logger.info>(i)
        yield i*i

Hi, I’m trying to create a conditional Flow of Flows that runs a flow and then based on that flow’s final state either runs

flow_if_success

or

flow_if_failure

. Basically, I want to know how I can take the final state signal from

conditional-flow

and decide whether I want to run

flow_if_success

or

flow_if_failure

. I was having trouble matching the task signal with a value so that the case works properly. I was hoping you would be able to help me out. Would I need to use the

get_task_run_result

or is there a better way I can just grab the task result? Here is a snippet of my current test code:

var1 = Parameter("var1", default = 4)
var2 = Parameter("var2", default = 12)

conditional_id = StartFlowRun(flow_name="Conditional-Flow", project_name="Test", wait=True)
flow_if_success_id = StartFlowRun(flow_name="Flow_if_success", project_name="Test", wait=True)
flow_if_failure_id = StartFlowRun(flow_name="Flow_if_failure", project_name="Test", wait=True)


# Idea: Conditional flow runs first. If it succeeds with Success signal, run flow_if_success, if it fails with failure signal, run flow_if_failure
with Flow("Conditional FoF") as parent_flow:
    conditional_run = conditional_id()

    with case (conditional_run, Success): #Conditional_run never matches Success even if successful task run
        flow_if_success_run = flow_if_success_id(parameters=dict(num=var1))
    with case (conditional_run, Failed): 
        flow_if_failure_run = flow_if_failure_id(parameters=dict(number=var2))

I'm still trying to understand when I can set task attributes during the task constructor (vs when a copy of the task is made and the values are not carried over). Will post code example in 🧵

Is anyone aware of any CI vendors that follow a similar execution model to Prefect? i.e. one where the source code is never on servers owned by the vendor?

Getting this error on Prefect Cloud, but not locally.

Clients have non-trivial state that is local and unpickleable.

The load still seems to run OK, but it is marked as failure. How can I get rid of this error?

Hi All, How to run local or docker agent processes backend on a windows system? similar to Supervisor.

Hi all, is it possible to know the status of a flow with a particular run_name ? I want to query flow runs using python….we want to check if a flow is currently running else we will be creating a run for it.

What's the easiest way to cancel a flow based on a condition (like say you have a pipeline that checks for a file and only continues if it's found)?

I have been running prefect server for the last month and my containers and services have stayed up and healthy. I am not sure what changed in the last few hours but my agents can no longer connect to the server and the UI no longer renders. I am not sure how to debug this. From what I can see, the services are still up. For the agents, Timeout exception, the UI tries to load but doesn't render. I can shut down and restart all containers but I would like to keep the metadata stored in the database. Can anyone help?

Hi, I am trying to run prefect on azure AKS. we are using Helm, but not using Postgres subchart, but external postgres. it used to work until few days ago. when installing it we get in graghql

db-upgrade] Error applying Hasura metadata from /prefect-server/services/hasura/migrations/metadata.yaml

I could not find anything that changed from our side. any idea? Thanks 🙏

Hello, I am executing one flow in a remote distributed dask cluster. Previously, I executed my code locally and gathered the flow results in the local easy way (e.g.

subflow_res.result[deployed_model].result

). When I wrote the code, I read the docs, knowing this way was not valid for remote executions. I would like to know the proper way to gather results when working in a remote distributed environment. Can anyone help me?

is there (going to be) any way to run prefect 1.0 and 2.0 in the same installation for backwards compatibility? it'd be nice to have a way to experiment with one or two flows without mixing environments/branches

Docker image prefecthq/prefect:0.15.13-python3.9 comes with google-cloud-aiplatform==1.9.0 and google-cloud-storage=2.1.0 preinstalled. But google-cloud-aiplatform==1.9.0 requires google-cloud-storage >=1.32.0,<2.0.0dev ...

Hi everyone — I need to send some Prefect logs to Elasticsearch, is there a way to modify the prefect logger to do that in addition to what it already does?

Hey everyone, im looking to add a timeout to a task that has a max_retry on it. Does anyone know if the timeout is per retry or for all the retries combined?

does anyone know how to optimize the memory and cpu request values for a KubernetesRun? I ran a flow and would like to see how much of each is required for it, so I can request the correct values next time

Are flow and task UUIDs unique even across tenants and projects? How are they seeded and could we expect collisions of task ID’s between tenants or projects?

Hi all, I'm looking to run a command within my Docker container in production in order to generate a slack notification for a dbt open-source package. Is there a specific type of task I would need to write for this within my Python file? The command is as follows:

re_data notify slack \
--start-date 2021-01-01 \
--end-date 2021-01-31 \
--webhook-url <https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX> \
--subtitle="[Optional] Markdown text to be added as a subtitle in the slack message generated"

I have a case in which a LocalDaskExecutor task pulling data from a db gets the data and completes, but then never actually returns. It seems to stop without returning. I had a problem like this a release or three back in Prefect that resulted in a bug fix, but I haven't seen such an instance since. Is the issue back to our knowledge?

Does anyone have agent deployments in multiple cloud providers pointing to the same prefect cloud account? I have a use case where I plan to access some data in GCP, while my agents are deployed on AWS EKS clusters - I’ve considered going with some sort of peering approach, or simply standing up an agent on GCP GKE

Hi, do you may know how to add Context from the UI? I have variable defined in

[context]

local

/.prefect/config.yaml

but when released to Cloud, it says the CONTEXT not exists. But I don’t find where can create that on UI.

Hi, I am looking for a Prefect task for doing an BigQuery export to GCS. Something similar to BigQueryTask but instead of writing to a

table_dest

, to write to a GCS bucket. Is there a Task or a way to do this?

I'm running the following flow using

create_flow_run

to run a flow multiple times with different parameters. Here is my code and the error i'm getting.

Hi How to call outside flow from task. I am getting error

is it possible to get prefect to take in a boolean parameter? Or do all parameters need to be strings?

Hello, It might be a silly question but I can't really figure out the answer. One of the tasks of our flow is retried while it was previously already successful (see screenshot). And because this is a task calling a DELETE API endpoint, the retried task failed (because the object has already been removed). Although I could easily handle such thing, I would like to understand a bit more what's going on. I suspect something failed somewhere in the flow, and the flow is restarted from scratch, and all the tasks are restarted? I could not really find anything in the doc about that. What would cause a flow to retry a task that has been previously successful? Some precision: prefect core 0.15.12 prefect server 2022.01.12