Hello! I've been struggling with the "maximum number of scheduled runs per flow" problem. I'm aware that this is supposed to be solved as in https://github.com/PrefectHQ/server/pull/281, but I can't seem to put it to work with the Helm chart. I'm trying to set it in the

values.yaml

file such as

prefectConfig:
  services:
    towel:
      max_scheduled_runs_per_flow: 50

and when I try to upgrade my deployments with this, the towel pod doesn't get redeployed and the issue is still there. Is this way correct or should I do anything else?

Hey, is there any easy way to get notifications through email or slack if a flow run on prefect cloud fails?

Dear Prefect Community, What is the API payload I have to use to register a particular FLOW or TASK via APIs? Do I need to provide the entire Python code or the path of the .py file? I don't want to use CLI or Prefect UI to do the same. If you can provide an example with details, that would be great.

Hi everyone; I’m running into the following error:

ConnectionClosedError('Connection was closed before we received a valid response from endpoint URL: "<insert_s3_url_here>.prefect_result".',)

It is causing our flows to fail intermittently and I’m not sure how to debug this.

Hi everyone, I have a quick question about the scaling of prefect ECS agents and clusters of Fargate type. How exactly does it work? Is it automatic scaling or do I have to configure the auto scaling for ECS clusters myself?

Not sure how to start, but I wanted to talk a bit about a concern I have with predictable task usage and task-based billing. When I originally started looking into the pricing, I had the concern most people would have about the need to use a lot of tasks in prefect (fetching secrets, unpacking parameters, working with output from other tasks, etc.). Then I came across the following documentation (https://www.prefect.io/pricing/) which helped alleviate some of these fears.

Prefect Core encourages the use of small "helper" tasks. Won't those these inflate my billable usage?
Tasks that run for less than one second do not count as billable usage, so you can continue to design workflows that follow best practice and include as many helper tasks as you need.

Now that I’m using Prefect more and watching my task usage. I’ve noticed that both Secrets (EnvVarSecret in this case) which behind the scenes are a task and very simple tasks that just wrap parameters/task results into another structure vary quite a bit in runtime and often exceed the <1 second rule and thus will get billed. This is especially worrisome to me for secrets these tasks are necessary for securely handling things and some of our flows have quite a few secrets. I’m not sure what a solution would be, but wanted to see if anybody has noticed this and if there could be anything done to ensure these simple tasks don’t inflate actual bills.

Hi I am a beginner

I have a python script in my Jupyter notebook and want to connect this to prefect tool, can any one help me with that

Hi prefectionists, I have a question about running a flow locally with the CLI. I’ve been using

--param

to set parameters locally. I’ve just started working on a flow that will depend on a prefect secret, and would love to be able to set a value for the secret locally. Is there a way to do this? My naive hope was that

prefect run

would have a

--secret

flag similar to --param but this doesn’t seem to be the case

I’ve got a question. It seems when you use an idempotency key, the key sticks around for 24hrs even if the flow run finished, meaning you can’t run the flow with the same key again. The problem with this is that it would be nice to be able to say “while a flow is running with this key, another one cannot, but when the flow finishes, the key becomes available again.” Is there a way to do this? Is there some sort of graphql or anything else that can be executed to free up an idempotency key as soon as a flow finishes?

Hi, Has anybody execute a flow (registered in prefect cloud) using Azure Function?

Hello, I'm trying to download a file from s3 using a prefect docker agent , save it into some location (for e.g: ./data/file.txt) in my docker container, but the problem is when downloading the file is pulled down with some extension added to it something that looks like this: ./data/file.txt.405fgh and this is making difficult to access the file. Can someone tell me how to get rid of the extension that is being append to the filename? This is how the error appears on prefect cloud UI

Hey all, we ran into an unexpected (to us) behaviour today when a flow was considered successful even though some of it’s tasks failed and I’m curious to hear about potential solutions. We have this pattern of tasks in a flow with dependencies according to the arrows:

task1

->

task2

->

task3

Task3 is the last task of the flow and has other upstream dependencies which might cause task3 to be skipped in a flow run. Today a flow run got the following states:

Failed

->

TriggerFailed

->

Skipped

I assume the flow is considered successful since task3 was skipped and skipped states are considered successful (that makes sense). But we don’t want to consider a flow run successful if any tasks in the flow has a failed state. Refreshing my skills in the docs I’m thinking of handling it with state handlers, but I assume that a state handler on the flow would still see the flow state as successful due to the last skipped task. Is the only way forward here to have a state handler on each of the tasks in the flow?

How strongly unique are the <adjective>-<animal> Flow run names that Prefect Cloud generates? Should I append a timestamp/uuid if I want a strong guarantee against collisions?

is it possible to cancel a flow run through the gql api?

I've suddenly received the following error twice in a row on a scheduled flow that previously ran fine. Any ideas?

Hi - I am trying to find out if the upstream_tasks can take wildcards - I am kind of struggling to find the documentation on this kwarg.

Hi everyone, I stumbled upon an error as the prefect engine serializer tried to pickle a task result:

TypeError: cannot pickle 'lxml.etree.XMLSchema' object

Of course the result of one of my task is of this type and I would prefer not to change it. I tried to launch that process again with

config.flows.checkpointing = "false"

in the

~/.prefect/config.toml

file but I got the same error... This is an error that I cannot reproduce locally with

prefect run <flow>

. Do you guys have any leads on how I can solve this issue or at least reproduce the error locally?

Hi! I've been reading about tasks mapping, is there an equivalent to flow mapping ? Also, when mapping a task, are all tasks launched in parallel ? I would need to control the order of execution

Is there a good heuristic for understanding when to actually expect a performance improvement from converting a task that, say, operates over a list in sequence to a mapped task? I have a task that loops over hundreds of CSV files, parses and transforms the data, and writes the parsed data to an output file (one output file for each input file)...I was anticipating that converting this process to a mapped task and increasing my CPU count (I'm running this on Fargate via ECS, so I upped from 0.5 to 2 vCPUs in this specific case) would help out a lot by parallelizing this file-parsing procedure, but it actually has caused that node in my DAG to take much longer than it used to. I'm wondering if the overhead of this mapped task getting compiled and sent to the LocalDaskExecutor is just significantly outweighing the gains from the parallelization, because each mapped task is so small (the files being parsed are not large), and this would only make sense to do if the files were much, much larger?

Hey - has anyone here used Prefect with ML (or other data processing) and then outputted charts to a PDF or PNG? We’re exploring this as an option for getting insights into our ML models.

Hey all! Your friendly host of PrefectLive 📺 letting you know we won't be live on Twitch this week. We'll see you there next week though! There is a Fireside 🔥 Chat this Friday at 3p Eastern/12p Pacific where @Jeremiah and @Chris White will cover the P 1.0 release candidate, the latest Orion updates and any questions the community might have. Sign up to attend here, or join the event directly on ! DM me any questions you want them to cover live 🙂

Does Prefect with MySQL support this type of stored procedure? I got an error when the code runs fine querying our database directly -- ``SET @Date1 := MAKEDATE(YEAR(CURDATE()), 1) + INTERVAL QUARTER(CURDATE()) QUARTER - INTERVAL 1 QUARTER;`

SET procedure2 := bla bla

etc..

I am having trouble seeing task runs by task in Prefect Cloud. Here’s an example https://cloud.prefect.io/anaconda/task/b3cfcc12-9ba8-4960-8dee-7c677c23b477?runs, all of these pages that I’ve looked at are empty. I’ve tested this through the API and I can’t get it to return anything when using a

where

filter with a task id, but it works when I query by flow run id.

is there a way to provide custom names for tasks such as

case(False)

?

Hi Is there anything to consider when saving JSON as a

Result

using Prefect? I dump the dict objects to

str

JSONs using

json.dumps

and then save them using

S3Result

or

LocalResult

and noticed both of them get some extra characters added to the beginning or end of json object:

��QXQ >> to the beginning
�. >> to the end

which makes it un-parseable. Saving the same object using

with open()

on local machine does not bear those extra characters

Hi, I'm trying to create a flow that is dependent on 2 other flows. Is it possible (using the imperative API) to add tasks to the final flow?

I have a task (

task_B

) downstream of a mapped task (

task_A

). Some of the input for

task_B

(output of

task_A

) is

prefect.engine.signals.SKIP(None)

. In

task_B

, do I have to filter out these skip signals? If output of

task_B

is mapped to

task_C

...

task_X

, do I have to do

if isinstance(input, prefect.signals.SKIP): raise prefect.signals.SKIP

for every single task in the chain?

HI all, Hope you all are good and safe, Can I have a realtime task in prefect ETL?? Thanks in advance 🙏😁

Hi, I'm new to prefect and I'm facing a TypeError issue when I try to register my prefect workflow to cloud prefect. Does anyone have a solution ? Thanks in advance