Hello, a dumb question, I am aware that prefect Parameters are used to define flow inputs. What would happen if I use python variables that were defined out of the flow instead of define Parameters and adding the variables in flow.run?

Hey 🙂 I tried to implement a subflow in my main flow (like this) :

@task
def subtask():
   return 1

with Flow("subflow") as subflow:
   subtask()

@task
def main_task():
   subflow.run()

with Flow("main_flow") as main_flow:
   main_task()

But I faced this Error : _Unexpected error while running flow: KeyError('Task slug init_dirs-1 is not found in the current Flow. This is usually caused by a mismatch between the flow version stored in the Prefect backend and the flow that was loaded from storage.\n- Did you change the flow without re-registering it?\n- Did you register the flow without updating it in your storage location (if applicable)?')_ I precise that I had already register the _main_flow_ (without the subflow inside) and it worked, but then I updated it and I registered the subflow, and it failed, could you help me please ? 🙏 The task "_init_dirs_" is in the main_flow, finishes in a success state, but this error append when subflow.run() is called (and init_dirs is not in the subflow)

Hi 🙂, I am scoring the documentation but haven’t found the answer yet: on prefect-cloud (in particular) is it possible to run a flow when it receives an http request? What would this look like in prefect?

Hi, I am trying to run a flow given its name and project. Since the client can only do this given an ID I was trying to find the ID via graphql. However I fail to construct a proper query to only find the active version of the flow.

{
  flow_group(
    where: {flows: {name: {_ilike: "%name%"}, archived: {_eq:false}}}  ) {
    id
    name
    flows{
      name
      id
      archived
    }
  }
}

Will not actually filter for not archived versions. What am I doing wrong?

Hi there, I am trying to get more information about how a Kubernetes Agent in prefect cloud is configured. The setup was done by a colleague and is working fine, but I have a hard time to understand which clusters/workloads within the Kubernetes Engine on GCP are actually used by the prefect agent. The config tab in the web UI is rather empty 🙂 Thanks for your help!

Hi, we have a Parameter that is a date type. This gives us a date picker in the UI, but it is setting a datetime instead of a date (see screenshot).

Can a flow with multi tasks be deployed as multi Kubernetes jobs, because each task has a different docker image?

Hello! I've been struggling with the "maximum number of scheduled runs per flow" problem. I'm aware that this is supposed to be solved as in https://github.com/PrefectHQ/server/pull/281, but I can't seem to put it to work with the Helm chart. I'm trying to set it in the

values.yaml

file such as

prefectConfig:
  services:
    towel:
      max_scheduled_runs_per_flow: 50

and when I try to upgrade my deployments with this, the towel pod doesn't get redeployed and the issue is still there. Is this way correct or should I do anything else?

Hey, is there any easy way to get notifications through email or slack if a flow run on prefect cloud fails?

Dear Prefect Community, What is the API payload I have to use to register a particular FLOW or TASK via APIs? Do I need to provide the entire Python code or the path of the .py file? I don't want to use CLI or Prefect UI to do the same. If you can provide an example with details, that would be great.

Hi everyone; I’m running into the following error:

ConnectionClosedError('Connection was closed before we received a valid response from endpoint URL: "<insert_s3_url_here>.prefect_result".',)

It is causing our flows to fail intermittently and I’m not sure how to debug this.

Hi everyone, I have a quick question about the scaling of prefect ECS agents and clusters of Fargate type. How exactly does it work? Is it automatic scaling or do I have to configure the auto scaling for ECS clusters myself?

Not sure how to start, but I wanted to talk a bit about a concern I have with predictable task usage and task-based billing. When I originally started looking into the pricing, I had the concern most people would have about the need to use a lot of tasks in prefect (fetching secrets, unpacking parameters, working with output from other tasks, etc.). Then I came across the following documentation (https://www.prefect.io/pricing/) which helped alleviate some of these fears.

Prefect Core encourages the use of small "helper" tasks. Won't those these inflate my billable usage?
Tasks that run for less than one second do not count as billable usage, so you can continue to design workflows that follow best practice and include as many helper tasks as you need.

Now that I’m using Prefect more and watching my task usage. I’ve noticed that both Secrets (EnvVarSecret in this case) which behind the scenes are a task and very simple tasks that just wrap parameters/task results into another structure vary quite a bit in runtime and often exceed the <1 second rule and thus will get billed. This is especially worrisome to me for secrets these tasks are necessary for securely handling things and some of our flows have quite a few secrets. I’m not sure what a solution would be, but wanted to see if anybody has noticed this and if there could be anything done to ensure these simple tasks don’t inflate actual bills.

Hi I am a beginner

I have a python script in my Jupyter notebook and want to connect this to prefect tool, can any one help me with that

Hi prefectionists, I have a question about running a flow locally with the CLI. I’ve been using

--param

to set parameters locally. I’ve just started working on a flow that will depend on a prefect secret, and would love to be able to set a value for the secret locally. Is there a way to do this? My naive hope was that

prefect run

would have a

--secret

flag similar to --param but this doesn’t seem to be the case

I’ve got a question. It seems when you use an idempotency key, the key sticks around for 24hrs even if the flow run finished, meaning you can’t run the flow with the same key again. The problem with this is that it would be nice to be able to say “while a flow is running with this key, another one cannot, but when the flow finishes, the key becomes available again.” Is there a way to do this? Is there some sort of graphql or anything else that can be executed to free up an idempotency key as soon as a flow finishes?

Hi, Has anybody execute a flow (registered in prefect cloud) using Azure Function?

Hello, I'm trying to download a file from s3 using a prefect docker agent , save it into some location (for e.g: ./data/file.txt) in my docker container, but the problem is when downloading the file is pulled down with some extension added to it something that looks like this: ./data/file.txt.405fgh and this is making difficult to access the file. Can someone tell me how to get rid of the extension that is being append to the filename? This is how the error appears on prefect cloud UI

Hey all, we ran into an unexpected (to us) behaviour today when a flow was considered successful even though some of it’s tasks failed and I’m curious to hear about potential solutions. We have this pattern of tasks in a flow with dependencies according to the arrows:

task1

->

task2

->

task3

Task3 is the last task of the flow and has other upstream dependencies which might cause task3 to be skipped in a flow run. Today a flow run got the following states:

Failed

->

TriggerFailed

->

Skipped

I assume the flow is considered successful since task3 was skipped and skipped states are considered successful (that makes sense). But we don’t want to consider a flow run successful if any tasks in the flow has a failed state. Refreshing my skills in the docs I’m thinking of handling it with state handlers, but I assume that a state handler on the flow would still see the flow state as successful due to the last skipped task. Is the only way forward here to have a state handler on each of the tasks in the flow?

How strongly unique are the <adjective>-<animal> Flow run names that Prefect Cloud generates? Should I append a timestamp/uuid if I want a strong guarantee against collisions?

is it possible to cancel a flow run through the gql api?

I've suddenly received the following error twice in a row on a scheduled flow that previously ran fine. Any ideas?

Hi - I am trying to find out if the upstream_tasks can take wildcards - I am kind of struggling to find the documentation on this kwarg.

Hi everyone, I stumbled upon an error as the prefect engine serializer tried to pickle a task result:

TypeError: cannot pickle 'lxml.etree.XMLSchema' object

Of course the result of one of my task is of this type and I would prefer not to change it. I tried to launch that process again with

config.flows.checkpointing = "false"

in the

~/.prefect/config.toml

file but I got the same error... This is an error that I cannot reproduce locally with

prefect run <flow>

. Do you guys have any leads on how I can solve this issue or at least reproduce the error locally?

Hi! I've been reading about tasks mapping, is there an equivalent to flow mapping ? Also, when mapping a task, are all tasks launched in parallel ? I would need to control the order of execution

Is there a good heuristic for understanding when to actually expect a performance improvement from converting a task that, say, operates over a list in sequence to a mapped task? I have a task that loops over hundreds of CSV files, parses and transforms the data, and writes the parsed data to an output file (one output file for each input file)...I was anticipating that converting this process to a mapped task and increasing my CPU count (I'm running this on Fargate via ECS, so I upped from 0.5 to 2 vCPUs in this specific case) would help out a lot by parallelizing this file-parsing procedure, but it actually has caused that node in my DAG to take much longer than it used to. I'm wondering if the overhead of this mapped task getting compiled and sent to the LocalDaskExecutor is just significantly outweighing the gains from the parallelization, because each mapped task is so small (the files being parsed are not large), and this would only make sense to do if the files were much, much larger?

Hey - has anyone here used Prefect with ML (or other data processing) and then outputted charts to a PDF or PNG? We’re exploring this as an option for getting insights into our ML models.

Hey all! Your friendly host of PrefectLive 📺 letting you know we won't be live on Twitch this week. We'll see you there next week though! There is a Fireside 🔥 Chat this Friday at 3p Eastern/12p Pacific where @Jeremiah and @Chris White will cover the 😛refect: 1.0 release candidate, the latest Orion updates and any questions the community might have. Sign up to attend here, or join the event directly on ! DM me any questions you want them to cover live 🙂