What is the best practice for storing metadata about object locations (e.g. in various datalake zones) when using prefect to move this data around from zone to zone? We feel we lose track of where data lives or which data comes from which source after having initially processed it using a prefect Task if we do not somehow store metadata. However, we are unsure where we should be storing this metadata and if there are some best practices around it.

Hi, I'm able to set up a flow and orchestrate it with a deployment spec. Now I'd like to know what the best solution is for running a backfill set of runs. Could you give some pointers?

Can Prefect Orion Server APIs be used to build and orchestrate workflows with Java Developer knowledge. We dont have python developers in other teams, can Java developers build their own dockerized jobs and have them scheduled/orchestrated via Prefect Orion ?

Hi All, What a product! • Any tips on best practice for designing flows / sub-flows / tasks for batched web-requests in 2.0? Should tasks or sub-flows be the preferred function call for a batch? • Prefect is consuming a lot of RAM as it is fanning out to multiple web-request (i presume output / cache is stored for each request), is batching and then awaiting batches the best approach to avoid this? • Is there a way to view the parameters of a flow / task in the flow / task name in the UI? For example, for a task "collect" that performs a web request that collects id "x", it would be nice to see "collect-{id}-{task-id}" in the UI for the task run. Thanks!

Hi everyone, What’s the recommended approach in beta to splitting up alpha/beta/gamma environment? We’re in beta so there is a single workspace per email address that’s shared (perfect@someco.com), so we can’t do different envs per workspace. One thought was a single workspace, with a single S3 storage (storage cli is so weird BTW, no ability to remove a storage) with multiple work queues. Then there would be one agent per env, each consuming a different work-queue for some isolation? Then use tags to separate which flow runs go to which work queue? What are the recommendations in terms of architecture here?

Any tips on handling old images left over from a Local agent? Is there anyway for the Agent to clear those out after the flow runs? I’d rather not have to create a cron job to delete old images from the EC2 instance we are running on.

Is it possible to deploy or register a workflow from say agent 1 to run on agent 2? Do I need to log into host with agent 2 and register/deploy the flow from there?

Hi, if a task fails due to bad inputs from an upstream task, is there a way to modify the outputs of the upstream task before restarting the failed task? I'm using Prefect 1.2.1

We are currently using Github releases (tags more specifically) and CircleCI to publish flows into AWS ECR, we are doing this from a monolithic repository so that our data scientists can publish their flows in a standardized way without the need for them to worry about infrastructure, etc. We use a hash on the flow file to determine if the file has changed since the last release and deploy all the flows that have changed. We are running into an issue in Staging whereby one release from one person is clobbering the release made by another person (since they are working in different branches). Does anyone have a similar setup? How have you addressed the issue above? I supposed some kind of version ordering would address this issue. As in if the registered version in Cloud is > than the version to be deployed, then don’t deploy.

We were considering the following architecture in Azure: 1. Prefect Cloud 2. Azure container instances (as execution layer) However we have some issues when running the containers on Azure. Does anybody know any good tutorials in form of articles or videos that uses the Docker Agent with Docker? Furthermore, we are also considering the scalability of this setup in the long run. I assume that most deploy their own prefect server to kubernetes together with many pods as the execution layer. However, if we wanted to start smaller due to the cost. What could be a viable solution on Azure?

This message was deleted.

I'm triggering a flow from the cloud UI to run on kubernetes. Kubernetes agent is authenticated. The kubernetes agent is spawning jobs, but they fail with this error:

Hi all! I am wondering what is the best practice, in prefect 2.0 orion, to trigger a flow run if a run of another flow fails. My concrete use case is the following: I would like to send a slack message as an alert in case a periodic flow (let's call it _flow_A_) fails. I have already set up _flow_B_ that sends a slack message using prefect-slack collection. I just need to trigger it with some input if a run of _flow_A_ fails

Hi, what is the best way to run "subtasks"? I know i can just use in taskA

taskB.run()

but then it's treated as a normal python function (so no logging, and no other prefect's magic is available for this taskB) I managed to workaround this creating a Flow in a Task, something like this:

@task
def taskA(config: dict):
#...
    with Flow() as flowB:
        taskB()
#...
    a = flow.runB()

it seems to work - i get logs for flowB and taskB, etc I can visualize both flows using flow.visualize() - on two separate diagrams - but it's all right for me. But is this a correct approach, or should I do this some other way?

Hi, We are exploring alternative options to Airflow and were wondering if Prefect is compatible with build systems. We use bazel and some of our scripts are only visible when run through bazel. Can we configure flows to run through a build system?

Hi, can I generate flow diagram in Prefect 2.0 from code? In 1.0 i could use flow.visualize() to get pdf with diagram.

Hi! I'm running prefect agent with Azure VM, suggestions to save cost of the idle VM (I only need like 20 minutes nightly to run the flows)? Some built in functionality of Prefect that can help boot/shutdown to match with jobs

Hi! So I started playing with Prefect2.0 and here is my question: tasks / and flows are visible on radar - "normal" python functions are not. If I want to achieve "granularity" on the radar I have to use subflows? As I cannot run task from task ?

Hello again. How can I run the same Flow parallelly? I have used code from https://discourse.prefect.io/t/how-can-i-run-multiple-subflows-or-child-flows-in-parallel/96 and it works. But when I simplify it and modify to run same Flow (not different 4) then I get

RuntimeError("The task runner is already started!")

Modified code below:

import asyncio
from prefect import flow

@flow
async def subflow_1():
    print("Subflow 1 started!")
    await asyncio.sleep(1)


@flow
async def main_flow():
    parallel_subflows = [subflow_1(), subflow_1()]
    await asyncio.gather(*parallel_subflows)

if __name__ == "__main__":
    main_flow_state = asyncio.run(main_flow())

Hi folks, I've used Google Kubernetes engine a bit in a previous role, but we were running airflow on a VM and just sending jobs off to k8s, so it was pretty different than the parallel compute options we've got with Prefect. I switched to Prefect about a year ago (what a nice change), and I was most recently running a pretty simple setup with just a beefed up EC2 since my light pipelines were nowhere near hitting schedule conflicts. Now I'm looking to move my new company's pipelines over to Prefect and I need parallelization. So, looks like I need to go with ECS or EKS (we're on AWS). I haven't had to set up either of these services before. I guess my main question is: where does the agent actually sit? I get the config files etc., but this'll be my first time not running the agent on a VM, and the VM route is so simple because I just go into the machine and start the agent from the terminal. Where does the agent get initialized in the case of ECS or EKS?

Hi , how can i get unique id for every task that is run in parallel ? if i have task A and i run it with : runs = A.map(listOfInputs, unmapped(input2)) when i print this in A:

task_id = prefect.context.get("task_id", "")

i have the same task_id for every task A run …..

Hi there, Have you setup flow to run with different environment e.g. cloud vs local ? We have ECS agents running flows with label prod and uat, and we would like these flows to run with different run config when running locally for development. I hope my question is clear. many thanks

Hi team - have a question, we’re going to start implementing some scheduling infrastructure and I’m planning to implement Prefect, my first thought is to implement prefect 2.0 but the warning regarding “being in beta” and “don’t use for prod” flows concerns me. At the same time, I wonder if it may not be the best use of my time to learn Prefect 1.0 if 2.0 will be GA in the near future. Thoughts? Ideas? Food recommendations?

Hi Prefect Community! Does anyone have any idea how to set the Python Version for the Kubernetes Agent? Currently getting this error:

python: (flow built with '3.10.4', currently running with '3.7.13')")

But there doesnt seem to be any docs on how to set this python version in the agent deployment cli

hello all! what’s the best way to pass a Prefect keyword to a new Task class? I want to do something like

@task(key=value)

within a structure like

class MyNewTask(Task): 
    def run():
         pass

where would I pass those key/value pairs when initializing the new capital-T Task?

Hi guys, we have a general questions. We are considering Prefect 2.0 (not cloud) but question about the Queue - what's the queue engine in the backend? more specifically, if we need to transfer 1000 files (or 10000) - do we need to add external queue like rabbit or kafka or can prefect handle it good? would love your 2 cents folks! thanks

hi people, I'm new in prefect • i have a instance ec2(aws) that is running in prefect cloud • basically read some data from rds(aws) and store in S3(aws) • it is scheduled every day to do this process. i am running local agent in the same instance ec2 through screen (background), but I don't think this is not a good solution, does anybody know which solution will be suitable for my case? , thanks

Some of my prefect tasks need to use a database connection. Connections are not serializable, so tasks would have to construct one on run, but this is pricy. If you have a bunch of tasks that require the connection is there a way to provide them with something like a connection pool?

Hello, I'm trying out prefect and wondering if i can use an agent to invoke a .net process which already exists?