Can i have two agents logged in to two different workspaces, running on the same pc?

Can i access the log database hosted in orion cloud if i want to make a local dashboard based on flow runs'?

Hi here, how can I run same subflow multiple times parallely?

can anybody let me know? Thanks in advance

hello Prefect team, hello community, is there more information available on how to make the ray autoscaler and prefect 2 task concurrency limits work smoothly together? currently, when i submit a large number of tasks with a concurrency limit tag, the ray autoscaler will see the large amount of tasks and scale accordingly, however only a small number of tasks is allowed to run concurrently due to tag limits and most of the upscaled resources won't be used, which causes unnecessary costs especially for tasks with high resource requirements. i think this is due to the fact that the concurrency limit check happens during propose_state which happens inside the call that is submitted to the task runner. is there any way how this can be avoided? thanks a lot for any input.

Prefect v1 question: is there any way to use task looping with a function that creates tasks? I have a group of tasks that I'm using with

apply_map

and what I'd really like to do is have each of those be the body of a loop. It looks like you can only loop one single task though. Am I missing something? Thanks!

could someone advise how to get orion report api logs? tried

PREFECT_DEBUG_MODE

PREFECT_LOGGING_LEVEL

PREFECT_LOGGING_SERVER_LEVEL

. Non helps

hi! i am self hosting prefect (orion as well as the flows) in kubernetes cluster . i used https://github.com/PrefectHQ/prefect-helm to setup my orion server and https://github.com/anna-geller/dataflow-ops-aws-eks/blob/main/.github/workflows/eks_prefect_agent.yml for deploying agents. im using kubernetes job block as the infrastructure block and s3 block for the storage. My flows have high memory usage compared to cpu usage ( tending to 1-2 cores and 12gb memory) im running 5 flows at a time with 2000-3000 batches of 50 tasks and running 1000s of flows throughout the day . i have implemented task concurrency limits and flow concurrency limits on every work queue. Im consistently experiencing flow run crashes . Besides concurrency limits , what options do i have to control the load and/or are there any methods to control the resource usage of the jobs in my cluster?

Hi! I’m also self hosting Orion in a K8s cluster, and running most of our flows as K8sJobs with success. However, we have some flows that need to be run on Fargate as ECSTasks due to some circumstances. The agents spins up the ECSTask just fine, but on startup of prefect inside the ecstask prefect tries to contact prefect-orion, which inevitably fails because it attempts a kube-dns hostname. (the orion instance also isn’t exposed to the outside) I was under the assumption that it only needed a channel to the agent, but thinking about it does make sense that it needs Orion as well. Is this correctly assessed? Any workarounds to this? We would at the very least need to make it target a proper host instead of the kube-dns name. Then perhaps the two can safely be allowed to connect. Self-hosted Orion doesnt have auth, so exposing it to 0.0.0.0/0 might not be an option.

When using

from prefect.tasks.shell import ShellTask

When using

from prefect.tasks.shell import ShellTask

it throws out and error

ModuleNotFoundError: No module named 'prefect.tasks.shell'; 'prefect.tasks' is not a package

I know i can use

prefect-shell

to overcome this and run python scripts but i couldn't find any resolution on the above error

Hi Guys! New to Prefect, I have made good progress integrating Prefect with Snowflake and able to write queries and manage workflows. But, I was wondering if there is prefect integration with Snowpark?

Hey, another query for you guys 🙂 Is it possible to serve self-hosted Orion on a path. Putting it on a hostname + path without doing anything else makes index.html reference assets at root, and not at the path at which the actually now are reachable! Is it possible to let Orion now at which path it’s hosted at? Any input appreciated !

Hello Folks. Question. I have a flow defined with several task using "map" I want to run a final task once all map tasks have finished. This is my code:

with Flow(
    FLOW_NAME,
    run_config=UniversalRun(labels=LABELS),
    terminal_state_handler=workflow_terminal_state_handler,
) as flow:

    file_list = list_unprocessed_files(var)

    dataframes = get_dataframes.map(file_list)

    dataframes = filter_dataframe.map(
        dataframes, resource_type_name=unmapped(var)
    )

    # this is the final task
    print_process_summary_log()

Hi there, I'm working on migration a flow from Prefect 1 to 2. I use Snowflake, and am having some trouble with the `prefect-dbt`/`prefect-snowflake` collections. I think there is an issue related to my use of private key Snowflake authentication. Here are some details... by the way, I know many of us have slowed down for the holidays and I'm not in a rush for a response.

Anyone running into issues launching Orion from the latest prefect 2.7.4 docker image? My container is crashing on with these logs (check thread for logs), thanks @User for pointing it out

Hi Prefect Team - We now have a working dev environment with Prefect 2. The ECS Service is running the agent properly. However, for each flow we are creating a new task definition. Will we need to create a separate ECS Service to run the flows? If we create a separate ECS Service and Task Definition for the flows, do we put this task definition in the Prefect block for the ECS Task?

How do I import a function from another file in a Prefect 1.0 script? My Prefect script is stored in a folder in my repo. I also stored the function script within the same folder, but whenever I try to execute the Flow from Prefect IO it fails (No Module Found Error) here is my example: github |_my_repo|_ |folder| |function file |my flow

Merry Christmas 🥳 🎄

I hope this is a relatively simple question: using prefect 2.0, how can I get a task to return two values? as a function it is something like

@task
def f(val):
   return val * 2, val * 4

then the call in my flow looks like

a, b = f.map([1, 2, 3])

but I get ValueError: not enough values to unpack (expected 2, got 1) I have confirmed that the code works as expected when not decorated with mapping over

@task

(that is, two inputs are returned)

i know prefect 1.0 had

nout

as a task arg, but I can't find the equivalent in the docs...

#needHelp Using Prefect 2.7, I'm trying to connect a MYSQL databse to write data through a prefect task. once I run

cursor.executemany(query, values),

I run the

connection.commit()

to to transfer all data from buffer to the table in dataset. Error faced in the following command:

connection.commit()

Error:

00:12:33.556 | ERROR   | Task run 'load_records-41eb504c-0' - Encountered exception during execution:
Traceback (most recent call last):
  File "/home/shubhamwani/nice/lib/python3.8/site-packages/prefect/engine.py", line 1346, in orchestrate_task_run
    result = await run_sync(task.fn, *args, **kwargs)
  File "/home/shubhamwani/nice/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 69, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/home/shubhamwani/nice/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/shubhamwani/nice/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/shubhamwani/nice/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/shubhamwani/nice/prefect_flows/load/prefect2_tasks.py", line 166, in load_records
    (
TypeError: cannot unpack non-iterable NoneType object

I've tried using different sql connectors(mysqlclient, mysql-connector, pymysql), but the error persists. is this somehow related to prefect? anyone else facing this issue? Any help is welcomed, Thanks.

prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/2f48cb6f->
0049-4569-879d-7123d9113e31/workspaces/9da885d9-9d9f-4ee9-9b42-84bfb4a3d9aa/work_queues/name/queue_5'
Response: {'exception_message': 'Internal Server Error'}

Getting this error in prefect cloud , code got stuck with this issue. #prefect-community #prefect-cloud

Hi i am trying to restart the docker agent which is running on linux. i am not able to restart the docker agent

and the flows are failing as below

anybody who can add some views on this

Hi, it looks like the agent isn’t healthy for some reason. Are there logs from the agent?

Hi, I’m trying to move my Prefect 2 setup to an Azure serverless setup. The Prefect Agent is deployed in an Azure Container Instance (ACI). Flows are ran with an azure-container-instance-job block as infra. When I run the agent locally the flow properly executes, but I get “*KeyError: “No class found for dispatch key ‘azure-container-instance-job’ in registry for type ‘Block’.“”,* when executing through the container-hosted agent (image prefecthq/prefect:2-python3.9) . Any clues?

Hello everyone, I have a quick couple of questions around persistent input and result storage: 1) will restarted mapped tasks use results for the partial mapped result or does the task need to finish completely? 2) will the stored results be used even if there are changes to the code in the task if params are the same? Apologies if these are easily answerable through the docs, I couldn't locate the answer in my first search. I am also currently using v1. Thanks, Ollie

Is there any way to check if my flow is a sub flow or not? thank you.