What's the correct way to spin up sub flows in parallel (like truly parallel, not concurrent with

asyncio

)? I'm doing the following for now, but it does not preserve flow-subflow relationships (and, it makes sense because the parent process isn't passing any prefect information to the child process in multiprocessing). The alternative here would be to use a Dask runner for all the subflows, in which case every task would be run truly in parallel, but all the subflows would be in the same process as the parent flow. However, it feels more natural to map flows with processes, rather than tasks with processes.

import multiprocessing
import sub_flow

def sub_flow_wrapper(foo):
  sub_main(foo)

@flow
def main():
  with multiprocessing.Pool() as pool:
    pool.starmap(sub_flow_wrapper, [foo1, foo2, foo3])

Hey guys, I'd really appreciate some help. I've deployed prefect server with what I believe is the simplest possible method. I just have a single Windows Server, and currently a single flow. I start the server with: prefect server start --host 0.0.0.0 And I run my python script with: python main.py It contains: if name == "__main__": my_flow.serve(name="deploymentv1") The deployment is scheduled on the server at 4am every day. Firstly, is this the simplest and most robust method? I really have found the documentation on this stuff really really lacking. The docs and the AI's get clouded with obsolete Prefect V1 content. I suspect I could put the scheduling in the serve command, but the current setup should work. This has been running fine for 2 weeks, but today it just completely failed to schedule the flow run! This is a complete show stopper if I can't rely on it to schedule. Not sure how I can debug this. I tried looking through the server console output and noticed a lot of these messages: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked [SQL: DELETE FROM flow_run_notification_queue WHERE flow_run_notification_queue.id IN (SELECT 1 FROM (SELECT 1) WHERE 1!=1)] I looked that up and its a known thing? I'm about to wrap the server start in the following batch file to see if it improves and so I can have useful logs. server.bat @echo off set PREFECT_API_SERVICES_FLOW_RUN_NOTIFICATIONS_ENABLED=false prefect server start --host 0.0.0.0 >> server.txt 2>&1 rem email me that the server stopped However I can't proceed with prefect if I can't rely on it to run my flow each day. A bad PLan B is I just use windows task scheduler to run the python flow scripts, but I would lose a lot of the reason for using prefect in the first place if I do that.

Trying my luck again, hi everyone, I am recently facing a strange problem. I have a flow (which is a subflow of another one) that runs multiple tasks. Once in a while the tasks do not even start (get stuck on pending state) and the logs of each tasks says "Crash detected! Execution was cancelled by the runtime environment." The main flow is stuck on running state and the log says: " Flow run '...' - Crash detected! Request to http://.../api/task_runs faield: PoolTimeout: . The main problem is that the tasks & subflow get stuck and even a timeout (timeout_seconds param) does not help to release the freeze... (The flow is still in running state...) I am running the flows as a k8s job on openshift and all the flows and tasks are async. Is it a known problem? Is calling an async function (as a subflow or task) could be problematic?😵‍💫

Hi, Having some problems using getting around CORS issues with Prefect V2.0 accessible to another host on the lan while running in a docker container. Details in thread per @Kevin Kho

@Marvin I can't see individual task runs of a flow anymore under Tasks Runs tab, neither can I see the flow graph.

Hey everyone! Is there an easy way to delay the execution of a task. Im sending POST calls with a task but then I need to wait 30 sec because parsing the logs for data validation. is there an easy way to do this

@Marvin I have a general logger built with structlog and integrated with datadog traces. What is the appropriate way to use my existing logger in prefect tasks and flows ?

@Marvin how does

timeout_seconds

work when a flow goes over the specified timeout limit?

@Marvin can you help me in writing a multi-agent workflow with marvin to have a web search and reading different file types for external information

@Marvin What's the deal with

ephemeral-prefect/api

?