@Marvin is there a reason that when my flow submits one subflow it starts immediately, when it submits 2 at the same time, it takes 60 seconds to be submitted and with 3 subflows at the same time, it takes 90 seconds

The delay you're experiencing with subflow submissions in Prefect could be related to concurrency limits or priority settings within your work pools or queues. Prefect allows you to set concurrency limits at various levels, such as work pools, work queues, or deployment levels. These limits can affect how quickly subflows are submitted and executed. Here's a brief summary of what might be happening: 1. Concurrency Limits: If there are concurrency limits set on your work queues or pools, they could be restricting the number of subflows that can be executed concurrently. This would explain why additional subflows have a delayed start. 2. Priority Settings: The priority of your subflows within the queue can also affect their submission order. If the queue is handling other high-priority tasks, your subflows might be queued until those are processed. 3. Infrastructure Constraints: The infrastructure backing your Prefect setup could also be a bottleneck. If the resources (like CPU or memory) are limited, this could delay the start of additional subflows. For more detailed information on configuring concurrency limits and understanding execution priority, you might want to explore these resources: - Prefect Work Pools and Queues - Task Run Limits If these settings are not the cause, you might be facing an issue specific to your Prefect version or setup. Checking for known issues on the Prefect GitHub issues page might provide additional insights.

@Marvin he delay is consistent and reproducible, my work pool concurrency limit is 1000, which is well over my workload

Given that your work pool concurrency limit is set to 1000 and the delay is consistent and reproducible, it suggests that the issue might not be directly related to concurrency limits. Here are a few other possibilities to consider: 1. Network Latency or API Rate Limiting: If your Prefect setup is communicating with a remote server or API, network latency or rate limiting could be causing delays in subflow submissions. 2. Scheduling Overhead: There might be an inherent scheduling overhead in the way Prefect handles multiple subflow submissions simultaneously. This could be a part of the orchestration logic or a limitation in the current version of Prefect. 3. Prefect Version-Specific Behavior: It's possible that this behavior is specific to the version of Prefect you are using. Checking the release notes for any related fixes or changes might be helpful. 4. Logging and Monitoring: Enable detailed logging to see if there are any noticeable patterns or errors occurring during the submission process. This might give insights into where the delay originates. 5. Infrastructure Bottlenecks: Even though the concurrency limit is high, other infrastructure components (like databases or external services your flows interact with) could be bottlenecked. If none of these seem to be the cause, it might be worth reaching out to the Prefect community or support for more detailed troubleshooting.

@Marvin the subflows are run as processes

Running subflows as separate processes introduces additional layers where delays might occur. Here are some considerations specific to process-based execution: 1. Process Initialization Overhead: Each subflow running as a separate process involves overhead for process creation and initialization. This might explain the incremental delay with more subflows, as the system may need more time to allocate resources and start each process. 2. Inter-process Communication (IPC): If your subflows communicate with each other or with a parent flow, delays in IPC can contribute to the overall delay. This is especially relevant if the communication channels are saturated or if there's contention. 3. System Resource Contention: Even if the concurrency limit is high, the underlying system resources (CPU, memory) might be contended when multiple processes are spawned simultaneously. This can lead to delays as the operating system schedules these processes. 4. Python Global Interpreter Lock (GIL): Although the GIL should not directly affect process-based execution in Python, any shared resources or data structures being accessed concurrently might introduce bottlenecks. 5. Prefect's Internal Scheduling: There could be internal scheduling mechanisms within Prefect that handle process-based subflows differently, possibly introducing delays when multiple are submitted simultaneously. To address these issues, consider: - Monitoring system resource usage during subflow execution to identify bottlenecks. - Reviewing Prefect's configuration and logs for any hints on process-related delays. - Investigating whether Prefect has any known issues or optimizations related to process-based execution in the version you are using. If these suggestions don't resolve the issue, consulting with the Prefect community or support might provide further insights.