@Marvin Is it possible to access the list of concurrent prefect tasks to see how many tasks are submitted?

Yes, it is possible to access the list of concurrent Prefect tasks. Prefect provides a

ConcurrentTaskRunner

that can run tasks concurrently, allowing tasks to switch when blocking on IO. Tasks will be submitted to a thread pool maintained by anyio. A task in Prefect is a function that represents a discrete unit of work in a Prefect workflow. They can take inputs, perform work, and return an output. For tasks requiring parallel execution, Prefect provides

DaskTaskRunner

and

RayTaskRunner

. For more detailed information, you can refer to the Prefect documentation on Task Runners and Tasks.

@Marvin how do I count how many tasks have been submitted

To count the number of tasks that have been submitted in Prefect, you can use the Prefect API or UI to get information about your flow runs. Each flow run will have a list of task runs, and you can count the number of these task runs to get the number of tasks that have been submitted. Here is a simple example of how you might do this using the Prefect API:

from prefect import Client

# Instantiate the client
client = Client()

# Get all flow runs
flow_runs = client.get_flow_runs()

# Initialize a counter
task_count = 0

# Loop through each flow run
for flow_run in flow_runs:
    # Get the task runs for the flow run
    task_runs = client.get_task_runs(flow_run_id=flow_run.id)
    # Add the number of task runs to the counter
    task_count += len(task_runs)

print(f"The number of tasks that have been submitted is {task_count}")

This will print out the total number of tasks that have been submitted across all flow runs. Please note that you may need to adjust this code depending on your specific use case, such as if you only want to count tasks from certain flow runs or tasks with a certain status. Please refer to the Prefect API documentation for more details.