Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Is there a task limit within a flow? I have a flow that is large and maps into a large number of individual tasks. It has stopped running the very last at all and I have no errors, it just sits there. Even running DEBUG logs, it never even attempts to start the last task.

The task in question runs in other flows without issue.

Hey <@U01HV134CKG>, are you using LocalDaskExecutor or DaskExecutor?

So I’ve seen this behavior happen with “threads” and some users have had success by moving to “processes”.

I get ye olde "RecursionError" with processes

`pickle.PicklingError: Could not pickle object as excessively deep recursion required.`

Do you immediately know which among your task inputs/outputs can’t be pickled?

Yeah, my task that maps into about 10k + runs

Flow def:
```with Flow(
    "Project Environments",
    # executor=LocalExecutor()
    executor=LocalDaskExecutor(scheduler="processes")
    # result=GCSResult(bucket="platformsh-vendor-prefect"),
) as flow:
    client = psh.get_client()
    project_ids = list_active_projects()
    environments = fetch_environments.map(project_ids, unmapped(client))
    schema = generate_schema(flatten(environments))
    object_uri = upload_gcs(flatten(environments), "project", "project_environments")
    status = gcs_to_bq(
        object_uri,
        "project_environments",
        dataset="GCP_ACCOUNTS_DATASET",
        schema=schema,
        write_disposition="WRITE_TRUNCATE",
    )```


`fetch_environments` is what doesn't pickle up

I would doubt it because I don't have that issue when the Dask scheduler is threads

ok, I moved the client and it is running... Usually there is a different error about pickling a client object

going to try an lower my dataset in the hopes that I can have a different result

Yup, task is triggered with a smaller dataset (and less tasks). Is there a known limit, whether intended or not on tasks?

There is no limit imposed by Prefect. The limits are more around the limits of your hardware. Could it be memory issues?

Nah, it doesn't seem to be getting close, and usually that ends with a `killed` message from the OS