https://prefect.io
Join Slack
Is there a task limit within a flow? I have a flow that is large and maps into a large number of ind...
# ask-community
t
Is there a task limit within a flow? I have a flow that is large and maps into a large number of individual tasks. It has stopped running the very last at all and I have no errors, it just sits there. Even running DEBUG logs, it never even attempts to start the last task.
The task in question runs in other flows without issue.
k
Hey @Tim Enders, are you using LocalDaskExecutor or DaskExecutor?
t
LocalDask
k
So I’ve seen this behavior happen with “threads” and some users have had success by moving to “processes”.
t
ok
I get ye olde "RecursionError" with processes
pickle.PicklingError: Could not pickle object as excessively deep recursion required.
k
Do you immediately know which among your task inputs/outputs can’t be pickled?
t
Yeah, my task that maps into about 10k + runs
Flow def:
Copy code
with Flow(
    "Project Environments",
    # executor=LocalExecutor()
    executor=LocalDaskExecutor(scheduler="processes")
    # result=GCSResult(bucket="platformsh-vendor-prefect"),
) as flow:
    client = psh.get_client()
    project_ids = list_active_projects()
    environments = fetch_environments.map(project_ids, unmapped(client))
    schema = generate_schema(flatten(environments))
    object_uri = upload_gcs(flatten(environments), "project", "project_environments")
    status = gcs_to_bq(
        object_uri,
        "project_environments",
        dataset="GCP_ACCOUNTS_DATASET",
        schema=schema,
        write_disposition="WRITE_TRUNCATE",
    )
fetch_environments
is what doesn't pickle up
k
Is it because of the 
unmapped(client)
?
t
I would doubt it because I don't have that issue when the Dask scheduler is threads
ok, I moved the client and it is running... Usually there is a different error about pickling a client object
k
Running as a process?
t
yeah
Welp, it still hangs on that last task
going to try an lower my dataset in the hopes that I can have a different result
Yup, task is triggered with a smaller dataset (and less tasks). Is there a known limit, whether intended or not on tasks?
k
There is no limit imposed by Prefect. The limits are more around the limits of your hardware. Could it be memory issues?
t
Nah, it doesn't seem to be getting close, and usually that ends with a 
killed
message from the OS
k
Will ask the team for any ideas
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by