Thread
#prefect-community
    r

    Robert Bastian

    1 year ago
    Hello! I’m seeing an issue where mapped tasks are limited to 2 in Prefect Cloud but are unlimited locally. Is this a standard plan limitation?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hello again @Robert Bastian! Where are you running the agent and are you using the LocalDaskExecutor/DaskExecutor?
    No it is not a standard plan limitation
    r

    Robert Bastian

    1 year ago
    Apologies Kevin, I think this is actually a race condition. I am using LocalDaskExecutor, but it appears that the task I’m using to “reduce” the mapping isn’t waiting for all the mapped tasks to complete.
    with Flow("cx1-ercot-lmp-scraper-v1", executor=LocalDaskExecutor()) as flow:
        nomad_url = Parameter("nomad_url", default="<http://nomad-ue1.dev.drillinginfo.com>")
        job_name = Parameter("job_name", default=ercot_scrapers)
        region = Parameter("region", default="vmw-ash")
        dispatched_job = dispatch_job.map(job_name=job_name, nomad_url=unmapped(nomad_url), region=unmapped(region))
        markdown = jinja_task(dispatched_jobs=dispatched_job)
        create_artifact(markdown)
    What I’ve seen work previously is that the “jinja_task” that generates uses a template to generate markdown will wait for all mapped tasks.
    OK, I think the issue is that my task isn’t thread safe. Sorry I jumped the gun on this one.
    Kevin Kho

    Kevin Kho

    1 year ago
    Ah ok. Glad you figured it out.