j

    jcozar

    1 year ago
    Hi, a few days ago I scheduled a flow to run daily. I use LocalDaskExecutor with 8 workers and the processes scheduler (I need to run tasks in parallel). I am using Docker as storage and ECS (fargate) as run config. The problem is that today the execution failed because some mapped tasks were always in Queued state with the following logs:
    ...
    07:17:34 INFO CloudTaskRunner Task 'XXX[12]': Starting task run...
    07:17:35 INFO CloudTaskRunner Task 'XXX[12]': Finished task run for task with final state: 'Queued'
    07:18:05 INFO CloudTaskRunner Task 'XXX[12]': Starting task run...
    07:18:05 INFO CloudTaskRunner Task 'XXX[12]': Finished task run for task with final state: 'Queued'
    ...
    It seems the tasks started but finished in Queued state again… What could be the problem here? I am using task concurrency for those tasks if that helps. Thank you!
    I was using 256 and 512 for cpu and mem in my flow. My tasks are lightweight as they are just using boto3 starting fargate tasks, but using 8 workers for local dask executor might not be enough. However I didn’t see any log from dask crashing or somethig (neither the fargate task running the prefect flow).
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @jcozar, just making sure you’re on Cloud and not Server?
    j

    jcozar

    1 year ago
    yes, I am on cloud
    And I confirm that using more cpu and mem solved the problem. I suppose that 8 workers for just 0.25vpu and 512MB ram is not enought and something chashed (but with no logs)
    Kevin Kho

    Kevin Kho

    1 year ago
    Thanks for the info!