Hi, a few days ago I scheduled a flow to run daily...
# prefect-serverj
jcozar
06/27/2021, 9:01 AMHi, a few days ago I scheduled a flow to run daily. I use LocalDaskExecutor with 8 workers and the processes scheduler (I need to run tasks in parallel). I am using Docker as storage and ECS (fargate) as run config.
The problem is that today the execution failed because some mapped tasks were always in Queued state with the following logs:
Copy code
...
07:17:34 INFO CloudTaskRunner Task 'XXX[12]': Starting task run...
07:17:35 INFO CloudTaskRunner Task 'XXX[12]': Finished task run for task with final state: 'Queued'
07:18:05 INFO CloudTaskRunner Task 'XXX[12]': Starting task run...
07:18:05 INFO CloudTaskRunner Task 'XXX[12]': Finished task run for task with final state: 'Queued'
...✅ 1
jcozar
06/27/2021, 10:44 AMI was using 256 and 512 for cpu and mem in my flow. My tasks are lightweight as they are just using boto3 starting fargate tasks, but using 8 workers for local dask executor might not be enough. However I didn’t see any log from dask crashing or somethig (neither the fargate task running the prefect flow).
k
Kevin Kho
06/27/2021, 9:15 PM
Hey @jcozar, just making sure you’re on Cloud and not Server?
j
jcozar
06/28/2021, 8:29 AMyes, I am on cloud
jcozar
06/28/2021, 8:29 AMAnd I confirm that using more cpu and mem solved the problem. I suppose that 8 workers for just 0.25vpu and 512MB ram is not enought and something chashed (but with no logs)
👍 1
k
Kevin Kho
06/28/2021, 1:36 PM
Thanks for the info!
2 Views