Hi Everyone, I have a flow that have mapped tasks that use DaskCloudProvider Environnement and i have set the number of workers to 4. For some reasons only one worker is created. Here is the code:
Hi @Mac Gréco Péralte Chéry!
I’m not as familiar with the DaskCloudProviderEnvironment, but I would double-check that your FargateCluster has adequate resources to spawn multiple workers
m
Mac Gréco Péralte Chéry
11/13/2020, 12:56 PM
Hello @Dylan, to be more precise, when i run my flow i see the fargate prefect task created into my AWS console but it seems that my flow is executed in this task but as far as i remember this first task should instanciate another fargate task that will dynamicaly create a dask cluster.
It used to be like that before i updated to the latest version of prefect (0.12.1 ->0.13.15)
Mac Gréco Péralte Chéry
11/13/2020, 12:57 PM
I am using an prefect cloud and and Ec2 instance to run my agents
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.