https://prefect.io logo
a

Austin Vecchio

02/14/2022, 4:28 PM
I know for an ECS task, it is possible to specify the amount of RAM/CPU that is available to each task. Would it be possible to set these same constraints on individual tasks?
k

Kevin Kho

02/14/2022, 4:29 PM
Like on the Prefect side right? Definitely not for now. Potentially in the future if we end up with a Ray Executor? Cuz Ray provides this functionality I think
Sorry the word task is overloaded here. I don’t know if you mean ECS task or Prefect task
a

Austin Vecchio

02/14/2022, 4:38 PM
I do apologize for the miscommunication. In this case it is a prefect task that will run on ECS. I run command line applications in the Nextflow engine that each have varying resource requirements. I was hoping there exited a way to 'limit' the resources needed for each individual prefect task such that a task that 'writes to a file' would not need the same resources that would require 100GB of RAM.
k

Kevin Kho

02/14/2022, 4:40 PM
No it’s not your fault 😅. The only way to do this at the moment is splitting up the different steps into different Flows because you can specify resources on a Flow level (ECS task settings) and then you can chain them together with a “main flow”
a

Austin Vecchio

02/14/2022, 4:42 PM
That is an interesting workaround. I will look into that. Thank you for the pointer!
3 Views