Question. If a task were to fail due to insufficient memory/cpu requirements, would it be possible to restart the task or workflow and request more resources?

Kind of. If you use

create_flow_run

, you can pass in a RunConfig and the RunConfig can be bumped up if you are using something like k8s. So you need an outer flow to manage that and read the error. Not 100% sure it’ll work though. The easier approach is you read something like a file size and then stick it in the RunConfig

Okay. To elaborate, I could be processing a series of tasks that can vary in the required amount of resources and I would prefer not to request resources that will not be utilized. This is a feature that I am utilizing from the Nextflow manager and AWS Batch. Although I am not too familiar with K8s or some other cloud computing systems, would it be alright if I look into adding the functionality to Prefect?

Ohhh you said task task I thought you meant Flow. So you want a retry on AWS Batch but upping the resources right?

Apologies on semantics. I know that the resources (cpu/memory) is controlled for a Prefect workflow through the run config. I was just wondering if it would be possible, regardless of the cloud environment (ECS/K8s/Vertext) if I could restart a workflow if a program fails for certain tasks and assign additional memory/cpu.

In a call but will respond in a bit

Thanks for the update. And it is fine, I am only evaluating Prefect and seeing what capabilities exist.

Yes you can up the Batch resources on retries (not directly because task inputs can’t be changed) but we can do stuff like keeping the last attempted values in the KV Store. This is on the task level (not for Flow)

This all makes sense and this will not be a challenging way to manipulate the resources. I hope to be able to try it out soon.

If independent and there are no dependencies, yes the other ones will still fire and then you can click the restart button yep

Understood. Thank you for all of the information!