Hi - I'm using Prefect w/ Dask on kubernetes and I'm seeing unusual restarts without any clear reason. that's said this usually happens when some of dask workers are not ready or stuck in a pending state e.g. 4 workers, only 2 are active and the other 2 are pending due to CPU, memory .. etc, is there something I can do? I don't mind tasks being restarted however most of them are time expensive. I know about input/output caching but I'm thinking more of caching per task run?

task = mytask(...)
other1 = other1.map(task)
other2.map(other1)

Are you on Server or Cloud? Cuz Cloud has a feature called Version Locking for this problem where the re-run won’t fire because it sees the task completed successfully already. The problem here is Dask retrying it because it just sees some tasks that failed and tries to start the task again. For caching per mapped run, see this for the caching and this for making files per mapped task

I'm self-hosting everything. so the Server doesn't have this functionality? I'm also using

S3Result

and

minio

as s3 bucket provider, I can't rely on any external service due to project limitations.

Yeah Server doesn’t have Version Locking. You can try setting the targets though

Could you please explain more? I'm not sure wdym by "setting targets"?

This is the docs for caching using targets