https://prefect.io logo
Title
a

Ahmed Ezzat

01/16/2022, 4:34 AM
Hi - I'm using Prefect w/ Dask on kubernetes and I'm seeing unusual restarts without any clear reason. that's said this usually happens when some of dask workers are not ready or stuck in a pending state e.g. 4 workers, only 2 are active and the other 2 are pending due to CPU, memory .. etc, is there something I can do? I don't mind tasks being restarted however most of them are time expensive. I know about input/output caching but I'm thinking more of caching per task run?
task = mytask(...)
other1 = other1.map(task)
other2.map(other1)
k

Kevin Kho

01/16/2022, 5:18 AM
Are you on Server or Cloud? Cuz Cloud has a feature called Version Locking for this problem where the re-run won’t fire because it sees the task completed successfully already. The problem here is Dask retrying it because it just sees some tasks that failed and tries to start the task again. For caching per mapped run, see this for the caching and this for making files per mapped task
a

Ahmed Ezzat

01/16/2022, 11:22 AM
I'm self-hosting everything. so the Server doesn't have this functionality? I'm also using
S3Result
and
minio
as s3 bucket provider, I can't rely on any external service due to project limitations.
k

Kevin Kho

01/16/2022, 9:36 PM
Yeah Server doesn’t have Version Locking. You can try setting the targets though
a

Ahmed Ezzat

01/18/2022, 8:30 PM
Could you please explain more? I'm not sure wdym by "setting targets"?
k

Kevin Kho

01/18/2022, 8:34 PM
This is the docs for caching using targets
I think it should clear it up for you