Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

image.png

Hi, I am getting large gaps in execution when running with the RayTaskRunner. There are around 500+ tasks in this flow. It starts out great but then hits hickups? Anyone know how I can fix this or at least look into what the actual problem is. I have limited the number of cpus on the ray cluster in order to avoid ray's resources stomping on prefect's (I am running both locally). - Note the end tasks that aren't running in parallel.

if I call `wait` for each list of tasks then the gaps are more noticeable. Is Prefect copying data here or is `ray.get` being called?

Do you have task concurrency limits? I noticed when we had those set that prefect only checked every X seconds if there were slots available to send out new tasks

I don’t believe so. How do I set/check?

concurrency tab - which will hopefully be empty for you. If its not then, then Ive got no clue, sorry mate