for our flow, but to be able to limit the parallelisation (due to each task requiring a lot of resources when being run) to - for example - only two tasks at a time, is that possible? I read the docs but still don’t fully understand if we have to use a
for this, and if so - would it by default run locally (if we don’t give it any other config)? and kind of tangenial (and not directly related to prefect) but is there some advantage of something like the AWS fargate cluster for dask of a k8s dask cluster? is the former just easier to set up or something?
However this information is quite limited. Is there a way to get more information which of the parameters failed to pass the pydantic check?
09:47:51.189 | INFO | prefect.engine - Flow run 'shiny-falcon' received invalid parameters and is marked as failed
but got around that by increasing the memory requested from k8s. Now I am stuck on having the ZombieKiller stop tasks that should be reporting back status since they continue to produce logs with the message
I have attempted to set the
No heartbeat detected from the remote task; marking the run as failed.
file as well as in the
environment variables but no matter the combination I setup I still run into heartbeat errors killing a process. I am curious if there is a way to send a heartbeat from within the code base or if there is another approach I should take to get around my time out issue. Cheers!