Another question I have is whether it's possible to have a worker essentially drop a deployment run to allow another worker to pick it up. i.e. Worker A picks up a deployment in the queue, it then realises that something about it's infrastructure is balked, e.g. no internet. Instead of failing and retrying on the same worker a bunch of times unnecessarily, I want the worker to drop the flow and put it back to the queue. a la SQS message visibility timeouts.
k
KG
06/25/2024, 2:06 PM
You can use Automations for this. e.g. when flow state enters Crashed, change state to Scheduled
if you have 2 distinct infrastructure methods you could instead trigger a different deployment, one that is the same flow code but deployed with a different infra block
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.