https://prefect.io logo
Title
j

James Zhang

01/06/2023, 12:54 PM
hey guys, I’m trying to start multiple deployment runs in a Flow using
prefect.deployments.run_deployment()
, is it possible to start these deployment runs concurrently?
1
b

Ben Muller

01/06/2023, 7:38 PM
Just set timeout=0
1
🙌 2
b

Bianca Hoch

01/06/2023, 7:51 PM
Dropping docs here, just in case.
j

James Zhang

01/07/2023, 1:27 PM
It does the trick, thanks! the only thing is, because it returns the function immediately, there’s no way to know the result of the deployment run?
j

James Zhang

01/12/2023, 9:43 AM
I ended up using this pattern with ConcurrentTaskRunner:
@flow
def main_flow():
    for x in y:
      start_single_deployment.submit()
      ....

@task
def start_single_deployment():
    run_deployment(...)
in this way i can still get the states from the
run_deployment()
in subflows
b

Ben Muller

01/12/2023, 9:49 AM
Does it work? I remember trying it and running into some issues with async and not being able to return the state?
It was a while ago so details are fuzzy
Eg if the deployment fails the parent flow won't know about it
j

James Zhang

01/12/2023, 9:54 AM
i’m on version 2.7.3 it works as expected… the deployment runs are somehow treated as subflows. so if a deployment run fails, hence the subflow failes, hence the mainflow is informed. only catch is, if the pod where deployment runs on somehow killed by AutoScaler, the mainflow wouldn’t know the subflow hangs… heck, even the subflow wouldn’t know the pod is killed…
🙌 1
b

Ben Muller

01/12/2023, 9:55 AM
Nice. Sounds like it's perfect. I'm surprised because I thought you're not returning anything from the task, so the parent flow wouldn't realise the state of the run deployment call.
j

James Zhang

01/12/2023, 9:57 AM
the
run_deployment()
returns a Future, so I guess the task runner in mainflow will watch that
b

Ben Muller

01/12/2023, 10:00 AM
Ah, in your example code you don't have a return.
j

James Zhang

01/12/2023, 10:06 AM
didn’t look into the source code, i’m just guessing according to the doc: https://docs.prefect.io/api-ref/prefect/deployments/#prefect.deployments.run_deployment 😛