Jai P
08/31/2022, 7:07 PM@flow
def my_subflow(num):
intense_calculation(num)
@flow
def my_parent_flow():
subflow = [my_subflow(i) for i in range(100000)]
asyncio.gather(*subflows)
so my_parent_flow
is triggered on instance_1. does this mean each subflow is executed concurrently on the same instance (instance_1)? or, does this mean each subflow executes on instance_2...instance_n as those instances are available?Nate
08/31/2022, 9:34 PM@flow
that invokes them
Here's a similar discussion where I show an example of how can we can currently orchestrate instances of flow runs from deployments on their own infra (using the client), the caveat being that there's no explicit link between parent and children (however you could prefix child flow run names with the parent name if that were helpful)
We're in the process of expanding our support for parallel subflows on their own infra, so stay tuned!Jai P
08/31/2022, 9:46 PMmy_parent_flow
kicked off in a container, that means all subflows are kicked off within the same container unless I explicitly make the network call to kick off other flow runs (which would then be picked up by whatever containers are available to pick those up in, possibly in the same k8s pod, possibly in a separate cluster, etc.?Nate
09/01/2022, 5:47 PMinstance_1
and instance_2
would each make those network calls independently according to the parameters you passed these flows and in turn, the task actually making the call (e.g. running a get_api_endpoint
task in each flow with different payloads according to the params you passed instance_1
and instance_2
)Jai P
09/01/2022, 6:18 PM@task(cache_key_fn=task_input_hash)
def some_task(input):
return f"some_task_{input}"
@flow
def my_flow(params):
some_task(params["input"])
and then i have one flow run of my_flow
that executes on instance_1
, and then another flow run of my_flow
10 minutes later that happens to execute on instance_2
with the exact same parameters, will some_task
need to actually execute? or, will the cached result from the earlier flow run be used?
basically, where do cached results of a task get stored? in memory to the instance that ran the task, or somewhere separate so that any instance can use the result of a previous run of that task?Nate
09/01/2022, 8:34 PMsome_task
on the same machine multiple times, it'd be cached after the first invocation and available to pull for subsequent calls on the same machine with the same params. However, if you tried calling some_task
on a different machine with the same params, you can't currently pull results from that previous invocation on the first machineJai P
09/01/2022, 9:02 PM