Joao Moniz
06/21/2023, 2:32 PMprefect.deployments.run_deployment
from a flow, it shows the new flows that are being triggered by this function as subflows in the UI?
Shouldn't they be regular
flows instead of subflows
? That's what I understood from the docs.Joao Moniz
06/21/2023, 2:32 PMJoao Moniz
06/21/2023, 2:35 PMNate
06/21/2023, 5:24 PMrun_deployment
is effectively calling a flow (just it could be on a different infrastructure / defined in a different storage location and is associated with an existing deployment)Nate
06/21/2023, 5:25 PMJoao Moniz
06/22/2023, 5:33 AMasyncio.gather
on EKS. In order to not overload the infra, we were limiting the number of tasks running concurrently by using Task Run Concurrency tags, but at some point the tasks being executed just stop and they get deadlock.
We thought it could be a lack-of-resource issue situation, but even after we increased the memory/cpu, they got deadlock without being constrained by available resources.
From the docs we've found this piece of info below, so that's why we were asking about subflows.Joao Moniz
06/22/2023, 10:02 AMasyncio.gather
, we went with prefect concurrent method (using task.submit
)
• Using S3 to persist task results
But somehow they keep getting stucked 😕