Is it possible to trigger multiple concurrent Flow Runs for the same Flow? I’m getting this error when I attempt to do so:
Copy code
RuntimeError: The task runner is already started!
Context:
I have an application hosted on GCP Cloud Run that allows multiple requests per container. Whenever my application receives an API request with a
user_id
param, it triggers a flow run to update data for that user.
When I trigger the first Flow Run via an API request, everything works fine. But if I call the API again while the first Run is ongoing, my second API request with throw the above RuntimeError.
More context:
My application is written in Flask and served with Gunicorn. It’s deployed on GCP Cloud Run. Here’s the gunicorn config in my dockerfile
I believe I could fix this by only allowing 1 request per worker and scaling workers when I receive more requests. But this is not ideal for performance reasons (time required to start each new worker).
r
Ryan Peden
12/19/2022, 9:25 PM
I believe you could also work around this by using a separate task runner instance for each flow run. So, something like:
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.