Hi ! I have a flow which triggers multiple subflow...
# ask-community
r
Hi ! I have a flow which triggers multiple subflows. I want to run them on multiple workers since they don't depend on each other. But I want to retrieve the return statement of every subflow in my main flow which is a basic string. Here's my current infra : • prefect api server within a docker container • postgres db within a docker container • prefect-worker within a docker container (process work pool) I have one process work pool with a deployment (which is the subflow I wanna run) I run my main flow locally on my computer (for dev purposes). It runs correctly the subflows within the prefect-worker container but when I use
await result.state.result(_fetch_=True, _retry_result_failure_=True, _raise_on_failure_=False)
, my main flow run into
ValueError: Path /root/.prefect/storage/f2ffbc1d077445acac57a0be51101714 does not exist.
Here are my questions : • Where is located that storage ? in the prefect-worker container ? • Can I avoid persistent storage ? • Can I avoid setting up an S3 storage only for storing a simple string ? Thank you.
a
Hey @Romain Lefebvre! Yes, that storage is located in the prefect-worker containers. Flows will persist their results to the local filesystem when result persistence is needed, but no remote storage is provided. If you want to retrieve the result from these subflows you'll need to use persistent storage since the subflows are running in a different memory space. You could use a volume mount in your prefect-worker container so that your main flow and subflows share a common filesystem.