Hi all, I'm doing multiple flow.run()'s in differe...
# ask-communityj
Jacques
04/22/2020, 7:59 PMHi all, I'm doing multiple flow.run()'s in different threads - but when I do this output caching breaks. I'm trying to debug, and I think the issue is that the prefect context isn't shared between runs in multiple threads. Docs here: https://docs.prefect.io/api/latest/utilities/context.html#context-2 state that context is thread safe, so I tried creating a context using
shared_context = prefect.utilities.context.Context()my_flow.run(context=shared_context)j
Jim Crist-Harif
04/22/2020, 8:03 PMThe context is thread local, so it's thread safe in that you can have prefect run in multiple threads, but the context isn't shared between threads.
Jim Crist-Harif
04/22/2020, 8:04 PMSo that explains the issue you're running into. I'm not sure how to handle your actual problem of cross-thread caching though.
n
nicholas
04/22/2020, 8:12 PM
@Jacques - is the issue that the cached outputs of tasks in one thread aren't available to the tasks running in the other threads?
j
Jacques
04/22/2020, 8:13 PMExactly that yes
Jacques
04/22/2020, 8:14 PMIf I'm following then its going to be cached if I call the cached task multiple times in the same flow in the same thread, however, there is no cross thread cache sharing mechanism. I imagine implementing a cross-thread caching mechanism would be tricky, but perhaps I could "cheat" a bit by doing something like reading cache out of previous run's contexts and pre-populating the context of future threads with past caches? This would miss cache if started too close, but would miss less over time.
n
nicholas
04/22/2020, 8:31 PM
Cached outputs are still stored on the context, meaning the results would still be thread-specific. Are you using Core Server or Cloud, or are you working with Core only?
nicholas
04/22/2020, 8:34 PM
I think for this use case you'd need some sort of persistence layer, which can't be achieved with Core-only. This is possible with result handlers and caching with Cloud/Server; here's an example of @Laura Lorenz (she/her) working on this in the PyData talks from a couple of weeks ago:
j
Jacques
04/22/2020, 8:37 PMI'm using core at the moment. Running inside a lambda so need to keep things light.
Jacques
04/22/2020, 8:38 PMThanks for this, going to watch now!
😄 1
2 Views