Avi A04/30/2020, 1:11 PM
operator, then at this time all data is stored in memory across children tasks. If you want to follow this issue, an enhancement to mapping is coming that will improve the handling of children tasks.
Avi A04/30/2020, 1:30 PM
to scatter the work across a Dask cluster, then you won’t run into memory errors.
Avi A04/30/2020, 1:35 PM
Avi A04/30/2020, 2:21 PM
but I’m using
and then I want to run the task via the UI. Where do I specify the executor?
object to your flow that tells Prefect how to run the flow (and which executor to use). A simple one to start with is the RemoteEnvironment because it lets you specify any executor - here are the intro docs with some example code.
Avi A05/03/2020, 6:27 AM
seems to have solved the OOM issues, but the process still hangs at some point and IDK why. I ended up doing the Extraction+Load together in the same task since I wanted to add a SKIP mechanism which would’ve created an unhealthy dependence between the tasks anyway. Now I have the flow running and it’s super fun
for mapping and certain use cases, which will be solved in the upcoming mapping refactor - what you’re describing sounds like some of the symptoms we’ve seen there.
Avi A05/03/2020, 3:16 PM