Avi A
04/30/2020, 1:11 PMLocalResultHandler
Jeremiah
04/30/2020, 1:29 PM.map
operator, then at this time all data is stored in memory across children tasks. If you want to follow this issue, an enhancement to mapping is coming that will improve the handling of children tasks.Avi A
04/30/2020, 1:30 PMJeremiah
04/30/2020, 1:31 PMDaskExecutor
to scatter the work across a Dask cluster, then you won’t run into memory errors.Avi A
04/30/2020, 1:35 PMJeremiah
04/30/2020, 1:36 PMAvi A
04/30/2020, 2:21 PMflow.run
but I’m using flow.register
and then I want to run the task via the UI. Where do I specify the executor?Jeremiah
04/30/2020, 4:46 PMEnvironment
object to your flow that tells Prefect how to run the flow (and which executor to use). A simple one to start with is the RemoteEnvironment because it lets you specify any executor - here are the intro docs with some example code.Avi A
05/03/2020, 6:27 AMLocalDaskExecutor
seems to have solved the OOM issues, but the process still hangs at some point and IDK why. I ended up doing the Extraction+Load together in the same task since I wanted to add a SKIP mechanism which would’ve created an unhealthy dependence between the tasks anyway. Now I have the flow running and it’s super funJeremiah
05/03/2020, 3:14 PMworker_client
for mapping and certain use cases, which will be solved in the upcoming mapping refactor - what you’re describing sounds like some of the symptoms we’ve seen there.Avi A
05/03/2020, 3:16 PM