Milos Tomic04/02/2020, 8:24 AM
operator to load chunks of your data in parallel. Third, you can use our
signal to work with paginated data. For reference, we use a combination of all three for our internal data warehouse (which we hope to open-source soon) - we map over tables and loop over a paginated query inside each mapped task. 2. Yes, Prefect automatically gains support for parallelism through threads or processes via Dask (which can be run locally, in addition to as a remote cluster). Check out the LocalDaskExecutor. Given the choice, we recommend a local Dask cluster over a LocalDaskExecutor because it has better guarantees about shared memory during parallel execution 3. Any infrastructure or process you’re looking for in particular? Hope you have an easy time getting going!