Christopher Harris
05/14/2020, 4:26 PMbatch_size
increments, and map the individual data packets across the remaining DAG.
In a way this kind of seems like a “workflow loop” with the parameters for the first node constantly updating.Dylan
Christopher Harris
05/14/2020, 4:54 PMDylan
Ingest Flow
and Process Flow
.
Ingest flow has two tasks: pull and persist
and create process flow run
. pull and persist
would pull the data and write it to cloud storage (GCS or S3) and returns a reference to the bucket. create process flow run
would then talk to the Prefect Server/Cloud graphql api to kick off a run of Process Flow
with the storage reference uri as a parameter.