Does prefect support micro-batching? In more detail: We’re trying to migrate our existing generator pattern to prefect - and we’re hoping we can change to a micro-batching model. Basically, our first node in our pipeline is responsible for pulling data from a location and pushing it out to the rest of the pipeline (a DAG). We were hoping to use the LOOP construct to have that “source node” pull data in

batch_size

increments, and map the individual data packets across the remaining DAG. In a way this kind of seems like a “workflow loop” with the parameters for the first node constantly updating.

Hi @Christopher Harris! I want to make sure that I understand your question. Are you hoping that your first task loops forever and farms out work to the downstream tasks?

Hey Dylan! Mostly - yes. In reality the first task would loop forever until pulling from the source yields nothing - then it would terminate.

Ahh okay cool