Hello, I currently have a deployment set up that uses GCP Compute Engine for the polling agent and GCP Cloud Run Jobs for the execution of the flows. Currently I am using Prefect to run NLP processes on large batches of documents. To do this, I am loading in a set of 1000 documents, then splitting into batches of 50 documents and using

.map

to run a task on each batch of 50 documents. It appears that when I use

.map

, it is running all of the tasks within a single GCP Cloud Run Job. I was wondering, is it possible to configure Prefect to spin up new Cloud Run Jobs for each of these mapped tasks? Thanks in advance for any help!

Not directly with

map

, but you could loop over the files and create deployments with

build_from_flow

and then loop through the new deployments and run them with

run_deployment

Thanks for getting back to me! Do you have any examples of this or recipes?

Here’s a simpler setup than the one I suggested above: In your flow function, take the file name as a parameter. Then create the deployment something like this:

from flows import my_nlp_flow
from prefect.deployments import Deployment

def create_deployment():
    for file in files:
        deployment = Deployment.build_from_flow(
            flow=my_nlp_flow,
            name="my_nlp_deployment"  
        )

if __name__ == '__main__':
    deployment.apply()

And then create flow runs by iterating over the files something like this:

from prefect.deployments import run_deployment

for file_name in files:
    run_deployment(name="my_nlp_flow/my_nlp_deployment", parameters={'file_name':file_name})

Then I would think about CI/CD after you have that working. Docs references: https://docs.prefect.io/concepts/deployments/#create-a-flow-run-in-a-python-script https://docs.prefect.io/concepts/deployments/#create-a-deployment-from-a-python-object https://docs.prefect.io/api-ref/prefect/deployments/?h=run_deployment#prefect.deployments.run_deployment

Thanks, I got it working! In my orchestrator flow I'm creating a deployment then iterating through the documents and running the deployment with each one.