Hello, I currently have a deployment set up that u...
# prefect-getting-started
Hello, I currently have a deployment set up that uses GCP Compute Engine for the polling agent and GCP Cloud Run Jobs for the execution of the flows. Currently I am using Prefect to run NLP processes on large batches of documents. To do this, I am loading in a set of 1000 documents, then splitting into batches of 50 documents and using
to run a task on each batch of 50 documents. It appears that when I use
, it is running all of the tasks within a single GCP Cloud Run Job. I was wondering, is it possible to configure Prefect to spin up new Cloud Run Jobs for each of these mapped tasks? Thanks in advance for any help!
Not directly with
, but you could loop over the files and create deployments with
and then loop through the new deployments and run them with
Thanks for getting back to me! Do you have any examples of this or recipes?
I'm currently using Anna's GCP template to deploy flows. Is it possible to deploy this sort of thing in the same manner?
Here’s a simpler setup than the one I suggested above: In your flow function, take the file name as a parameter. Then create the deployment something like this:
Copy code
from flows import my_nlp_flow
from prefect.deployments import Deployment

def create_deployment():
    for file in files:
        deployment = Deployment.build_from_flow(

if __name__ == '__main__':
And then create flow runs by iterating over the files something like this:
Copy code
from prefect.deployments import run_deployment

for file_name in files:
    run_deployment(name="my_nlp_flow/my_nlp_deployment", parameters={'file_name':file_name})
Then I would think about CI/CD after you have that working. Docs references: https://docs.prefect.io/concepts/deployments/#create-a-flow-run-in-a-python-script https://docs.prefect.io/concepts/deployments/#create-a-deployment-from-a-python-object https://docs.prefect.io/api-ref/prefect/deployments/?h=run_deployment#prefect.deployments.run_deployment
Thanks, I got it working! In my orchestrator flow I'm creating a deployment then iterating through the documents and running the deployment with each one.
🙌 1
Only issue I'm facing now is that it does not appear to be running asynchronously as intended