https://prefect.io logo
Title
t

Thet Naing

12/31/2022, 6:57 PM
Hello, I currently have a deployment set up that uses GCP Compute Engine for the polling agent and GCP Cloud Run Jobs for the execution of the flows. Currently I am using Prefect to run NLP processes on large batches of documents. To do this, I am loading in a set of 1000 documents, then splitting into batches of 50 documents and using
.map
to run a task on each batch of 50 documents. It appears that when I use
.map
, it is running all of the tasks within a single GCP Cloud Run Job. I was wondering, is it possible to configure Prefect to spin up new Cloud Run Jobs for each of these mapped tasks? Thanks in advance for any help!
j

Jeff Hale

12/31/2022, 10:00 PM
Not directly with
map
, but you could loop over the files and create deployments with
build_from_flow
and then loop through the new deployments and run them with
run_deployment
t

Thet Naing

12/31/2022, 10:36 PM
Thanks for getting back to me! Do you have any examples of this or recipes?
I'm currently using Anna's GCP template to deploy flows. Is it possible to deploy this sort of thing in the same manner?
j

Jeff Hale

01/02/2023, 2:38 AM
Here’s a simpler setup than the one I suggested above: In your flow function, take the file name as a parameter. Then create the deployment something like this:
from flows import my_nlp_flow
from prefect.deployments import Deployment

def create_deployment():
    for file in files:
        deployment = Deployment.build_from_flow(
            flow=my_nlp_flow,
            name="my_nlp_deployment"  
        )

if __name__ == '__main__':
    deployment.apply()
And then create flow runs by iterating over the files something like this:
from prefect.deployments import run_deployment

for file_name in files:
    run_deployment(name="my_nlp_flow/my_nlp_deployment", parameters={'file_name':file_name})
Then I would think about CI/CD after you have that working. Docs references: https://docs.prefect.io/concepts/deployments/#create-a-flow-run-in-a-python-script https://docs.prefect.io/concepts/deployments/#create-a-deployment-from-a-python-object https://docs.prefect.io/api-ref/prefect/deployments/?h=run_deployment#prefect.deployments.run_deployment
t

Thet Naing

01/02/2023, 9:07 PM
Thanks, I got it working! In my orchestrator flow I'm creating a deployment then iterating through the documents and running the deployment with each one.
🙌 1
Only issue I'm facing now is that it does not appear to be running asynchronously as intended