Thet Naing
12/31/2022, 6:57 PM.map
to run a task on each batch of 50 documents. It appears that when I use .map
, it is running all of the tasks within a single GCP Cloud Run Job. I was wondering, is it possible to configure Prefect to spin up new Cloud Run Jobs for each of these mapped tasks? Thanks in advance for any help!Jeff Hale
12/31/2022, 10:00 PMmap
, but you could loop over the files and create deployments with build_from_flow
and then loop through the new deployments and run them with run_deployment
Thet Naing
12/31/2022, 10:36 PMJeff Hale
01/02/2023, 2:38 AMfrom flows import my_nlp_flow
from prefect.deployments import Deployment
def create_deployment():
for file in files:
deployment = Deployment.build_from_flow(
flow=my_nlp_flow,
name="my_nlp_deployment"
)
if __name__ == '__main__':
deployment.apply()
And then create flow runs by iterating over the files something like this:
from prefect.deployments import run_deployment
for file_name in files:
run_deployment(name="my_nlp_flow/my_nlp_deployment", parameters={'file_name':file_name})
Then I would think about CI/CD after you have that working.
Docs references:
https://docs.prefect.io/concepts/deployments/#create-a-flow-run-in-a-python-script
https://docs.prefect.io/concepts/deployments/#create-a-deployment-from-a-python-object
https://docs.prefect.io/api-ref/prefect/deployments/?h=run_deployment#prefect.deployments.run_deploymentThet Naing
01/02/2023, 9:07 PM