https://prefect.io logo
m

Michał Augoff

10/19/2022, 7:43 PM
Hi all, what is the recommended way to deploy flows that come pre-built in 3rd party packages? Think e.g.
scikit-learn
providing a flow to train a machine learning model and you can deploy it with your own model class like
Copy code
Deployment.build_from_flow(flow=sklearn.flows.training, …, parameters={"model": SomePythonClass})
Is that even feasible? I guess another way to look at it is “prefect collection but with flows, not tasks”
1
a

Andrew Huang

10/19/2022, 7:52 PM
Collections can have flows in them too like trigger_dbt_cloud_job_run_and_wait_for_completion (or any other integration, like DaskTaskRunner).
m

Michał Augoff

10/19/2022, 7:55 PM
yeah I saw that but my understanding is that they act like tasks anyway, that is you still need to create your own main flow that encapsulates them. Or do you have an example where they get deployed directly?
a

Andrew Huang

10/19/2022, 7:57 PM
m

Michał Augoff

10/19/2022, 7:58 PM
yes but I’m talking about deployment, not just one-off runs
a

Andrew Huang

10/19/2022, 8:02 PM
sorry I don’t quite understand. although it’s in a collection, it’s still a flow, and that means you can deploy it like any other flow, once you import it
m

Michał Augoff

10/19/2022, 8:07 PM
when I do this
Copy code
from prefect.deployments import Deployment
from <http://prefect_dbt.cloud.jobs|prefect_dbt.cloud.jobs> import trigger_dbt_cloud_job_run_and_wait_for_completion

Deployment.build_from_flow(
    flow=trigger_dbt_cloud_job_run_and_wait_for_completion,
    name="test-deployment"
)
I get this error
Copy code
ValueError: '/Users/michal.augoff/anaconda3/envs/prefect2-demo/lib/python3.8/site-packages/prefect_dbt/cloud/jobs.py' does not start with '/Users/michal.augoff/PycharmProjects/prefect-demo/test-deployment'
👀 1
a

Andrew Huang

10/19/2022, 8:14 PM
test.py:
Copy code
from <http://prefect_dbt.cloud.jobs|prefect_dbt.cloud.jobs> import trigger_dbt_cloud_job_run_and_wait_for_completion
and then cli:
Copy code
prefect deployment build test.py:trigger_dbt_cloud_job_run_and_wait_for_completion --name my_deployment
does this work for you?
m

Michał Augoff

10/19/2022, 9:01 PM
yes it does. So I guess the python equivalent would be explicitly specifying
Copy code
entrypoint=f"{Path(__file__).name}:trigger_dbt_cloud_job_run_and_wait_for_completion"
for
Deployment.build_from_flow
👀 1
8 Views