https://prefect.io
Join Slack
Newbie question: I have a flow in a deployed in a remote cluster that takes in - A JSON configuratio...
# ask-community
j
Newbie question: I have a flow in a deployed in a remote cluster that takes in • A JSON configuration that lives on my laptop • 2 S3 paths to for input and output directories. How do I pass these arguments along?
n
hey @Joe (hopefully I tagged the right Joe! 🙂 ) to make sure I understand, you have a flow like this?
Copy code
@flow
def f(some_json: dict[str, Any], source_path: str, target_path: str): ...
and you want to know how to pass params to a new run of this deployed flow?
j
That's right
n
cool! the easiest way is to use 
run_deployment
Copy code
from prefect.deployments import run_deployment

run_deployment("f/my-deployment", parameters=dict(some_json={...}, source="<s3://foo>", target="<s3://bar>"))
and then you can add on 
timeout=0
in there if you don't want to wait for that flow run to finish 
run_deployment
will return a 
flow_run
object
j
So how do I write my 
__name__ == "__main__"
section to account for the arguments? The examples I've seen all pass params from there.
n
run_deployment is just a convenient way to 
POST /create_flow_run_from_deployment
and then poll for completion of the run
well ok so just to be clear • deployments don't care about whats in the if name == main block, only about the body of your entrypoint flow • you can run flows interactively (not on remote infra) at any time (like normal python) by simply calling your flow in a if name == main block
j
Ah.
That explains my confusion.
So how does this warning in in DaskTaskRunner apply? https://docs.prefect.io/integrations/prefect-dask/index
Copy code
Because the DaskTaskRunner uses multiprocessing, calls to flows in scripts must be guarded with if __name__ == "__main__": or you will encounter warnings and errors.
n
the task runner are used by a flow regardless if that flow is run interactively or as the entrypoint of a deployment so i assume you're running that flow interactively locally and its not guarded by an if name main block?
j
I'm still figuring out all the details of the flow by running locally for now. I want to understand what I need to change so that it can be run remotely.
Currently, I'm using argparse to get it from command line and just pass them in as arguments.
n
its totally fine to use argparse or typer like that as a convenience when you're running flows locally, but deployments handle this for you so you can keep it and still use deployments
j
Great.
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by