Channels
#prefect-community
Title
a
Alberto de Santos
10/14/2020, 8:50 AMHi guys,
Can anyone give me a piece of advice on how to run a flow from python without manually inserting the flow_id?
j
Jenny
10/14/2020, 11:23 AMHi @Alberto de Santos where are you wanting to run your flow? If it's just on your local machine you could simply use flow.run()
a
Alberto de Santos
10/14/2020, 11:39 AMOn cloud, standard
I have tried all options but it is not clear to me how to make that automatically
j
Jenny
10/14/2020, 12:43 PMWithout knowing your exact use case it's a bit hard to advise but as background, you need a flow ID so that your agent knows which flow it is running. If you just want to check your flow locally, you can use flow.run() (as I mentioned above). If you want a flow to run in Cloud after you register it, you could use client.create_flow_run and use the flow id that is returned when you register a flow.
As a simple example:
Copy code
from prefect import Flow, task
from prefect.client.client import Client
@task()
def hello():
print('hello')
with Flow(name='simple-flow') as flow:
hello()
# flow.run()
id = flow.register('PROJECT_NAME')
client = Client()
client.create_flow_run(flow_id=id)a
Alberto de Santos
10/14/2020, 12:51 PMLovely
However, this is assuming I have an agent running, right?
j
Jenny
10/14/2020, 12:52 PMYes - you'll always need an agent running to run flows in Cloud.
a
Alberto de Santos
10/14/2020, 12:53 PMHow does the agent know, it should run this flow?
Should I tell the flow run which agent?
j
Jenny
10/14/2020, 1:01 PMThe agent will poll the api to see if there are flow runs waiting to be run. You can customize which agent runs which flow run using labels - there's more about it in the agents section of the docs:
https://docs.prefect.io/orchestration/agents/overview.html
j
James Phoenix
10/14/2020, 2:15 PMHey @Jenny
What are the best ways to utilise an agent? I.e. will I need to have it polling on a VM or?
What are the other deployment methods to host the polling agent?
j
Jenny
10/14/2020, 2:19 PMHi @James Phoenix - You can use a local agent or Docker, Kubernetes or Fargate - the agents section in the docs have more info to answer those questions.
😍 1
j
James Phoenix
10/14/2020, 2:20 PMMy question is doesn't the agent have to be consistently turned on?
I.e. it has to be running as a background process listening for tasks from the cloud api?
Because for example Fargate is ephemeral, so I wondered about that?
j
Jenny
10/14/2020, 2:35 PMYep - technically they are querying the API for task runs and flow runs and you'll need them on whenever you want to run a flow run from the API. Fargate Agents aren't generally installed to run on Fargate itself - they can be spun up anywhere with the correct variables set.
👍 1
✅ 1
j
James Phoenix
10/14/2020, 2:39 PMCool thanks @Jenny I'll have a review 🙂
👍 1
a
Alberto de Santos
10/14/2020, 4:28 PMAnd another question, coming back to your example @Jenny
Copy code
from prefect import Flow, task
from prefect.client.client import Client
@task()
def hello():
print('hello')
with Flow(name='simple-flow') as flow:
hello()
# flow.run()
id = flow.register('PROJECT_NAME')
client = Client()
client.create_flow_run(flow_id=id)j
Jenny
10/14/2020, 5:15 PMHi @Alberto de Santos - if you need to you can query for a flow id using the client/ API: https://docs.prefect.io/orchestration/concepts/api.html#authenticating-the-client-with-cloud