https://prefect.io logo
a

Alberto de Santos

10/14/2020, 8:50 AM
Hi guys, Can anyone give me a piece of advice on how to run a flow from python without manually inserting the flow_id?
j

Jenny

10/14/2020, 11:23 AM
Hi @Alberto de Santos where are you wanting to run your flow? If it's just on your local machine you could simply use flow.run()
a

Alberto de Santos

10/14/2020, 11:39 AM
On cloud, standard
I have tried all options but it is not clear to me how to make that automatically
j

Jenny

10/14/2020, 12:43 PM
Without knowing your exact use case it's a bit hard to advise but as background, you need a flow ID so that your agent knows which flow it is running. If you just want to check your flow locally, you can use flow.run() (as I mentioned above). If you want a flow to run in Cloud after you register it, you could use client.create_flow_run and use the flow id that is returned when you register a flow.
As a simple example:
Copy code
from prefect import Flow, task
from prefect.client.client import Client

@task()
def hello():
    print('hello')

with Flow(name='simple-flow') as flow:
    hello()    

# flow.run()
id = flow.register('PROJECT_NAME')
client = Client()
client.create_flow_run(flow_id=id)
a

Alberto de Santos

10/14/2020, 12:51 PM
Lovely
However, this is assuming I have an agent running, right?
j

Jenny

10/14/2020, 12:52 PM
Yes - you'll always need an agent running to run flows in Cloud.
a

Alberto de Santos

10/14/2020, 12:53 PM
How does the agent know, it should run this flow?
Should I tell the flow run which agent?
j

Jenny

10/14/2020, 1:01 PM
The agent will poll the api to see if there are flow runs waiting to be run. You can customize which agent runs which flow run using labels - there's more about it in the agents section of the docs: https://docs.prefect.io/orchestration/agents/overview.html
j

James Phoenix

10/14/2020, 2:15 PM
Hey @Jenny What are the best ways to utilise an agent? I.e. will I need to have it polling on a VM or? What are the other deployment methods to host the polling agent?
j

Jenny

10/14/2020, 2:19 PM
Hi @James Phoenix - You can use a local agent or Docker, Kubernetes or Fargate - the agents section in the docs have more info to answer those questions.
😍 1
j

James Phoenix

10/14/2020, 2:20 PM
My question is doesn't the agent have to be consistently turned on?
I.e. it has to be running as a background process listening for tasks from the cloud api?
Because for example Fargate is ephemeral, so I wondered about that?
j

Jenny

10/14/2020, 2:35 PM
Yep - technically they are querying the API for task runs and flow runs and you'll need them on whenever you want to run a flow run from the API. Fargate Agents aren't generally installed to run on Fargate itself - they can be spun up anywhere with the correct variables set.
👍 1
1
j

James Phoenix

10/14/2020, 2:39 PM
Cool thanks @Jenny I'll have a review 🙂
👍 1
a

Alberto de Santos

10/14/2020, 4:28 PM
And another question, coming back to your example @Jenny
Copy code
from prefect import Flow, task
from prefect.client.client import Client

@task()
def hello():
    print('hello')

with Flow(name='simple-flow') as flow:
    hello()    

# flow.run()
id = flow.register('PROJECT_NAME')
client = Client()
client.create_flow_run(flow_id=id)
Imagine that I register the flow in a script different from the one I execute the client, how could I then know the id flow? Is there here any best practice?
j

Jenny

10/14/2020, 5:15 PM
Hi @Alberto de Santos - if you need to you can query for a flow id using the client/ API: https://docs.prefect.io/orchestration/concepts/api.html#authenticating-the-client-with-cloud