https://prefect.io logo
j

Jay Sundaram

03/19/2021, 1:55 AM
Hi all, I'm looking for some help with StartFlowRun. Executed the following running on my local machine:
Copy code
prefect backend server
prefect server start
prefect create project etl-project
prefect agent local start --label etl-label
prefect register flow --file simple_flow.py --name a-simple-etl-flow -l etl-label -p etl-project
In the UI, I can click on QUICK RUN and observe the flow execute. Next , in another simple script named start_flow_run.py:
Copy code
from prefect.tasks.prefect.flow_run import StartFlowRun

kickoff_task = StartFlowRun(
    project_name='etl-project',
    flow_name='a-simple-etl-flow'
)
which I execute like this:
Copy code
python start_flow_run.py
But nothing happens. The agent doesn't detect it; no activity in the UI. I was expecting my registered flow named 'a-simple-etl-flow' to execute. Please advise. Thanks.
I just figured it out. I just needed to add kickoff_task.run() to the start_flow_run.py. The updated script looks like this:
Copy code
from prefect.tasks.prefect.flow_run import StartFlowRun

kickoff_task = StartFlowRun(
    project_name='etl-project',
    flow_name='a-simple-etl-flow'
)

kickoff_task.run()
Might want to update the example in the documentation to indicate that kickoff.run() is required.
Now, if my 'a-simple-etl-flow' flow is stored in S3 and the agent is running on an AMI (EC2), how should the start_flow_run.py be updated to launch/start that flow? And does the start_flow_run.py script need to be executed on the same AMI that the agent is running on?
c

Chris White

03/19/2021, 2:58 AM
Hi Jay, this task is not intended to be run this way (although it’s perfectly fine if you do want to use it like this); this task, like any Prefect task, should be added to a Flow. Check out this example: https://docs.prefect.io/core/idioms/flow-to-flow.html And in the future, could you please avoid posting large code blocks and reposting in the main channel from a thread? We try to keep the main channel tidy so that it’s easily scrollable - thank you!! 😄 🙏
j

Jay Sundaram

03/19/2021, 3:00 AM
Hi @Chris White, thank you- I'll take a look.
👍 1
Hi @Chris White, What is the proper way to execute a flow, programmatically i.e.: not from the UI and not a scheduled job. Looking for a way to execute a flow given an external event. Thank you.
c

Chris White

03/19/2021, 3:18 AM
Great question - ultimately, you’ll want to use the graphql api to achieve this; there are many ways of configuring a service / webhook to call the API that depend on your setup. Check out these docs for some examples: • https://docs.prefect.io/orchestration/concepts/flow_runs.htmlhttps://medium.com/the-prefect-blog/event-driven-workflows-with-aws-lambda-2ef9d8cc8f1a
j

Jay Sundaram

03/19/2021, 3:36 AM
Thank you for this info. I'll try the graphql api. About the medium article, questions: 1. Will this only work with Prefect Cloud and not with Prefect Server? 2. The Prefect Cloud API token should be a RUNNER token, correct?
c

Chris White

03/19/2021, 3:43 AM
1. The GraphQL route is also available in Server (please note that it is snake_cased though, and not camelCased as in the article), but you do need to ensure that where ever you call it from has network access to your Server API 2. You can call this route from either a RUNNER or a TENANT token 👍
j

Jay Sundaram

03/19/2021, 2:18 PM
Would like to try this out on server first. How do we create a RUNNER token for server?
Copy code
prefect auth create-token -n runner_token -s RUNNER                                                                 
Usage: prefect auth [OPTIONS] COMMAND [ARGS]...

Error: Auth commands with server are not currently supported.
In the UI (I believe the only other way to create RUNNER tokens), the API Token option is grayed out:
c

Chris White

03/19/2021, 4:36 PM
Tokens are only used for authentication with Cloud, there is no token concept in Server so you don’t need to worry about that step if using Server
j

Jay Sundaram

03/21/2021, 7:10 PM
Hi Chris, What is the API endpoint/URL for Server?
c

Chris White

03/21/2021, 7:12 PM
it depends on how you deployed it, but assuming you’re using the default configuration it should be
<https://localhost:8080/graphql>
j

Jay Sundaram

03/21/2021, 7:22 PM
I get the following: HTTP Error 405: Not Allowed. I tried with the following code.
c

Chris White

03/21/2021, 7:23 PM
Oh you shouldn’t have to go through all of that - in that example I didn’t have access to the Prefect library because I was using a default AWS environment; you can use the prefect library:
Copy code
from prefect import Client

c = Client() # will autodetect your configuration

c.create_flow_run(flow_id)
j

Jay Sundaram

03/21/2021, 7:30 PM
This worked like a charm. Thank you.
💯 1
Did you mention something about specifying the Version Group ID instead of the Flow ID- or am I remembering incorrectly?
c

Chris White

03/21/2021, 7:36 PM
Yup you can pass
version_group_id=XXXX
as a keyword argument instead of flow id and the backend will trigger a run for the latest version of the flow without you having to update the flow ID each time
j

Jay Sundaram

03/21/2021, 7:42 PM
Nice! Thank you.
👍 1
3 Views