Charles Lariviere
12/01/2020, 3:41 PMflow.run()
(instead of flow.register()
) and running with a local-agent, everything works great. Though, when I run the following script to register it in our Prefect Cloud account and execute it through the UI, it works (as in, the query gets executed) but I get the following error: Unexpected error: TypeError("can't pickle _thread.lock objects")
.
My question is; what are best practices for structuring these scripts? Once you’ve built the flow, should you always end it with flow.run()
and flow.register()
? Do you manually register flows when making changes but not include the register()
method inside the actual script?import prefect
from prefect import task, Flow
from prefect.tasks.snowflake import SnowflakeQuery
query = """
SHOW DATABASES;
"""
snowflake = SnowflakeQuery(
{account info},
query=query
)
flow = Flow("hello-snowflake", tasks=[snowflake])
flow.register(project_name="analytics")
Jim Crist-Harif
12/01/2020, 3:55 PMResults
enabled). We can fix that.Marvin
12/01/2020, 3:55 PMJim Crist-Harif
12/01/2020, 3:59 PMMy question is; what are best practices for structuring these scripts?
Once you’ve built the flow, should you always end it withYou have full flexibility here. Generally I recommend when possible users write their whole flow in a single file (makes it easier to ship around via prefect's storage or dask). If running locally or debugging, I generally either import the flow and callandflow.run()
? Do you manually register flows when making changes but not include theflow.register()
method inside the actual script?register()
flow.run
or end the script with flow.run()
.
When registering, I generally call flow.register
in the script itself, but you can also use prefect register flow -f yourflow.py -p your-project
to register using the CLI (then no flow.register
call is needed in the file itself).Charles Lariviere
12/01/2020, 4:00 PMflow.register()
flow.run()
and run the flow on a daily schedule — would it then get registered (and increment the version number in Prefect Cloud) every time it is run?Jim Crist-Harif
12/01/2020, 4:05 PMflow.run
does a local run only. Runs this way never communicate with a backend (prefect cloud/server), they're local only. This does not schedule a flow run with a backend.
• flow.register
registers a flow with a backend. Each call will bump the version number.
If you're running a flow with a Schedule
:
• flow.run
will start a long running process that will wait in between runs in accordance with the schedule
• flow.register
will register the flow with the backend (cloud/server). The backend will then schedule flow runs in accordance with that schedule, which are eventually picked up by the agent.
During a backend-executed (using cloud/server) flow run, it doesn't matter what the originating "script" ended with, all calls to `flow.run`/`flow.register` in the script are ignored.Charles Lariviere
12/01/2020, 5:29 PM