Steph Clacksman
02/22/2022, 10:31 AMwith Flow("my-flow) as flow:
# do some things here
# EITHER flow.register("project_name")
# OR flow.run(executor=DaskExecutor())
I want to run my flow through prefect cloud, so I understand that I need to register my flow , but when I want to run it it looks like I need to edit the code, which seems a bit weird when it's dockerised and running on a remote server.
Is it normal to register the flow like above before building the docker image, then switch it to the run command before building? Or is there something I'm missing?Anna Geller
prefect register --project YOUR_PROJECT_NAME -p /path/to/your/flow.py
When you provide the exact path to your flow, it should register more than 0 flows 🙂
And then, if you want to run the flow locally, you can run it by providing a path:
prefect run -p yourflow.py
and if you want to trigger a flow run on a backend (running on agent), you can run:
prefect run --name yourflowname --project yourproject --watch
if your flow name is unique, you don't need to specify the project above.
if you want to run it agentless but still run in a backend, use --execute:
prefect run --name yourflowname --execute
if __name__ == '__main__':
flow.register("project") # or:
# flow.run()
Using main will work, since Prefect won't execute it (it only looks at the Flow object), so even if you have flow.run() there, it doesn't matter and you can keep it within your code if you wish.Steph Clacksman
02/22/2022, 12:21 PMflow.run()
in my code as I need the dask executor.
I've managed to get the flow registered using cli now. Though I have realised I need docker storage, and whenever I try to register with docker storage I get this error:
ValueError: File /tmp/tmphu08cn99/ already exists in /tmp/tmphu08cn99
(different tmp.... value each time). Any ideas why this might happen?Anna Geller
Steph Clacksman
02/22/2022, 12:30 PMDOCKER_STORAGE = Docker(
image_name="my_flow",
files={
"/pathtoapp/pipeline/tasks/": "tasks/",
"/pathtoapp/pipeline/utils/": "utils/"
},
env_vars={
"PYTHONPATH": "$PYTHONPATH:tasks/:utils",
}
)
Anna Geller
FROM prefecthq/prefect:0.15.13-python3.8
RUN /usr/local/bin/python -m pip install --upgrade pip
WORKDIR /opt/prefect
COPY tasks/ /opt/prefect/tasks/
COPY utils/ /opt/prefect/utils/
Once you set WORKDIR and copy your files this way, you shouldn't need to set PYTHONPATH.
Then in your Docker storage you can do:
DOCKER_STORAGE = Docker(
image_name="my_flow",
dockerfile="Dockerfile") # path to Dockerfile
Steph Clacksman
02/22/2022, 2:30 PM