https://prefect.io logo
Channels
Powered by
s

Steph Clacksman

02/22/2022, 10:31 AM
Hello, beginner's question here! I see a lot of examples that are like this:
Copy code
with Flow("my-flow) as flow:
    # do some things here
# EITHER flow.register("project_name")
# OR flow.run(executor=DaskExecutor())
I want to run my flow through prefect cloud, so I understand that I need to register my flow , but when I want to run it it looks like I need to edit the code, which seems a bit weird when it's dockerised and running on a remote server. Is it normal to register the flow like above before building the docker image, then switch it to the run command before building? Or is there something I'm missing?
I have tried registering using the cli, but always get 0 flows registered.
a

Anna Geller

02/22/2022, 10:37 AM
Excellent question and I'm honestly a bit biased in favor of the CLI. I would recommend registering and running flow through the CLI. Here is how you can use it:
Copy code
prefect register --project YOUR_PROJECT_NAME -p /path/to/your/flow.py
When you provide the exact path to your flow, it should register more than 0 flows 🙂 And then, if you want to run the flow locally, you can run it by providing a path:
Copy code
prefect run -p yourflow.py
and if you want to trigger a flow run on a backend (running on agent), you can run:
Copy code
prefect run --name yourflowname --project yourproject --watch
if your flow name is unique, you don't need to specify the project above. if you want to run it agentless but still run in a backend, use --execute:
Copy code
prefect run --name yourflowname --execute
this means: if you're using the CLI to register/run the flow, no need to specify the flow.run() or flow.register() in your code. Alternatively, you can do:
Copy code
if __name__ == '__main__':
    flow.register("project") # or:
    # flow.run()
Using main will work, since Prefect won't execute it (it only looks at the Flow object), so even if you have flow.run() there, it doesn't matter and you can keep it within your code if you wish.
s

Steph Clacksman

02/22/2022, 12:21 PM
Thanks. That makes sense, though I think I may need 
flow.run()
in my code as I need the dask executor. I've managed to get the flow registered using cli now. Though I have realised I need docker storage, and whenever I try to register with docker storage I get this error: 
ValueError: File /tmp/tmphu08cn99/ already exists in /tmp/tmphu08cn99
(different tmp.... value each time). Any ideas why this might happen?
a

Anna Geller

02/22/2022, 12:24 PM
Why is that? You can attach executor to your Flow object directly
When you register with Docker Storage, it builds an image and potentially pushes to registry - this temp is probably your pickled flow object
s

Steph Clacksman

02/22/2022, 12:30 PM
Oh yeah, I have my dask executor on my Flow object now, thansk!
👍 1
Yes, I want to build an image to register with docker storage, but keep getting that error.
Copy code
DOCKER_STORAGE = Docker(
    image_name="my_flow",
    files={
        "/pathtoapp/pipeline/tasks/": "tasks/",
        "/pathtoapp/pipeline/utils/": "utils/"
    },
    env_vars={
        "PYTHONPATH": "$PYTHONPATH:tasks/:utils",
    }
)
a

Anna Geller

02/22/2022, 12:39 PM
Maybe you can build your own Dockerfile? It would probably make it clearer what files end up where:
Copy code
FROM prefecthq/prefect:0.15.13-python3.8
RUN /usr/local/bin/python -m pip install --upgrade pip
WORKDIR /opt/prefect
COPY tasks/ /opt/prefect/tasks/
COPY utils/ /opt/prefect/utils/
Once you set WORKDIR and copy your files this way, you shouldn't need to set PYTHONPATH. Then in your Docker storage you can do:
Copy code
DOCKER_STORAGE = Docker(
    image_name="my_flow", 
    dockerfile="Dockerfile") # path to Dockerfile
Here is a sample repo with Dockerfile if you need more examples
s

Steph Clacksman

02/22/2022, 2:30 PM
That's great, thanks! Adding those files to my dockerfile and adding the dockerfile to the docker storage definition got it registering successfully. To get it running without module not found errors I followed your setup.py from your deployment examples.
🙌 1
8 Views
Powered by