https://prefect.io logo
n

Nate Lambeth

01/15/2021, 5:20 PM
hello all! i am in the process of trying to hack together an ETL flow. i've successfully gotten prefect server running on a remote host in a docker container, and it can run a simple flow with no problem as long as the flow is built into the container's image. what i don't quite follow is how to register another flow without stopping and rebuilding the container--it looks like there used to be a tutorial on using prefect with Docker, but it was removed? https://docs.prefect.io/orchestration/tutorial/docker.html is there an example or tutorial somewhere that explains this? i can't figure out how to run the python script containing the flow registration code in the running prefect environment
n

nicholas

01/15/2021, 5:22 PM
Hi @Nate Lambeth - if I follow your question correctly, you're looking to register a new flow with your Server instance, correct?
n

Nate Lambeth

01/15/2021, 5:22 PM
hi! correct
n

nicholas

01/15/2021, 5:24 PM
Ok perfect; you should be able to do all of this locally, assuming you've exposed the correct endpoints on your server (which it sounds like you have!). To start, you'll want to make sure you've set
endpoint
correctly in your local
~/.prefect/config.toml
:
Copy code
[server]
endpoint = "YOUR_MACHINES_PUBLIC_IP:4200/graphql"
Then before you run your flow's registration script locally, you'll want to call:
prefect backend server
, which tells Prefect you want to hit a Server backend (and so skips the Cloud authentication steps). Once that's done, you should be able to call
flow.register(project_name="My Project")
from any local python script and you'll see the flow in your UI 🙂
Does that clear up the process a little?
n

Nate Lambeth

01/15/2021, 5:31 PM
maybe i was unclear. i have prefect server in a docker container on a host. when run, that container starts the prefect server and a local agent and registers a dummy flow which runs fine. the issue is, i'd like to register another flow in that container without stopping it and rebuilding it to include the new flow. i'm not much of a docker expert; i could have sworn there was a docker tutorial i read a few months ago, but the url is dead
n

nicholas

01/15/2021, 5:35 PM
Hm it sounds like you might be doing something we don't fully support; Prefect Server already runs in several containers (through
docker-compose
), and it sounds like you're wrapping the Server instance in another Docker container?
Either way, if you can connect to that Server instance you should be able to follow the steps above to register new flows, they don't need to live on the Server instance
n

Nate Lambeth

01/15/2021, 5:38 PM
i've tried following directions for running the docker image but it just exits. i should have saved the docker tutorial back in the fall
n

nicholas

01/15/2021, 6:05 PM
@Nate Lambeth that doc you linked is for the prefect (core) docker image and isn't meant as a deployment scenario for Prefect Server. Have you seen this doc for single-node deployment of Prefect Server?
n

Nate Lambeth

01/15/2021, 6:14 PM
yes, i've seen that. it runs fine on my local machine, but i get errors from docker related to the version set in docker-compose.yaml when on its intended host. it wants me to set version 2 in the compose file even though docker-compose and docker are both up to date for the 3.7 set in the file
n

nicholas

01/15/2021, 9:21 PM
Sorry @Nate Lambeth - I'm a little confused about what issue you're running into now; can you clarify? My understanding initially was that you had successfully stood up Prefect Server and were able to connect to it (you could see the UI and the UI would reflect the flow that you'd registered) but that you were unclear how to register another flow. It sounds like you're having trouble standing up the server now though, or am I misunderstanding?