Darragh
05/12/2020, 10:20 AMZachary Hughes
05/12/2020, 1:25 PM~/.prefect/config.toml
file to include the API address of your Prefect Server and ensure you've run prefect backend server
. Updating your config.toml
to look something like the below should do the trick:
[server]
[server.ui]
graphql_url = "YOUR_MACHINES_PUBLIC_IP:4200/graphql"
• You don't need to use Docker storage, but I'd strongly it for ease of use. Using docker storage builds an image with the dependencies Prefect needs to run, in addition to any dependencies you might have specified yourself.
• If you'd rather bundle your flow files and use local storage, you can do that. It's worth noting that if you do register a flow locally, it'll be tagged with tags specific to the machine on which it was registered. You'll need to ensure that the agent you spin up to retrieve that flow has matching tags, otherwise the agent will not retrieve that flow.
And to your last question, the nice thing about using Server is that it's quite similar to Cloud-- this guide should do the trick, but let me know if you have any additional questions.
https://docs.prefect.io/orchestration/tutorial/first.html#write-flowDarragh
05/12/2020, 1:38 PMZachary Hughes
05/12/2020, 1:42 PMlocalhost
, so since you're trying to register to a remote server, you'll need to give it directions.
If I'm understanding what you're trying to do properly, then you'll want Prefect available in your CI/CD container to streamline the registration process. So once you've gotten Prefect into your container so you can register and pointed Prefect at the correct API, you should be good to go.Darragh
05/12/2020, 1:50 PMprefecthq/prefect
image is? ] and then python $FLOW_FILE
to get it to build
• a D-in-D container, install prefect using pip install prefect, and then build using the same python $FLOW_FILE
as above
Both of those would need the ~/.prefect/config.toml
to be updated to point at my AWS Prefect server.
Am I right so far?Zachary Hughes
05/12/2020, 2:10 PMDarragh
05/12/2020, 2:11 PMZachary Hughes
05/12/2020, 2:16 PMPedro Machado
05/12/2020, 11:14 PMDarragh
05/13/2020, 3:03 PMTyler Wanner
05/13/2020, 3:18 PMpython FLOW_FILE
in a prefect image. This should work locally fine with a docker storage object if you mount the docker daemon to the builder container (prefect image) as suggested by scott. However, it may not work in GitLab at the moment. I’ve no experience with GitLab but a lot of CI systems use remote docker daemons, which requires the use of $DOCKER_HOST under the hood… This value is not respected by the Docker storage atm, but a fix is in and ready to be released with 0.11.0docker run -v $PWD:$PWD -w $PWD -v /var/run/docker.sock:/var/run/docker.sock IMAGE_NAME command
By mapping the $PWD volume, and docker.sock, it allows the Docker instance inside the container to behave as if it was running natively on the Docker host.”Darragh
05/13/2020, 3:27 PMpython FLOW_FILE
in a prefect image” - is there such a thing as a pre-packaged prefect image for this kind of purpose?