Jovan Sakovic
05/06/2022, 1:06 PM--watch
flag.
However, as we’re spinning it all up with docker-compose, and we have multiple Prefect Projects, so I want to run a few of these prefect register
commands in the container’s entrypoint shell script (code example in 🧵)
Is there an easy way to:
• either run the prefect register
command in background, so it lets the rest run
• or, run prefect register
with specifying the project in the Flow script, so we’d need to run the command only once for all projects# Register all Prefect Flows, and watch them for changes
prefect register --project P3 -p /root/flows/project1 --watch
prefect register --project P2 -p /root/flows/project2 --watch
prefect register --project P1 -p /root/flows/project3 --watch
Or a different approach (not sure whether this method exists though) where in for example, the /root/flows/project1/run-dbt-flow.py
has this as the final line
flow.set_project('P1')
and then in the entrypoint shell script:
prefect register --project <arbitrary> -p /root/flows/**/* --watch
(or something along those lines) 😅Anna Geller
05/06/2022, 1:17 PMprefect register
command is invoked either:
• manually by the users when they want to register a new flow or a new version of an already existing flow
• automatically by CI upon merge
This is actually the first time I see someone using the --watch flag in the flow registration process.
Perhaps you could give a CI/CD pipeline a try? This would be easier to set up than a service watching code changesJovan Sakovic
05/06/2022, 1:21 PM--watch
flag got me really excited when I saw it be one of the features of the newer versions - exactly because I didn’t want to keep manually going into instances, exec-ing into the docker containers, and running python3 run-dbt-flow.py
Now, we’re working with a CI/CD pipeline, indeed - I was just curious whether there was a fancier way that this flag can achieve the same, rather then putting a docker exec -d prefect_agent bash -c "prefect register ……"
in the pipeline 😅Anna Geller
05/06/2022, 1:26 PMJovan Sakovic
05/06/2022, 1:43 PMAnna Geller
05/06/2022, 1:44 PMJovan Sakovic
05/06/2022, 1:48 PM