Hi all :wave: :prefect-duck: Working on having ou...
# prefect-community
j
Hi all 👋 prefect duck Working on having our Flows automatically re-registered when there are changes with `1.1.0`'s
--watch
flag. However, as we’re spinning it all up with docker-compose, and we have multiple Prefect Projects, so I want to run a few of these
prefect register
commands in the container’s entrypoint shell script (code example in 🧵) Is there an easy way to: • either run the
prefect register
command in background, so it lets the rest run • or, run
prefect register
with specifying the project in the Flow script, so we’d need to run the command only once for all projects
👋 1
marvin duck 1
Copy code
# Register all Prefect Flows, and watch them for changes
prefect register --project P3 -p /root/flows/project1 --watch
prefect register --project P2 -p /root/flows/project2 --watch
prefect register --project P1 -p /root/flows/project3 --watch
Or a different approach (not sure whether this method exists though) where in for example, the
/root/flows/project1/run-dbt-flow.py
has this as the final line
Copy code
flow.set_project('P1')
and then in the entrypoint shell script:
Copy code
prefect register --project <arbitrary> -p /root/flows/**/* --watch
(or something along those lines) 😅
a
Thanks for sharing your approach. Usually, the
prefect register
command is invoked either: • manually by the users when they want to register a new flow or a new version of an already existing flow • automatically by CI upon merge This is actually the first time I see someone using the --watch flag in the flow registration process. Perhaps you could give a CI/CD pipeline a try? This would be easier to set up than a service watching code changes
🙏 1
if you want to see some examples for that, check this Discourse tag
🙌 1
j
Thanks for the pointer @Anna Geller 🙌 This
--watch
flag got me really excited when I saw it be one of the features of the newer versions - exactly because I didn’t want to keep manually going into instances, exec-ing into the docker containers, and running
python3 run-dbt-flow.py
Now, we’re working with a CI/CD pipeline, indeed - I was just curious whether there was a fancier way that this flag can achieve the same, rather then putting a
docker exec -d prefect_agent bash -c "prefect register ……"
in the pipeline 😅
a
Yeah, this is also an interesting approach! There is definitely no right or wrong here. But somehow I think (it's really just my opinion) that running a persistent service watching for changes to the flow code could be costly (you'd need to run it as a service 24/7). Given that tools such as GitHub Actions or CircleCI are already watching changes pushed and merged to your repository, setting this via a CI/CD pipeline invoked upon merge to the main/master branch could be much simpler and less expensive.
j
That’s a fair point 😛 It’s a shame such a nice flag won’t be put to use yet :(( Thanks for your help 🙏 cc @Gabriel Kalunga
👍 1
1
a
Just to clarify: regarding your point "It’s a shame such a nice flag won’t be put to use yet :((" - you can totally use it! The question is whether you want to use it as an alternative to registering from CI/CD.
👌 1
j
completely understood - exactly that 👉 I’m just not keen (atm) to go through the hassle of working that part out, when there’s a known solution to it through CI/CD 😄
👍 1