It’s been a while since I’ve used Prefect - I knew...
# best-practices
s
It’s been a while since I’ve used Prefect - I knew a decent amount about deploying V1.0 server with Kubernetes, but it looks like there’s tons of changes in 2.0. These were some really good features I liked, and I want to see if they are available in V2, still: • keeping different flows in different repositories (i.e.
prefect register
as part of a CI was awesome) so that we don’t have to worry about competing dev environments. ◦ I think maybe what we’d want to do now for this is to use S3 storage, since it looks like git storage is only for public repos right now. • Different docker images for diferent flows: ◦ It looks like in V2, Agents are what the executors (?) in V1 were. Would using different images for different flows mean that we needed to have different agents available?
1
r
Hi Sam, Private Git storage was released yesterday: https://discourse.prefect.io/t/prefect-2-6-3-release-with-support-for-private-github-repositories-fixes-to-results-and-rrule-schedules-and-more/1753 Or, as you mentioned, you could use S3. You might choose to keep each flow in a different bucket, or you could use one bucket but keep each flow in a separate folder. As for using different Docker images for different flows, you could accomplish that with a single agent by creating a deployment for each flow and then pointing multiple deployments at the same work queue. Having said that, Prefect 2 agents are quite lightweight, so you might find it just as easy to run an agent per deployment.
upvote 1
s
Oh well, then, good timing on the git thing 😄 - thanks for the material to look through. We aren’t currently using kubernetes here - are there instructions on setting up agents (Work queues?) and or deployments via other methods? I am definitely off on the terminology in the new version, so hopefully that made sense.
r
Yes, there are quite a few options and we have examples both in the docs and on our Discourse board. Since you're familiar with Docker, you might like this example I put together showing how to run the various bits of Prefect 2 using Docker Compose: https://github.com/rpeden/prefect-docker-compose.
We've also added support for ECS and Google Cloud Run if you prefer running containerized flows that way. And I'm working on a finishing up a pull request that will let you do the same using Azure Container Instances. There are a few other options mentioned in the docs as well. If you haven't seen it yet, the Prefect 2 Guide is a great way to get up to speed on everything that's different from V1.
And of course, if you have any questions there are lots of us here on Slack who are happy to help.
a
Follow up question from another v1 user :D, are kubernetes tasks ported already?