https://prefect.io logo
Title
j

Jean-Michel Provencher

11/29/2022, 4:40 PM
Hello. Any plans in the future to have a terraform provider to manage organization resources? Things like workspaces, api keys, and things like that ? It would be pretty useful for enterprise that needs to manage their infrastructure with a fully automated IaC.
āœ… 3
āž• 6
šŸ’Æ 5
a

Anna Geller

11/29/2022, 5:09 PM
a little bit of a chicken and egg problem because you need to have a workspace URL and API key to interact with Prefect programmatically, and creating a workspace is a subject to quotas based on your plan e.g., you may not be able to create 5 new workspaces through terraform if you purchased only 3 but a totally understandable problem (it would be cool if possible), I believe a good middle ground is to be able to create all agents & blocks programmatically within a workspace and this is something I was trying to do in those 2 repos: ā€¢ https://github.com/anna-geller/dataflow-ops ā€¢ https://github.com/anna-geller/prefect-dataplatform and for agents, we have several terraform recipes here: https://github.com/PrefectHQ/prefect-recipes
essentially, this would allow you to create a new workspace and run this automation to create all agents, blocks (incl. all storage and infra blocks) + all deployments
šŸ‘ 1
@Jean-Michel Provencher I talked to the platform team and we added it to the Cloud backlog šŸ™Œ once we release that, we'll announce it in #announcements channel
j

Jean-Michel Provencher

11/29/2022, 6:25 PM
I would be tempted to say that even workspaces could be created via terraform. The terraform provider could simply throw an error if you are over the maximum number of workspaces for your account
For my understanding, what exactly was added to the cloud backlog ? The terraform provider idea?
a

Anna Geller

11/29/2022, 8:24 PM
the problem was added, how the solution will look like is up to our platform experts šŸ™Œ we get you, we understand the problem
j

Jean-Michel Provencher

11/29/2022, 8:24 PM
šŸ‘ Thanks
m

Maikel Penz

12/13/2022, 2:14 AM
+1 to this. I agree setting up extra Workspaces can be useful but priority for me would be to be able to deploy Agents, Automations, Notifications, Blocks, etc.. via terraform
šŸ‘ 2
šŸ™Œ 2
j

Jean-Michel Provencher

12/19/2022, 4:58 PM
Also, the dream would be to actually be able to create "deploments" using terraform. It would make the life of teams using terraform so much easier. Instead of having to manage a CI/CD on their side that actually perform deployment create and deployment apply commands from the CLI.
a

Anna Geller

12/19/2022, 5:33 PM
Jean, creating deployments doesn't provision any infra though, it only sends 1 API call with deployment metadata to Prefect so not sure if that's a Terraform use case but anyway we have an internal ticket for that, we'll update in #announcements if we add a feature to do that. Thanks for this request and adding details
j

Jean-Michel Provencher

12/19/2022, 6:52 PM
I know, but managing lifecycles of deployments (create, update, delete) is far from being easy using the cli/python. Having terraform for that would make it easy to manage deployments for flows. Meaning ā€¢ Adding a terraform resource for a flow deployment would create a deployment in prefect ā€¢ Modifying any attributes to the deployment would update the deployment ā€¢ Removing that resource from terraform would delete the deployment Currently there's no "clean" way of managing the lifecycle of these deployments other than for example deleting all deployments during each release, then recreating them all for examples.
:gratitude-thank-you: 2
m

Maikel Penz

12/19/2022, 8:25 PM
deployments is a tricky one. I agree with Jean that managing deployments with Python or the CLI only helps us deploying them and not managing their lifecycle. But in my case for example I struggle to see it working as terraform infrastructure, as we have data analysts deploying workflows all the time, and asking them to manage terraform sounds very out of their scope of work. We haven't migrated to 2.0 yet, but in 1.0 that's why we created a wrapper library on top of Prefect to automate some of these API calls and make it simpler for everyone
āœ… 1
:gratitude-thank-you: 2
j

Jean-Michel Provencher

12/19/2022, 9:17 PM
yeah, I have the same problem. Still I would prefer to have a state with the proper lifecycle and I could create a terraform module with really simple variables that data engineer can fill in that would create their flow automatically during our team deployment to production.
šŸ‘ 2
m

Maciej

02/15/2023, 12:17 AM
I'm a little late to the party here, but my main use case would be for creating blocks. Having a workspace with API keys set up ahead of time and provided as variables is fine. And I don't mind managing deployments separately either. But creating blocks like S3 storage and ECSTask should definitely be tightly coupled with the Terraform that creates these S3 buckets and ECS task definitions.