Hello, is there an option not to store versions in prefect at all? We have git version insread.
07/11/2022, 10:58 AM
there might be, are you asking for 1.0 or 2.0?
but if you are using your own versioning, then the one provided by Prefect can be used as a bonus for the UI? You can still track down the code artifacts via Git commit hash or image tag, but it's not the same as your flow version
in fact, your code artifacts may change while your flow doesn't - e.g. when dependencies change but flow is the same
07/11/2022, 11:26 AM
I'm asking for 1.0
The one provided in UI can't be a bonus, if you keep your own file names and store them in S3. Prefect just overrides the files, keeping lot's of redundant useless data in db and ui.
We have 2x t3.medium instance with 3x pods for every service except DB.
And it's a problem for prefect to remove 10 version of flow + history for last 3 month. Real problem. Prefect doesn't do that in batch mode via DB, it does it via single requests via Graphql and as a result:
• UI always throws an error, "cant remove flow, please ask support".
• DB keeps hundreds of connections active when removing data and Prefect itself just ddosing itself.
• When this happens GraphQL pod just stops responding healthcheck and K8s just kills the pod.
07/11/2022, 11:53 AM
This sounds like a complex issue indeed and I understand your struggle here. I could open an issue in the Prefect Server repo and you or some contributor could look at it. But Prefect 2.0 (which is planned to come out of beta status this summer) doesn't suffer from this problem so perhaps instead of investing time in it, you could explore how to approach that in Prefect 2.0 and plan your migration to 2.0?
07/11/2022, 11:57 AM
I wanted to contribute to 1.0 as found a lot of minor bugs when researching and asking questions for last 3 month, but that makes no sense.
The only reason why we're using 1.0 is that because of 2.0 is non live; will have to check the same issues in 2.0
07/11/2022, 12:19 PM
you can check that already since:
1. We don't plan to do any versioning changes until GA
2. GA is planned for very soon (end of July)