David Evans
04/28/2022, 11:58 AMprefecthq:prefect
? or would it just need python:3
?) for each flow? And I guess these docker images would run pip install -r requirements.txt
as a build layer. But if we can achieve this with a virtual environment instead I think that would be preferable (I'm thinking in terms of the flow needed for an engineer to try something out by pushing it to the runner from their local machine)
(I can see the high-level concept here but I'm struggling to see how it will look in practice for the various use-cases)David Evans
04/28/2022, 12:02 PMAnna Geller
David Evans
04/28/2022, 1:24 PM--path='**/*'
any more, so this will have to discover all our flows by some hand-cranked way?
Overall that looks like it would fix the issue of long-running tasks being interrupted, as well as the issue of multiple engineers testing changes at the same time, but also sounds like it will be super slow! โ perhaps longer than the replace-the-agent approach โ and will need us to build our own tooling to abstract it all away for when somebody wants to just test something from their local machine). Am I still missing something here that can simplify things?Anna Geller
David Evans
04/28/2022, 3:11 PMdo you mean your CI building a Docker image would be slow?yes, as well as the need for it to load the new (presumably not cached) docker image when running each flow we had a chat internally and we think the custom python packages is a good option - we'll explore using AWS CodeArtifact for that As for revisiting this later, I think we can, although I personally may well have moved on to an unrelated project by then, but there will still be somebody here interested in this stuff.
Anna Geller
we had a chat internally and we think the custom python packages is a good option - we'll explore using AWS CodeArtifact for thatWow, that's so cool! I agree that this is a great idea. I would really appreciate it if you could share more about your progress on that. Not necessarily a blog post if you don't want to, but even if you could share some Github Gist or code repository showing how you approached it, that would be fantastic!
As for revisiting this later, I think we can, although I personally may well have moved on to an unrelated project by then, but there will still be somebody here interested in this stuff.That's cool to hear. You can subscribe ๐ to this Discourse tag to get an email notification about any new release (a.o. about that topic)