I am a little confused about the package dependencies when using S3 as a storage block and then have an agent running somewhere else, read the code from there. Does the server with the agent have to have all the dependecies pre installed or can i somehow pull them from the bucket and have them installed? thank you
c
Chris White
03/09/2023, 9:30 PM
Hey Tomás! This is a valid question - right now the server with the agent (or more precisely, the environment where the flow will run which could be a container) needs the dependencies pre-installed. This can be annoying, especially for work that runs in a subprocess on a remote machine; we're internally working on alterations to our deployment patterns that would allow for runtime changes to the environment pulled from the same location where the flow code is stored (a bucket, github, etc.). These changes will take some time to validate, document & release, but they're in active development. If you monitor for updates related to "prefect workers", the changes will most likely be rolled out in parallel with workers
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.