<@ULVA73B9P> in prefect 3, storage is a different ...
# ask-marvin
t
@Marvin in prefect 3, storage is a different machine than infrastructure. That is true for dynamic infrastructure. But is it true for static infrastructure? Say I use serve on a machine for a long running infrastructure, I have to have all the source code of flows on that machine right? I cannot have the source code on another machine, or else how can I pull these code into the infrastructure
m
In Prefect 3, the approach to handling infrastructure and storage can vary based on your needs. The distinction between static and dynamic infrastructure primarily affects how flows are deployed and run, not necessarily where the flow's code resides. 1. Static Infrastructure: When using static infrastructure for long-running processes, you essentially have a server or environment set up where Prefect's "serve" mechanic runs. In this setup, you indeed need to have access to your flow's source code on the server where the Prefect serve process is running. Since it's a long-running setup, it expects that all necessary configurations, including the code, are accessible in the environment where it's deployed. This kind of setup offers more control but requires that the infrastructure remains running and properly configured. 2. Dynamic Infrastructure: On the other hand, dynamic infrastructure setups often decouple storage from the running environment. With the use of work pools and pull steps, it's possible to have your code stored in a remote location or repository (like GitHub) and pulled onto the infrastructure dynamically each time the flow is run. This setup provides more flexibility and scalability. The documentation about Prefect 3 indicates that you can have a highly decoupled setup where "pull steps allow users to decouple their workflow architecture," meaning your flow's code can be dynamically retrieved from remote filesystems even in situations normally requiring static hosting. This allows for flexibility in deployment strategies, but for static infrastructure, maintaining the code locally is common practice unless specific remote pull mechanisms are set up. Here's more information about Prefect's deployment approaches: Prefect Deployment Approaches. If your environment supports it, you might find ways to pull the necessary bits dynamically even in static infrastructure, but that's an additional layer of setup beyond a typical static configuration. Feel free to check the details in Prefect's deployment documentation or consider discussing specific configurations with Prefect community forums if you need a unique setup.
n
> I cannot have the source code on another machine, or else how can I pull these code into the infrastructure you can actually! this is what
from_source
is for
some_flow.serve()
is just shorthand for
Copy code
flow.from_source(source=str(Path(__file__).parent), entrypoint="file.py:fn").serve()
consider this example
Copy code
from prefect import flow

if __name__ == "__main__":
    flow.from_source(
        source="<https://gist.github.com/3eb9333625d465cb74381c5116be7aee.git>",
        entrypoint="tour_of_artifacts.py:main",
    ).serve()
which should actually just work, even though you don't have the code on your machine (until
from_source
clones it for you)
so you can start a long lived container with your served flow as your entrypoint, have your
source
point to a repo, and then just push code to that repo, and we'll pull the updated code automatically when its time to run