Nate Jahncke
04/13/2021, 12:49 AMstorage.directory
metadata that is automatically added to flows that are configured as mine are, I'm forced to register the flow from within the agent container, rather than just running it in vscode (which tacks on my own home dir as the storage.directory
and of course yields an exception on flow execution).
This certainly works, and is not a big deal at all, but I'm wondering if I can force the metadata somehow? Passing a directory to the Local() object of course won't allow me to use a path that only exists in the agent container, as it's for local storage, and while I could probably fake it and use a dir common to both, I would really prefer not to do that even if it works (as that directory in my local system would get filled up with files I don't want to exist outside of the container).
Maybe what I have in mind just doesn't work? Some of this is definitely down to my two-day-old understanding of Prefect's architecture, but the real goal is just keeping absolutely everything related to Prefect in the docker environment except for flow development. If possible.
As I said, not crucial as I have a totally working solution now and am eager to build some flows, but thanks in advance for any advice!Kevin Kho
04/13/2021, 2:26 AMDockerRun
working with LocalStorage
here: https://github.com/kvnkho/demos/tree/main/prefect/docker_with_local_storageLocal()
to point to a location in the Docker container.Nate Jahncke
04/13/2021, 2:48 AMKevin Kho
04/13/2021, 2:54 AMflow.run()
. For registering on the Cloud, yes it will access the path locally for the container. I think another option to consider is you can also mount a local drive when you make the container. Not quite sure what you mean about state of the container. Are you trying to keep track of a state?Nate Jahncke
04/13/2021, 2:05 PMKevin Kho
04/13/2021, 2:08 PMNate Jahncke
04/13/2021, 2:12 PMKevin Kho
04/13/2021, 2:18 PMNate Jahncke
04/13/2021, 2:23 PMprefect-agent
) that I want to run flows in. At present I can do that just fine, as long as I shell into the prefect-agent
container and register the flow from inside of it. This is not a big deal for me. My hope, however, was that there was some combination of storage and run_config classes I could use to allow me to specify the prefect-agent
container, not an image, to use as the host of the flow. Just to skip a step and therefore some potential downstream issues, but as I said, it's not a big deal the way it is right now.Zanie
04/13/2021, 3:20 PMLocalStorage
. You may get some extra milage by setting path
and stored_as_script
which I believe will ignore the path your flow is at during development/registration and just attempt to pull it from the given path at runtime. You'll still need to place the flow file at that path in the agent container (I'd recommend mounting it). Generally, you'd be much better off using a remote storage such as S3 or Github, local storage is basically designed for use with a local agent on the same machine to get people started.Nate Jahncke
04/13/2021, 3:27 PMprefect-agent
container). We will definitely use remote storage outside of our development environments, but I think I have everything I need at this point to get everyone up and running. Thanks a ton to you and @Kevin Kho for all the help! You guys are great.Zanie
04/13/2021, 3:30 PMNate Jahncke
04/13/2021, 3:36 PMstack = inspect.stack()
). I would have thought __file__
would work, but it doesn't during agent execution 🤷
Here's my test Flow config:
# Configure Flow
with Flow(
name="test_flow",
schedule=IntervalSchedule(interval=dt.timedelta(minutes=1)),
storage=Local(
add_default_labels=False,
stored_as_script=True,
path=f"/flows/{os.path.basename(stack[0].filename)}"
)
) as flow:
# Define Flow
test_flow()
# Register Flow
flow.register(project_name="default_project", labels=["default_label"])
My prefect-agent
container has its /flows/
dir mapped to my local flows dir via docker-compose:
volumes:
- ${PREFECT_FLOWS_DIR}:/flows/
Now I can register flows from outside the docker environment and the runs are picked up by my agent container exactly as I'd hoped. Thanks for all your help, guys!Kevin Kho
04/13/2021, 5:34 PMZanie
04/14/2021, 2:01 AMMarvin
04/14/2021, 2:01 AM