A couple of issues came up when I tried tor register a new flow: I develop my flows in a remote docker container, with local path /workspaces/prefect-dev/flows. When I try to register with working directory at prefect-dev using:

prefect register --project "some-project" --path flows/some_flow.py

my flow will get registered using the entire path(/workspaces....) . My agent searches for flows at ~/.prefect/flows, so I get a "Failed to load and execute Flow's environment" error. At the same time, even though I did not specify any labels, and there's no reference to any label whatsoever in any part of the source code, that flow gets created with one label that looks like an id of some sort (ie. 447c6079d9b5). My local agent is created using:

prefect agent local start --name $$(hostname) --no-hostname-label

Both of these issues came up with my latest flow, but I have used the same setup for a few months now without any of these issues. Any idea on what might be going on?

Hey @Andre Muraro, the default label can be turned off with storage by using

Local(…,add_default_labels=False)

@Kevin Kho I do not configure storage at all. The agent and dev container s just get a mount for the flows directory. Maybe the default configurations changed?

What versions did you move from / to?

@Zanie python api went from 15.0 to 15.2. Server has been using latest(yeah, yeah, I know...). @Kevin Kho Adding

storage=Local(add_default_labels=False)

has indeed solved the label issue. However I'm still getting the flow environment issue. Seems to be mapping the full path, instead of the relative path as previously

I don’t think anything changed between 0.15 to 0.15.2. For the environment issue, you can provide the

directory

where to create and look for the Flow. Docs here

@Kevin Kho No, like I said, the setup is exactly the same, save for the version changes. I just tried the path and directory args for Local, but that did not work. ~/.prefect maps the path on the server to my development path - /vscode/home/.prefect, which is not even the one I'm actually using for the flows. The path on the agent would be /root/.prefect, but trying that one to register from the dev container results in a PermissionError.

Ok I think I am seeing different behavior between

flow.register()

and using the CLI for registration. Will ask the team about this, but I think that is why your behavior changed. Where you using

flow.register()

before instead of the CLI? For your setup, sorry trying to understand but am still a bit unclear. You are saying that the server

~/.prefect

maps to your development path

-_/vscode/home/.prefect

,_ which means you can’t specify the directory because the two paths are not the same? And then after trying that, you tried to register from the server to a local path on the server but that’s when you got the permission error?

It's possible I broke relative paths with https://github.com/PrefectHQ/prefect/pull/4761

@Kevin Kho I have always used the CLI, never tried the flow.register() method. What I meant is, ~/.prefect maps to /home/vscode/.prefect on my development machine, and to /root/.prefect on the server/agent. When I run Local(directory='~/.prefect') on the dev machine, it gets registered on the server as the /home/vscode path(which doesn't exist). After that, I tried using Local(directory='/root/.prefect'), both on my development machine, and the production environment(which, by the way, I won't have access for much longer, so even if that worked it would not have been feasible), which resulted in Permission on dev machine, and an "OSError [Errno30] Read-only file system" in the prod environment.

I think you want to pass

validate=False

to your

Local

storage so the user path is not expanded on registration

@Zanie Tried both

Local(directory='~/.prefect', add_default_labels=False, validate=False)

and

Local(path='~/.prefect/flows/some_flow.py', add_default_labels=False, validate=False, stored_as_script=True)

Both resulted in:

Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '~/'")

Yeah, the user path is never expanded again.

@Zanie @Kevin Kho Would reverting back to 0.15.0 solve this? is there any other alternative? The graphql api maybe? Would it allow me to change the path?

I don't understand how it was working on 0.15.0, I do not think that would solve it.

@Zanie That reverted to my original problem:

Local(directory='.prefect', add_default_labels=False)

State Message: Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/workspaces/prefect-dev/'") That mapped to my working directory in the dev machine...

You will need to disable validation still or it will resolve to an absolute path

@Zanie Still no success:

Local(directory='.prefect', add_default_labels=False, validate=False)

State Message: Failed to load and execute Flow's environment: TypeError("the 'package' argument is required to perform a relative import for '.prefect/extracao-spw-fi/2021-08-25t22-02-17-252954-00-00'")

The error is confusing, but that means that

os.path.isfile(".prefect/extracao-spw-fi/2021-08-25t22-02-17-252954-00-00")

returned

False

so it tried to load it as a module

yeah, agent runs at ~/.prefect in a container.

Local(directory='.prefect/flows', add_default_labels=False, validate=False)

State Message: Failed to load and execute Flow's environment: TypeError("the 'package' argument is required to perform a relative import for '.prefect/flows/extracao-spw-fi/2021-08-25t22-06-04-588590-00-00'") trying just

Local(directory='flows', add_default_labels=False, validate=False)

yielded an error during registration, instead of flow run time: Traceback (most recent call last): File "/usr/local/pip-global/bin/prefect", line 8, in <module> sys.exit(cli()) File "/usr/local/pip-global/click/core.py", line 829, in call return self.main(*args, **kwargs) File "/usr/local/pip-global/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/pip-global/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/pip-global/click/core.py", line 1236, in invoke return Command.invoke(self, ctx) File "/usr/local/pip-global/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/pip-global/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/usr/local/pip-global/click/decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/pip-global/prefect/cli/build_register.py", line 65, in inner return func(*args, **kwargs) File "/usr/local/pip-global/prefect/cli/build_register.py", line 827, in register register_internal( File "/usr/local/pip-global/prefect/cli/build_register.py", line 551, in register_internal stats += build_and_register( File "/usr/local/pip-global/prefect/cli/build_register.py", line 447, in build_and_register prepare_flows(flows, labels) File "/usr/local/pip-global/prefect/cli/build_register.py", line 320, in prepare_flows flow.storage.add_flow(flow) File "/usr/local/pip-global/prefect/storage/local.py", line 141, in add_flow os.makedirs(os.path.dirname(flow_location), exist_ok=True) File "/usr/local/lib/python3.9/os.py", line 225, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: 'flows/extracao-spw-fi'

How are you getting your flows to your agent?

they get published in a path in the agent machine. that path is mounted in the container at /root/.prefect/flows

With

directory=".prefect"

your flow would be written to a path relative to where you called the CLI e.g.

./.prefect/flow.pkl

ok, now I get what is happening. That did the trick:

Local(path='flows/fiim/extrair_cvm.py', add_default_labels=False, validate=False, stored_as_script=True)

Although I'm still confused as to why that worked fine without even instantiating a Local object at all and now suddenly it didn't work...

I'm also very perplexed that it worked before

second one,

prefect register --project "Automações" -p flows/fiim/extrair_cvm.py

The old one (

prefect register flow

) calls

flow.register()

and does not always resolve paths to absolute paths. The new one resolves to absolute paths unless you define a

Local

storage explicitly. I presume you switched to the new one recently since they're similar? Otherwise I can't think of a way it would have worked before.

No, always used the same method. Same docker file, same docker-compose, same paths, same devops, same commands, everything... Only changes were the versions I mentioned previously. I had a big publishing spree a couple months back, but I think this one today was the first since.

Interesting. Sorry to hear we broke it somehow. I'm not sure what happened.