https://prefect.io logo
Join Slack
Powered by
Hi guys! I'm having some troubles with a flow that...
# prefect-community
m
Hi guys! I'm having some troubles with a flow that was working well until yesterday: I added an import that I had forgotten, and registered the flow again. I'm using Prefect Cloud and I have a couple of agents in an EC2 instance. Now when I try tu run the flow I get a failed run with the message:
Copy code
Last State Message
[2 Sep 2020 3:24pm]: Failed to load and execute Flow's environment: FileNotFoundError(2, 'No such file or directory')
I don't get which file is the one that can't be found
j
Hi @Matias Godoy can you provide an example of what your flow's storage looks like and the import that you had changed to include?
m
That is the thing, I'm not specifying any storage option, so my guess is that it's using the default one
I had not changed this flow in a few months so maybe something has changed in the latest versions
let me share the basics of my flow:
j
Is the import that you changed a local module or something that is pip installable?
m
oh, the import was a 
from time import sleep
nothing fancy
j
Ah so the import wouldn't be it then. There's a good chance that since the flow is originally from an older version of prefect something could have changed 🙂
m
nice, let me show you what I have right now
Copy code
import prefect
from prefect import Flow, Parameter
from prefect.tasks.control_flow.conditional import merge
from prefect.tasks.control_flow.case import case

def main() -> None:
    with Flow('Full inspection') as flow:
        env = Parameter('env', default='DEV', required=False)
        user_token = Parameter('user_token')
        inspection_id = Parameter('inspection_id')
        is_video = Parameter('is_video', default=False, required=False)

        # some tasks

    # Register the flow in Prefect Cloud
    flow.register(project_name='Inspect')

if __name__ == '__main__':
    main()
that's basically my flow, and by running that script I register it in Cloud
I never specify storage or environment (to be honest I don't know what those are)
do you think my flow is missing something?
j
FWIW I am able to register and “run” that flow with no issue 🤔
Oh I think I know what’s happening
m
I tried also with an agent running in my local and it works
but the agents running in the EC2 instance pick it up and it fails 😞
j
Yeah your flow doesn’t have any storage specified therefore it is defaulting to Local storage and your agent running on EC2 can’t run the flow because it doesn’t have access to the flow file
m
okay! that's a start
j
Either you’ll have to move that flow to your EC2 instance manually in order to use local storage or you should look into using some of the other flow storage types https://docs.prefect.io/orchestration/execution/storage_options.html
m
I have the flow in the EC2 instance
because I have it in github, and I have cloned the repo there
and I start the agents like this: 
prefect agent start --label inspect-agent-2 --label prod --token aaa -p /home/ubuntu/inspect-prefect-orchestrator/flows/inspection
j
Can you confirm that when you register the flow it is stored in that path that you are setting on the agent?
m
let me check
j
Otherwise the default local storage flow location is in 
~/.prefect/flows/
m
when I register the flow nothing changes in that directory. I mean, it contains the python files that define the flow
but not the binary version
the weirdest thing is that it was working until I register a new flow yesterday
j
Yeah that is weird! Would you mind opening an issue on the repo with this information? Better to triage there than in this thread 🙂
Also one more thing, could you try running your agent with the 
--show-flow-logs
flag to see more of the traceback from where the error is raised?
m
sure! let me try that
ok, this looks more like it:
Copy code
[2020-09-02 14:17:23,189] INFO - agent | Found 1 flow run(s) to submit for execution.

2020-09-02 14:17:23,873 DEBG 'prefect-agent-1' stdout output:
[Errno 2] No such file or directory: '/Users/matto/.prefect/flows/full-inspection.prefect'

2020-09-02 14:17:23,925 DEBG 'prefect-agent-1' stdout output:
Traceback (most recent call last):
  File "/usr/bin/prefect", line 11, in <module>
    sys.exit(cli())
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/prefect/cli/execute.py", line 80, in cloud_flow
    raise exc
  File "/home/ubuntu/.local/lib/python3.6/site-packages/prefect/cli/execute.py", line 69, in cloud_flow
    flow = storage.get_flow(storage.flows[flow_data.name])
  File "/home/ubuntu/.local/lib/python3.6/site-packages/prefect/environments/storage/local.py", line 77, in get_flow
    return prefect.core.flow.Flow.load(flow_location)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/prefect/core/flow.py", line 1335, in load
    with open(str(fpath), "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/matto/.prefect/flows/full-inspection.prefect'

2020-09-02 14:17:25,027 DEBG 'prefect-agent-1' stdout output:
[2020-09-02 14:17:25,027] INFO - agent | Process PID 1464 returned non-zero exit code
I can see that the agents are looking for the flow binary in a directory that is in my local computer, but they are in an EC2 instance
j
Are you registering the flow on EC2 or on your local computer?
m
I thought I was registering in Prefect Cloud
I see the problem now
j
Yes you’re using Prefect Cloud but on which machine are you calling 
.register
?
Register is saving the flow with Local storage to that machine’s local file system
m
in my local
right
j
Great, so either register it on EC2 or use one of the other cloud-based storage options 🙂
m
so if I register it in EC2 it shoud work
j
Yep!
m
cool, I see
sorry for all the fuzz!
j
No problem!
m
I thought (I don't know why) that when you registered a flow it was uploaded to Prefect Cloud servers
it's clear now 🙂
j
Nope Prefect Cloud and Prefect Server never store any actual flow code!
m
nice, I'll try that and let you know how it went
ok, different error now: 
ValueError: Flow is not contained in this Storage
j
Interesting, did you just run the register script and agent on EC2?
m
correct
and I start the agents like this: 
prefect agent start --label inspect-agent-2 --label prod --token [my_token] -p /home/ubuntu/inspect-prefect-orchestrator/flows/inspection
that directory contains the python files for the flow
okay, I can see that I had prefect 12.0
I just updated to 13.5. We'll see if it helps
it worked! 👏
I had to re-login to Prefect Cloud after upgrading, but it worked
j
Awesome!
m
thanks a lot for your help
I'm a little concerned that something this will happen every time a new version of prefect comes out
maybe I'll have to automate the 
pip install --upgrade prefect
command to keep the agents up to date
j
When working with the Local agent you’re ultimately subject to the environment that the agent runs in which has the potential to cause issues with versions. That’s why a lot of users opt into using Docker storage because you are subject to the static version inside the container and the flow should always work regardless of newer releases
m
oh, that sounds like a cool alternative
so, could I use the same agents i'm running in EC2, but using Docker storage and forget about upgrading them forever?
and every time I make a change in a flow, will I have to SSH into the EC2 instance, pull the changes from github and register the new flow from there, right?
j
That is one option! Or even better you could use GitHub storage in combination with a Docker image so every time you run your flow it will use whichever code you have in your git repo automatically without needing to manually put it on EC2 yourself https://docs.prefect.io/core/idioms/file-based.html https://docs.prefect.io/orchestration/execution/storage_options.html#non-docker-storage-for-containerized-environments
m
well, that is really cool!
thanks a lot for all your time, I really appreciate it
2 Views
Open in Slack
PreviousNext
Powered by