https://prefect.io logo
Powered by
Title
m

Matias Godoy

09/02/2020, 1:50 PM
Hi guys! I'm having some troubles with a flow that was working well until yesterday: I added an import that I had forgotten, and registered the flow again. I'm using Prefect Cloud and I have a couple of agents in an EC2 instance. Now when I try tu run the flow I get a failed run with the message: 
Last State Message
[2 Sep 2020 3:24pm]: Failed to load and execute Flow's environment: FileNotFoundError(2, 'No such file or directory')
I don't get which file is the one that can't be found
j

josh

09/02/2020, 1:54 PM
Hi @Matias Godoy can you provide an example of what your flow's storage looks like and the import that you had changed to include?
m

Matias Godoy

09/02/2020, 1:55 PM
That is the thing, I'm not specifying any storage option, so my guess is that it's using the default one
I had not changed this flow in a few months so maybe something has changed in the latest versions
let me share the basics of my flow:
j

josh

09/02/2020, 1:55 PM
Is the import that you changed a local module or something that is pip installable?
m

Matias Godoy

09/02/2020, 1:56 PM
oh, the import was a 
from time import sleep
nothing fancy
j

josh

09/02/2020, 1:56 PM
Ah so the import wouldn't be it then. There's a good chance that since the flow is originally from an older version of prefect something could have changed 🙂
m

Matias Godoy

09/02/2020, 1:57 PM
nice, let me show you what I have right now
import prefect
from prefect import Flow, Parameter
from prefect.tasks.control_flow.conditional import merge
from prefect.tasks.control_flow.case import case

def main() -> None:
    with Flow('Full inspection') as flow:
        env = Parameter('env', default='DEV', required=False)
        user_token = Parameter('user_token')
        inspection_id = Parameter('inspection_id')
        is_video = Parameter('is_video', default=False, required=False)

        # some tasks

    # Register the flow in Prefect Cloud
    flow.register(project_name='Inspect')

if __name__ == '__main__':
    main()
that's basically my flow, and by running that script I register it in Cloud
I never specify storage or environment (to be honest I don't know what those are)
do you think my flow is missing something?
j

josh

09/02/2020, 2:03 PM
FWIW I am able to register and “run” that flow with no issue 🤔
Oh I think I know what’s happening
m

Matias Godoy

09/02/2020, 2:04 PM
I tried also with an agent running in my local and it works
but the agents running in the EC2 instance pick it up and it fails 😞
j

josh

09/02/2020, 2:04 PM
Yeah your flow doesn’t have any storage specified therefore it is defaulting to Local storage and your agent running on EC2 can’t run the flow because it doesn’t have access to the flow file
m

Matias Godoy

09/02/2020, 2:04 PM
okay! that's a start
j

josh

09/02/2020, 2:05 PM
Either you’ll have to move that flow to your EC2 instance manually in order to use local storage or you should look into using some of the other flow storage types https://docs.prefect.io/orchestration/execution/storage_options.html
m

Matias Godoy

09/02/2020, 2:05 PM
I have the flow in the EC2 instance
because I have it in github, and I have cloned the repo there
and I start the agents like this: 
prefect agent start --label inspect-agent-2 --label prod --token aaa -p /home/ubuntu/inspect-prefect-orchestrator/flows/inspection
j

josh

09/02/2020, 2:07 PM
Can you confirm that when you register the flow it is stored in that path that you are setting on the agent?
m

Matias Godoy

09/02/2020, 2:07 PM
let me check
j

josh

09/02/2020, 2:08 PM
Otherwise the default local storage flow location is in 
~/.prefect/flows/
m

Matias Godoy

09/02/2020, 2:08 PM
when I register the flow nothing changes in that directory. I mean, it contains the python files that define the flow
but not the binary version
the weirdest thing is that it was working until I register a new flow yesterday
j

josh

09/02/2020, 2:10 PM
Yeah that is weird! Would you mind opening an issue on the repo with this information? Better to triage there than in this thread 🙂
Also one more thing, could you try running your agent with the 
--show-flow-logs
flag to see more of the traceback from where the error is raised?
m

Matias Godoy

09/02/2020, 2:12 PM
sure! let me try that
ok, this looks more like it:
[2020-09-02 14:17:23,189] INFO - agent | Found 1 flow run(s) to submit for execution.

2020-09-02 14:17:23,873 DEBG 'prefect-agent-1' stdout output:
[Errno 2] No such file or directory: '/Users/matto/.prefect/flows/full-inspection.prefect'

2020-09-02 14:17:23,925 DEBG 'prefect-agent-1' stdout output:
Traceback (most recent call last):
  File "/usr/bin/prefect", line 11, in <module>
    sys.exit(cli())
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/prefect/cli/execute.py", line 80, in cloud_flow
    raise exc
  File "/home/ubuntu/.local/lib/python3.6/site-packages/prefect/cli/execute.py", line 69, in cloud_flow
    flow = storage.get_flow(storage.flows[flow_data.name])
  File "/home/ubuntu/.local/lib/python3.6/site-packages/prefect/environments/storage/local.py", line 77, in get_flow
    return prefect.core.flow.Flow.load(flow_location)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/prefect/core/flow.py", line 1335, in load
    with open(str(fpath), "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/matto/.prefect/flows/full-inspection.prefect'

2020-09-02 14:17:25,027 DEBG 'prefect-agent-1' stdout output:
[2020-09-02 14:17:25,027] INFO - agent | Process PID 1464 returned non-zero exit code
I can see that the agents are looking for the flow binary in a directory that is in my local computer, but they are in an EC2 instance
j

josh

09/02/2020, 2:20 PM
Are you registering the flow on EC2 or on your local computer?
m

Matias Godoy

09/02/2020, 2:21 PM
I thought I was registering in Prefect Cloud
I see the problem now
j

josh

09/02/2020, 2:21 PM
Yes you’re using Prefect Cloud but on which machine are you calling 
.register
?
Register is saving the flow with Local storage to that machine’s local file system
m

Matias Godoy

09/02/2020, 2:22 PM
in my local
right
j

josh

09/02/2020, 2:22 PM
Great, so either register it on EC2 or use one of the other cloud-based storage options 🙂
m

Matias Godoy

09/02/2020, 2:22 PM
so if I register it in EC2 it shoud work
j

josh

09/02/2020, 2:22 PM
Yep!
m

Matias Godoy

09/02/2020, 2:22 PM
cool, I see
sorry for all the fuzz!
j

josh

09/02/2020, 2:22 PM
No problem!
m

Matias Godoy

09/02/2020, 2:23 PM
I thought (I don't know why) that when you registered a flow it was uploaded to Prefect Cloud servers
it's clear now 🙂
j

josh

09/02/2020, 2:23 PM
Nope Prefect Cloud and Prefect Server never store any actual flow code!
m

Matias Godoy

09/02/2020, 2:24 PM
nice, I'll try that and let you know how it went
ok, different error now: 
ValueError: Flow is not contained in this Storage
j

josh

09/02/2020, 2:38 PM
Interesting, did you just run the register script and agent on EC2?
m

Matias Godoy

09/02/2020, 2:40 PM
correct
and I start the agents like this: 
prefect agent start --label inspect-agent-2 --label prod --token [my_token] -p /home/ubuntu/inspect-prefect-orchestrator/flows/inspection
that directory contains the python files for the flow
okay, I can see that I had prefect 12.0
I just updated to 13.5. We'll see if it helps
it worked! 👏
I had to re-login to Prefect Cloud after upgrading, but it worked
j

josh

09/02/2020, 3:00 PM
Awesome!
m

Matias Godoy

09/02/2020, 3:00 PM
thanks a lot for your help
I'm a little concerned that something this will happen every time a new version of prefect comes out
maybe I'll have to automate the 
pip install --upgrade prefect
command to keep the agents up to date
j

josh

09/02/2020, 3:04 PM
When working with the Local agent you’re ultimately subject to the environment that the agent runs in which has the potential to cause issues with versions. That’s why a lot of users opt into using Docker storage because you are subject to the static version inside the container and the flow should always work regardless of newer releases
m

Matias Godoy

09/02/2020, 3:05 PM
oh, that sounds like a cool alternative
so, could I use the same agents i'm running in EC2, but using Docker storage and forget about upgrading them forever?
and every time I make a change in a flow, will I have to SSH into the EC2 instance, pull the changes from github and register the new flow from there, right?
j

josh

09/02/2020, 3:10 PM
That is one option! Or even better you could use GitHub storage in combination with a Docker image so every time you run your flow it will use whichever code you have in your git repo automatically without needing to manually put it on EC2 yourself https://docs.prefect.io/core/idioms/file-based.html https://docs.prefect.io/orchestration/execution/storage_options.html#non-docker-storage-for-containerized-environments
m

Matias Godoy

09/02/2020, 3:13 PM
well, that is really cool!
thanks a lot for all your time, I really appreciate it
#prefect-communityJoin Slack
Powered by