https://prefect.io
Join Slack
Hello, I get a `FileNotFoundError, No such file or directory` while using Git storage to a private g...
# ask-community
g
Hello, I get a 
FileNotFoundError, No such file or directory
while using Git storage to a private git server. I set Secret as ENV and specify 
git_token_secret_name
. Does this mean that the agent can connect to the repository but don't find the Flow python file?
k
Hey @Gaylord Cherencey, that’s my interpretation as well yep
g
My code is stored as such in github
And I defined the storage like such:
Copy code
flow.storage = Git(repo="<my_org>/dt-prefect-demo",
                   flow_path="/examples/flows-composition/etl.py",
                   branch_name='PAI-3025',
                   git_token_secret_name='GITHUB',
                   repo_host='<my_org_server_address>')
Am I missing something here?
k
I’m not seeing anything immediately wrong here. I would maybe look into if this storage is picking up the right branch?
g
By looking at the agent logs?
k
I would honestly just try to see if the issue persists in master. I’m not sure the debug logs would help here.
Also, this is a lot faster if you debug with the 
GitStorage
class directly here . You can try something like:
Copy code
test = Git(repo="<my_org>/dt-prefect-demo",
                   flow_path="/examples/flows-composition/etl.py",
                   branch_name='PAI-3025',
                   git_token_secret_name='GITHUB',
                   repo_host='<my_org_server_address>')

test.get_flow()
g
Ok thanks I will give this a try 🙂
Hey @Kevin Kho followup on this. I did manage to reproduce the error on local but copying the GitStorage class. When I do the 
get_flow()
I still have 
ValueError: Flow is not contained in this Storage
but when I do 
add_flow
I get the 
FileNotFoundError
. I fixed this by changing this line in the storage class to 
'/'.join([temp_repo.temp_dir.name, self.flow_path])
but it doesn't change anything when I push the flow to Prefect Cloud I still end up with the file not found. Any suggestions from here?
z
Hi @Gaylord Cherencey can you try removing the leading 
/
from 
flow_path
? Providing an absolute path here will force python to read from that path instead of the temporary directory
Copy code
os.path.join(os.getcwd(), '/my/dir') # returns 'my/dir'

os.path.join(os.getcwd(), 'my/dir') # returns /current/path/and/then/my/dir
g
@Zach Angell you are my saviour. My final code look like this and run successfully. Silly mistake on my part 🤦‍♂️
Copy code
from prefect.storage import Git
from prefect import context

storage = Git(repo="<org>/<repo>",
                   flow_path="examples/flows-composition/etl.py",
                   branch_name='<branch_name>',
                   git_token_secret_name='GITHUB',
                   repo_host='<githib_hostname>',
                   stored_as_script=True)

with Flow("ETL", run_config=KubernetesRun(image="prefecthq/prefect:latest-python3.7")) as flow:
    e = extract()
    t = transform(e)
    l = load(t)

    flow.storage = storage

flow.register(project_name='project-a')
I was wondering is 
stored_as_script
parameter mandatory?
🚀 1
z
Nice! I don't think it's necessary for Git storage
g
9:00 AM in New Zealand what a way to start the day by finally see some green
🚀 1
14 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by