https://prefect.io
Join Slack
so flows work locally but `__file__` is not defined when prefect cloud runs the flow. Any idea what ...
# ask-community
r
so flows work locally but 
__file__
is not defined when prefect cloud runs the flow. Any idea what the issue is?
@Phillip Kelly
so it sounds like 
__file__
is not useable with GitHub storage? https://github.com/PrefectHQ/prefect/issues/3529 What about other local assets that the flow needs to load that are in the github repo (e.g. local .sql files). Will they even be accessible to the flow, is prefect not checking out the repo when it runs the flow?
what about relative imports from other modules in my github repo? does the agent not checkout the entire github repository when running my flow?
unbelievable, I guess it's not supported at all. I really don't understand the reasoning behind most of the decisions prefect makes.. https://github.com/PrefectHQ/prefect/issues/4574
k
Yes the Github Storage only gets the Flow file. We know this not user friendly so it is being re-designed for Prefect 2.0. For Prefect 1.0, you need to package the dependencies in a container.
Stuff like SQL files can be achieved with Git Storage here
r
is there any way to make the Git Storage use 
GITHUB_ACCESS_TOKEN
PAT env var? or does it have to use a prefect secret or ssh privkey?
k
I think no. You need to use a Secret or the ssh key
r
ok, we switched to Git storage and now 
__file__
is correct e.g. 
/tmp/tmpxy6wy7bw/cureatr/flows/...
- but we get 
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpxy6wy7bw/cureatr/flows/patient_funnel/patient_funnel.sql'
when trying to open a file relative to 
__file__
- that path seems correct. Do we have to do something specific to make Git storage clone the entire repo?
k
On that, Prefect will clone the repo, load in the SQL, and then delete the files so it has to be read in outside of tasks when the script is ran
r
ugh, ok. You guys sure don't make anything easy...
k
Yeah we know Git storage can’t be the same form for sure for 2.0
10 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by