Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

image.png

so flows work locally but `__file__` is not defined when prefect cloud runs the flow. Any idea what the issue is?

so it sounds like `__file__` is not useable with GitHub storage? <https://github.com/PrefectHQ/prefect/issues/3529>
What about other local assets that the flow needs to load that are in the github repo (e.g. local .sql files). Will they even be accessible to the flow, is prefect not checking out the repo when it runs the flow?

what about relative imports from other modules in my github repo? does the agent not checkout the entire github repository when running my flow?

unbelievable, I guess it's not supported at all. I really don't understand the reasoning behind most of the decisions prefect makes.. <https://github.com/PrefectHQ/prefect/issues/4574>

Yes the Github Storage only gets the Flow file. We know this not user friendly so it is being re-designed for Prefect 2.0. For Prefect 1.0, you need to package the dependencies in a container.

Stuff like SQL files can be achieved with Git Storage <https://docs.prefect.io/orchestration/flow_config/storage.html#loading-additional-files-with-git-storage|here>

is there any way to make the Git Storage use `GITHUB_ACCESS_TOKEN` PAT env var? or does it have to use a prefect secret or ssh privkey?

I think no. You need to use a Secret or the ssh key

ok, we switched to Git storage and now `__file__` is correct e.g. `/tmp/tmpxy6wy7bw/cureatr/flows/...` - but we get `FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpxy6wy7bw/cureatr/flows/patient_funnel/patient_funnel.sql'` when trying to open a file relative to `__file__` - that path seems correct. Do we have to do something specific to make Git storage clone the entire repo?

On that, Prefect will clone the repo, load in the SQL, and then delete the files so it has to be read in outside of tasks when the script is ran

ugh, ok. You guys sure don't make anything easy...

Yeah we know Git storage can’t be the same form for sure for 2.0