I take it that you need a remote filesystem block to deploy flows remotely? When I deploy with files...

Albert Wong

03/04/2023, 10:37 AM

I take it that you need a remote filesystem block to deploy flows remotely? When I deploy with filestorage, the remote prefect server is trying to load a file path that looks like a path in my local system. I have github, but haven't been able to figure out why the deploy does not upload files properly. I get the following warning:

Copy code

Deployment storage GitHub(repository='<'redacted'>, reference=None, 
access_token=SecretStr('**********'), include_git_objects=True) does not have upload capabilities; no files uploaded.  Pass --skip-upload to suppress 
this warning.

This is after I execute the following deploy command:

Copy code

prefect deployment build -n sync_repository_tables_flow -q default flows/sync_repository_tables_flow.py:sync_repository_tables_flow -a -sb github/prod-gbi-github-dataflowdeployments

Anyone know what I'm missing?

Christopher Boyd

03/06/2023, 2:40 PM

github is read only (

get-directory

is listed on the block) to retrieve storage, and is not a valid remote file store for pushing content

Christopher Boyd

03/06/2023, 2:41 PM

if you’re using a docker image, you can pass the path and entrypoint - otherwise, you’d need a file store that supports put-directory / write-path like azure, s3, gcs

Albert Wong

03/06/2023, 4:41 PM

Thanks @Christopher Boyd, this is compelling, and I'll take a look. Can I also manage different versions of python + libs through this method?

Albert Wong

03/09/2023, 2:57 AM

Hmm, looks like the docker infrastructure block cannot find the flow either. Do you mean I should build a custom image with the flows in the image? If so, is there any documentation on how to do that?

Albert Wong

03/09/2023, 3:01 AM

for reference, this is what I get when running the flow

Copy code

02:48:58.664 | INFO    | Flow run 'rare-rook' - Downloading flow code from storage at '/opt/prefect/flows'
02:48:58.668 | ERROR   | Flow run 'rare-rook' - Flow could not be retrieved from deployment.

Albert Wong

03/09/2023, 4:27 AM

actually, if github is readonly, can I just manually check in the flows into github for retrieval on the remote server? i.e. I deploy with the github storage block defined, check in the flow code, and then run?

Albert Wong

03/09/2023, 5:22 AM

oh shoot, that works! haha. There's a demo video here on the subject:

https://www.youtube.com/watch?v=Nv1eD0VDedA▾

Will Truong

03/17/2023, 4:28 AM

Thank you Albert, I am currently trying to implement this too.

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.