https://prefect.io logo
j

Jens

01/16/2023, 8:43 AM
Hi All, I'm new to prefect and try to build the following. I have some Code on Bitbucket. I create a Bitbucket-Storage-Block and create a deployment which fetchs a flow from Bitbucket and run it locally. On Bitbucket there are also some test-files which I try to read. So inside my Flow I pull this specific folder with block.get_directory() so that the date persists locally. Each time I trigger this flow, the folder is re-pulled from Bitbucket. Is there the possibilty that the pull is executed only if the files on the specifiy Bitbucket-Path are newer than the previous pulled local state?
a

Ankit

01/16/2023, 2:03 PM
Hi @Jens do you have a working example of how you created the block and referenced the files in bitbucket in your flow? Really struggling, thanks.
j

Jens

01/17/2023, 1:21 PM
Hi @Ankit I create the Bitbucket-Block with the UI. I create a Repository-Block and a Credential Block. After that I load this block in my code with bitbucket_repository_block = BitBucketRepository.load("<Block-Name>") and create a deployment and link the block to ist with storage=bitbucket_repository_block
a

Ankit

01/17/2023, 2:11 PM
Thanks @Jens Do you also pass path of the flow files?
j

Jens

01/17/2023, 2:20 PM
I think this is exactly the step I am hanging on to right now. I have a python deployment-Script which deploying my stuff to the orio-server. At this deployment I specifie to use the Bitbucket Block and the path to my Flow-Script. Inside this Flow-Script I use some tasks which located on a other folder but inside the Bitbucket-Repository and i get some "Module Not Found Errors". I try to make a small Example to understand my problem in a way better
a

Ankit

01/17/2023, 2:43 PM
but you will eventually need to pass different flow scripts through that, right? If I understood correctly, you only have one script with which you are trying it out for now?
j

Jens

01/19/2023, 9:22 AM
yes thats right. I solved it now. I have the following structure
root
-->Module_A (folder)
-->Module_B (folder)
git_pull_flow.py
git_pull_flow_deployment.py
main_flow.py
main_flow_deployment.py
So, first I run the git_pull_flow that pull my complete underlying code-structure Inside this flow I run a subprocess which trigger the pip install -r requirements.txt command With this, inside my main_flow I can reference all my Python-Module and the "No Module FOund" Error is solved 🙂
3 Views