Hi all, I might be missing it but, is there a way ...
# ask-communityj
Jelle Vegter
12/15/2021, 1:28 PMHi all, I might be missing it but, is there a way to add overwrite=True to the Prefect Azure BlobStorageUpload method?
a
Anna Geller
12/15/2021, 1:38 PM
@Jelle Vegter I’m no Azure expert, but object storage (S3, GCS) normally works this way: if you upload a specific file for the first time, it will create an object storage object in a specified bucket (or for Azure: container) and location.
If you redo the operation, the existing object (file) will be replaced by the new one. This means that there is no need to pass something like overwrite=True. If an object already exists, it will be automatically overridden.
j
Jelle Vegter
12/15/2021, 2:12 PMIn Azure this is not the case. This is the error: azure.core.exceptions.ResourceExistsError: The specified blob already exists.
RequestId:5bef0cf1-b01e-002e-6
Jelle Vegter
12/15/2021, 2:12 PMIn the Azure method: blob_client.upload_blob you need to pass overwrite=True to do it. Might be a good addition to the Prefect module?
a
Anna Geller
12/15/2021, 2:17 PM
I see. Azure’s blob storage seems to be really “special” 🙂
Feel free to submit a PR. Alternatively, you can use your own custom task, or leave blob_name empty - then the file will be stored with a random uuid as name (this wouldn’t overwrite, but create a new file each time).
k
Kevin Kho
12/15/2021, 2:23 PM
You are right. You’d need to change this like to take in
overwrite=TrueKevin Kho
12/15/2021, 2:27 PM
For tracking
j
Jelle Vegter
12/15/2021, 3:29 PMThanks! Untill then I'll continue to use my own functions
3 Views