j

    Jelle Vegter

    9 months ago
    Hi all, I might be missing it but, is there a way to add overwrite=True to the Prefect Azure BlobStorageUpload method?
    Anna Geller

    Anna Geller

    9 months ago
    @Jelle Vegter I’m no Azure expert, but object storage (S3, GCS) normally works this way: if you upload a specific file for the first time, it will create an object storage object in a specified bucket (or for Azure: container) and location. If you redo the operation, the existing object (file) will be replaced by the new one. This means that there is no need to pass something like overwrite=True. If an object already exists, it will be automatically overridden.
    j

    Jelle Vegter

    9 months ago
    In Azure this is not the case. This is the error: azure.core.exceptions.ResourceExistsError: The specified blob already exists. RequestId:5bef0cf1-b01e-002e-6
    In the Azure method: blob_client.upload_blob you need to pass overwrite=True to do it. Might be a good addition to the Prefect module?
    Anna Geller

    Anna Geller

    9 months ago
    I see. Azure’s blob storage seems to be really “special” 🙂 Feel free to submit a PR. Alternatively, you can use your own custom task, or leave blob_name empty - then the file will be stored with a random uuid as name (this wouldn’t overwrite, but create a new file each time).
    Kevin Kho

    Kevin Kho

    9 months ago
    You are right. You’d need to change this like to take in
    overwrite=True
    for now. We can open an issue though
    j

    Jelle Vegter

    9 months ago
    Thanks! Untill then I'll continue to use my own functions