Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

hi all, I’ve upgraded to Prefect 2.0.0, now how can I define a block for Azure blob storage? I just see S3 and GCP blocks…

it's not released yet - Azure has a painful API that handles everything differently - but it's in the works -- follow <#CKNSX5WG3|>

Currently working on a flow that involves blob storage.  I noticed the same thing and was just going to create a secret for it and then create the blob client manually as documented by Microsoft

I think that a quick workaround is to use remote file system and enable Azure Data Lake Storage Gen2 on the storage account

<@U02QJPN348P> have you proven out this work around? Do you have a basic example? Thanks!

just FYI - we'll probably add that in the next release <https://github.com/PrefectHQ/prefect/pull/6141>

This is great. I was just about to ask about an Azure storage block, in addition to S3 and GC. It looks like Azure could be done with RemoteFileSystem, but seems a decent amount more involved than the S3/GC convenience blocks.

<@U021GR5E5QQ> I haven’t experimented with RemoteFileSystem and data lake, at the end I went through another path (putting the code and dependencies in a docker container)

<@U02QJPN348P> that's how we are doing it in Prefect 1.0, but I'm struggling with the new yaml format for running the container and the code in the container in Prefect 2.0.

Do you happen to have an example of running the container and flow with the code stored in the container rather than in a storage block?

Also, for the record the RemoteStorage does not work with Azure Storage, because the fsspec support for Azure requires an adls package which isn't supplied in the current prefect RemoteStorage script. It just throws an error about needing the external package.

and yes, you would need to install adlfs separately <https://github.com/fsspec/adlfs>

I see, so I had to import it into the environment where I run the prefect commands from... thanks! I'll give this a shot for my own education while I wait for the merge.

<@U021GR5E5QQ> we built a docker image with the dependencies installed and the prefect code installed in a `/app` folder, then set within the container a folder `/run` to be the `WORKDIR` of the image. The most important part is to set a local file system block in the deployment of the flow, so I’ve created a local filesystem block in Prefect cloud pointing to `/app` , then tweaked the values under `storage` of the deployment, making sure to point to the file system block (setting `_block_document_id` to point to the block’s id that you can find in the URL when landing on the block’s page)

thanks for sharing your solution to this <@U02QJPN348P>