https://prefect.io logo
#prefect-community
Title
# prefect-community
r

Riccardo Tesselli

07/28/2022, 12:09 PM
hi all, I’ve upgraded to Prefect 2.0.0, now how can I define a block for Azure blob storage? I just see S3 and GCP blocks…
1
a

Anna Geller

07/28/2022, 12:22 PM
it's not released yet - Azure has a painful API that handles everything differently - but it's in the works -- follow #announcements
💩 1
upvote 3
azure 1
g

Geoffrey Greenleaf

07/28/2022, 2:27 PM
Currently working on a flow that involves blob storage. I noticed the same thing and was just going to create a secret for it and then create the blob client manually as documented by Microsoft
r

Riccardo Tesselli

07/28/2022, 2:29 PM
I think that a quick workaround is to use remote file system and enable Azure Data Lake Storage Gen2 on the storage account
s

Seth Coussens

07/29/2022, 6:26 PM
@Riccardo Tesselli have you proven out this work around? Do you have a basic example? Thanks!
a

Anna Geller

07/30/2022, 7:11 PM
just FYI - we'll probably add that in the next release https://github.com/PrefectHQ/prefect/pull/6141
🙌 3
❤️ 1
b

Ben

07/31/2022, 3:47 PM
This is great. I was just about to ask about an Azure storage block, in addition to S3 and GC. It looks like Azure could be done with RemoteFileSystem, but seems a decent amount more involved than the S3/GC convenience blocks.
🙌 1
r

Riccardo Tesselli

08/01/2022, 7:39 AM
@Seth Coussens I haven’t experimented with RemoteFileSystem and data lake, at the end I went through another path (putting the code and dependencies in a docker container)
s

Seth Coussens

08/01/2022, 3:30 PM
@Riccardo Tesselli that's how we are doing it in Prefect 1.0, but I'm struggling with the new yaml format for running the container and the code in the container in Prefect 2.0. Do you happen to have an example of running the container and flow with the code stored in the container rather than in a storage block?
Also, for the record the RemoteStorage does not work with Azure Storage, because the fsspec support for Azure requires an adls package which isn't supplied in the current prefect RemoteStorage script. It just throws an error about needing the external package.
a

Anna Geller

08/01/2022, 6:08 PM
looks like the PR will be merged soon
and yes, you would need to install adlfs separately https://github.com/fsspec/adlfs
s

Seth Coussens

08/01/2022, 7:46 PM
I see, so I had to import it into the environment where I run the prefect commands from... thanks! I'll give this a shot for my own education while I wait for the merge.
🙌 1
r

Riccardo Tesselli

08/02/2022, 7:38 AM
@Seth Coussens we built a docker image with the dependencies installed and the prefect code installed in a
/app
folder, then set within the container a folder
/run
to be the
WORKDIR
of the image. The most important part is to set a local file system block in the deployment of the flow, so I’ve created a local filesystem block in Prefect cloud pointing to
/app
, then tweaked the values under
storage
of the deployment, making sure to point to the file system block (setting
_block_document_id
to point to the block’s id that you can find in the URL when landing on the block’s page)
🙌 2
a

Anna Geller

08/02/2022, 11:00 AM
thanks for sharing your solution to this @Riccardo Tesselli
63 Views