https://prefect.io logo
Title
o

Oscar Björhn

08/03/2022, 8:15 AM
Is anyone else on 2.0.2 yet? I'm getting an interesting error when I try to deploy using a remote storage block (seems to work with a local file system). Code: prefect deployment build orchestration/flows/test_cleansed.py:default -n test_cleansed -t dev --storage-block azure/azure-dev --infra docker-container -o test_cleansed-deployment.yaml Error: TypeError: put_directory() got an unexpected keyword argument 'ignore_file' on line 470 of cli/deployment.py Judging by the source code in Github, it seems like the overload of put_directory() for the Azure storage block specifically omits the ignore_file argument. Perhaps these two features were not tested together, unless I'm missing something. I'll give people an hour to say that I'm doing something wrong, otherwise I'll create an issue (or a PR, might be an easy fix).
1
a

Anna Geller

08/03/2022, 10:17 AM
thanks a lot, I will try to reproduce
m

Mathijs Carlu

08/03/2022, 10:18 AM
I'm able to reproduce
o

Oscar Björhn

08/03/2022, 10:28 AM
Yeah, I will push a PR soon, just having issues verifying that my code fix works. But it should. :)
a

Anna Geller

08/03/2022, 10:29 AM
same here, I can do a PR myself, don't worry - we didn't synced on the block creation
thanks to you both for reporting
o

Oscar Björhn

08/03/2022, 10:30 AM
Alright, I suppose the solution is pretty obvious so there is no point in me doing it. No problem!
Not sure if I should post a follow-up question here, but I'll try: After my fix, my deployment seems to hang after uploading all files with "normal" names, folders and files starting with . never go anywhere. It is possible that the only reason files starting with . aren't uploaded is because it goes for folders after dealing with the normal-name files. Do you know if this has been tested with subfolders?
a

Anna Geller

08/03/2022, 10:38 AM
yup, I believe if you create a child folder and move everything there, it will work fine
looking at the same that somehow with azure it otherwise tries to upload the parent of it, not yet 100% sure
o

Oscar Björhn

08/03/2022, 10:42 AM
Alright, cool. I'll keep running some more experiments as well, see if I can find any more issues. One final question, hope you don't mind: Was this tested with a normal blob storage account or one with adsl gen 2 (aka hierarchical namespace) enabled? I've tried both and I seem to get further uploading files to the normal blob storage account, but since I can't complete an upload on either it's hard to know for sure. Edit: Nevermind, pretty sure it's meant to be a normal blob account.
🙌 1
:thank-you: 1
a

Anna Geller

08/03/2022, 10:47 AM
confirmed, a normal one, but adsl gen 2 should work too with adlfs
o

Oscar Björhn

08/03/2022, 10:51 AM
Okay. With adsl gen 2 I get a nasty "maximum recursion depth exceeded" message after a few minutes, combined with infinite "container does not exist" errors. Doesn't matter to me, switching back to blob, but maybe it'll be useful for you. Maybe not. 🙂
a

Anna Geller

08/03/2022, 11:18 AM
oh wow, Azure is truly exceptional 😄
o

Oscar Björhn

08/03/2022, 11:21 AM
Oh awesome, you fixed it! I will try your code out locally. Azure is... special. I am starting to see why a lot of data tech companies such as yours aren't always excited about supporting it from day 1.
🙌 1
"Successfully uploaded 61 files". Looks great, thanks for the quick fix!
🙌 2
:thank-you: 1