https://prefect.io logo
Powered by
s

Seth Coussens

09/13/2023, 5:13 AM
Is anyone else having issues registring flows all the sudden tonight? We've started getting the following error when attempting to register flow, while this same code and method has worked flawlessly previously.
Copy code
prefect deployment build flows/downloads.py:download_eobs -n hourly -t $TAG -t $VERSION_TAG -q $QUEUE -sb $STORAGE_BLOCK/$WORKSPACE/$IMAGE_NAME -ib $INFRASTRUCTURE_BLOCK --override image=$REGISTRY/flows/$WORKSPACE/$IMAGE_NAME:$VERSION_TAG --apply
3
  prefect deployment build flows/moves.py:move_eobs -n on-demand -t $TAG -t $VERSION_TAG -q $QUEUE -sb $STORAGE_BLOCK/$WORKSPACE/$IMAGE_NAME -ib $INFRASTRUCTURE_BLOCK --override image=$REGISTRY/flows/$WORKSPACE/$IMAGE_NAME:$VERSION_TAG --apply
4
  prefect deployment build flows/non_financials.py:process_non_financial_eobs -n on-demand -t $TAG -t $VERSION_TAG -q $QUEUE -sb $STORAGE_BLOCK/$WORKSPACE/$IMAGE_NAME -ib $INFRASTRUCTURE_BLOCK --override image=$REGISTRY/flows/$WORKSPACE/$IMAGE_NAME:$VERSION_TAG --apply
5
  prefect deployment build flows/sharefile_upload.py:sharefile_upload_eobs -n on-demand -t $TAG -t $VERSION_TAG -q $QUEUE -sb $STORAGE_BLOCK/$WORKSPACE/$IMAGE_NAME -ib $INFRASTRUCTURE_BLOCK --override image=$REGISTRY/flows/$WORKSPACE/$IMAGE_NAME:$VERSION_TAG --apply
6
  prefect deployment build flows/emerginet_upload.py:emerginet_upload_eobs -n on-demand -t $TAG -t $VERSION_TAG -q $QUEUE -sb $STORAGE_BLOCK/$WORKSPACE/$IMAGE_NAME -ib $INFRASTRUCTURE_BLOCK --override image=$REGISTRY/flows/$WORKSPACE/$IMAGE_NAME:$VERSION_TAG --apply
7
  shell: /usr/bin/bash -e {0}
8
  env:
9
    IMAGE_NAME: import-eobs
10
    TAG: eobs
11
    WORKSPACE: *********
12
    REGISTRY: *********
13
    QUEUE: production
14
    INFRASTRUCTURE_BLOCK: docker-container/default
15
    STORAGE_BLOCK: azure/flows
16
    VERSION_TAG: 9f50fd0505d455115a01fc7e7c087bce4a1ad149
17
    PREFECT_API_KEY: ***
18
    PREFECT_API_URL: ***
19
    pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64
20
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib
21
  
21
Found flow 'download-eobs'
22
Traceback (most recent call last):
23
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
24
    return fn(*args, **kwargs)
25
An exception occurred.
26
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
27
    return call()
28
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 382, in __call__
29
    return self.result()
30
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 282, in result
31
    return self.future.result(timeout=timeout)
32
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 168, in result
33
    return self.__get_result()
34
  File "/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
35
    raise self._exception
36
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 345, in _run_async
37
    result = await coro
38
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/cli/deployment.py", line 1163, in build
39
    deployment = await Deployment.build_from_flow(
40
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/deployments/deployments.py", line 850, in build_from_flow
41
    await deployment.upload_to_storage(ignore_file=ignore_file)
42
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/deployments/deployments.py", line 673, in upload_to_storage
43
    file_count = await self.storage.put_directory(
44
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/filesystems.py", line 739, in put_directory
45
    return await self.filesystem.put_directory(
46
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/filesystems.py", line 385, in put_directory
47
    self.filesystem.put_file(f, fpath, overwrite=True)
48
  File "/home/runner/.local/lib/python3.9/site-packages/prefect/filesystems.py", line 419, in filesystem
49
    self._filesystem = fsspec.filesystem(scheme, **self.settings)
50
  File "/home/runner/.local/lib/python3.9/site-packages/fsspec/registry.py", line 288, in filesystem
51
    cls = get_filesystem_class(protocol)
52
  File "/home/runner/.local/lib/python3.9/site-packages/fsspec/registry.py", line 234, in get_filesystem_class
53
    register_implementation(protocol, _import_class(bit["class"]))
54
  File "/home/runner/.local/lib/python3.9/site-packages/fsspec/registry.py", line 269, in _import_class
55
    mod = importlib.import_module(mod)
56
  File "/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/importlib/__init__.py", line 127, in import_module
57
    return _bootstrap._gcd_import(name[level:], package, level)
58
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
59
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
60
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
61
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
62
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
63
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
64
  File "/home/runner/.local/lib/python3.9/site-packages/adlfs/__init__.py", line 3, in <module>
65
    from .spec import AzureBlobFile, AzureBlobFileSystem
66
  File "/home/runner/.local/lib/python3.9/site-packages/adlfs/spec.py", line 115, in <module>
67
    class AzureBlobFileSystem(AsyncFileSystem):
68
  File "/home/runner/.local/lib/python3.9/site-packages/adlfs/spec.py", line 222, in AzureBlobFileSystem
69
    blocksize: int = create_configuration(storage_sdk="blob").max_block_size,
70
  File "/home/runner/.local/lib/python3.9/site-packages/azure/storage/blob/_shared/base_client.py", line 415, in create_configuration
71
    config.user_agent_policy = UserAgentPolicy(sdk_moniker=kwargs.pop('sdk_moniker'), **kwargs)
72
KeyError: 'sdk_moniker'
b

Bob De Schutter

09/13/2023, 9:05 AM
Yep, I have the same issue... probably related to a release in the azure sdk for python?
have you managed to fix it?
s

Seth Coussens

09/13/2023, 3:27 PM
@Bob De Schutter I've not managed a fix yet but have confirmed that it's related to a azure-blob-storage package used by the Azure SDK package.
b

Bob De Schutter

09/13/2023, 3:55 PM
@Seth Coussens I've fixed it temporarily by pip installing azure-storage-blob==12.17.0 before installing the adlfs package on my infra block
sth like this:
Copy code
env={
        "EXTRA_PIP_PACKAGES": "azure-storage-blob==12.17.0 adlfs"
    }
when creating a DockerContainer infra block
s

Seth Coussens

09/13/2023, 7:35 PM
@Bob De Schutter that's now how we install packages but yes, that should work as well.
Powered by