https://prefect.io logo
Title
s

Sarhan

02/17/2023, 2:47 AM
Hi, need some help on troubleshooting an issue I’m facing. I’ve been trying to deploy my flow using S3 as my storage and have an EC2 instance (Linux) run my agent. I followed this tutorial btw. The agent seems to be running cause it triggers at the scheduled time but I’ve been repeatedly getting this error though:
Downloading flow code from storage at '/deployment/'

Flow could not be retrieved from deployment.
And the traceback ends with (full traceback in thread):
AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'
Running the agent locally is perfectly fine so it’s only when deployed to EC2 that I’m getting the error. I’ve checked a number of things already including referencing this thread from Brian: • EC2 is able to connect to S3 and download files (used AWS CLI here) as well as run
head_object()
(Python - boto3 here) • Reinstalled
s3fs
. My error isn’t quite the same as Brian’s but tried it anyway. • Changed my versions of
pyOpenSSL
and
cryptography
following the first answer here. • made sure
path
and
entrypoint
are present and correct (side note that my flow is quite deep into my directory) Some additional notes: • Deployed using Python
Downloading flow code from storage at '/deployment/'
09:48:55 AM
ERROR
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 268, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/prefect/deployments.py", line 175, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 476, in get_directory
    return await self.filesystem.get_directory(
  File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 320, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "/usr/local/lib/python3.10/dist-packages/prefect/filesystems.py", line 401, in filesystem
    self._filesystem = fsspec.filesystem(scheme, **self.settings)
  File "/usr/local/lib/python3.10/dist-packages/fsspec/registry.py", line 251, in filesystem
    cls = get_filesystem_class(protocol)
  File "/usr/local/lib/python3.10/dist-packages/fsspec/registry.py", line 211, in get_filesystem_class
    register_implementation(protocol, _import_class(bit["class"]))
  File "/usr/local/lib/python3.10/dist-packages/fsspec/registry.py", line 234, in _import_class
    mod = importlib.import_module(mod)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/dist-packages/s3fs/__init__.py", line 1, in <module>
    from .core import S3FileSystem, S3File
  File "/usr/local/lib/python3.10/dist-packages/s3fs/core.py", line 28, in <module>
    import aiobotocore.session
  File "/usr/local/lib/python3.10/dist-packages/aiobotocore/session.py", line 1, in <module>
    from botocore import UNSIGNED, translate
  File "/usr/local/lib/python3.10/dist-packages/botocore/translate.py", line 16, in <module>
    from botocore.utils import merge_dicts
  File "/usr/local/lib/python3.10/dist-packages/botocore/utils.py", line 34, in <module>
    import botocore.httpsession
  File "/usr/local/lib/python3.10/dist-packages/botocore/httpsession.py", line 41, in <module>
    from urllib3.contrib.pyopenssl import orig_util_SSLContext as SSLContext
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 50, in <module>
    import OpenSSL.SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 3279, in <module>
    _lib.OpenSSL_add_all_algorithms()
AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'
j

James Sopkin

02/17/2023, 2:59 AM
@Sarhan did you install s3fs on the ec2 instance?
I'm assuming you did if you followed that tutorial
s

Sarhan

02/17/2023, 3:07 AM
Yeap, I did
j

James Sopkin

02/17/2023, 3:11 AM
I remember running into this exact issue before when trying to deploy a flow with CI/CD and it was pretty frustrating. The current versions of the flow that I ran into that issue before is using cryptography==39.0.0 pyopenssl==22.0.0 but I don't know if that will solve it for you
these dependency issues can be a pain to deal with
s

Sarhan

02/17/2023, 3:17 AM
I previously downgraded my cryptography to 38.0.4 and my pyOpenSSL is 23.0.0 based on the stackoverflow recommendations. Let me try changing it to match yours and see if that does anything.
No luck @James Sopkin 😢
j

James Sopkin

02/17/2023, 3:23 AM
hmm okay. Could you try force reinstalling with
pip install --force-reinstall "cryptography==38.0.4"
s

Sarhan

02/17/2023, 3:28 AM
Still hitting the error after force reinstalling cryptography
j

James Sopkin

02/17/2023, 3:33 AM
Ahh could you upgrade the pyopenssl again?
pip install --upgrade --force-reinstall pyopenssl
I just have a feeling that your vm did update the dependency versions, because this shouldn't be happening with the latest version of pyopenssl
s

Sarhan

02/17/2023, 3:44 AM
Tried to rerun the deployed flow after force reinstalling pyopenssl (which also upgrades cryptography to 39.0.1 btw) and still hitting the error