https://prefect.io logo
Channels
Powered by
l

Lance Cuthbert

09/28/2022, 6:09 PM
Good afternoon - starting this past Sunday morning (September 25th) we started receiving this error message when running flows. These flows have been running for several months without issue. There have been no changes to the flows, environment, versions, etc. We are running Prefect 1.0, Python 3.7 I have tried this on these images - prefecthq/prefect:0.15.13-python3.7 prefecthq/prefect:latest-python3.7 any thoughts?
👀 1
m

Mason Menges

09/28/2022, 6:41 PM
Hey @Lance Cuthbert I haven't seen this issue personally, would you be able to share a minimum example of the code that's running here in addition to the storage configuration. Also would you be able to move the error to this thread? it helps keep the main channel a bit cleaner.
l

Lance Cuthbert

09/28/2022, 6:43 PM
Error Message -
Copy code
[2022-09-28 18:02:59+0000] ERROR - prefect.S3 | Error downloading Flow from S3: deprecated() got an unexpected keyword argument 'name'
deprecated() got an unexpected keyword argument 'name'
/usr/local/lib/python3.7/site-packages/prefect/client/client.py:177: UserWarning: Client was created with an API token configured for authentication. API tokens are deprecated, please use API keys instead.
  "Client was created with an API token configured for authentication. "
Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 96, in flow_run
    raise exc
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 73, in flow_run
    flow = storage.get_flow(flow_data.name)
  File "/usr/local/lib/python3.7/site-packages/prefect/storage/s3.py", line 89, in get_flow
    obj = self._boto3_client.get_object(Bucket=self.bucket, Key=key)
  File "/usr/local/lib/python3.7/site-packages/prefect/storage/s3.py", line 219, in _boto3_client
    from prefect.utilities.aws import get_boto_client
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/aws.py", line 10, in <module>
    import boto3
  File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 29, in <module>
    import botocore.credentials
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 35, in <module>
    from botocore.config import Config
  File "/usr/local/lib/python3.7/site-packages/botocore/config.py", line 16, in <module>
    from botocore.endpoint import DEFAULT_TIMEOUT, MAX_POOL_CONNECTIONS
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 22, in <module>
    from botocore.awsrequest import create_request_object
  File "/usr/local/lib/python3.7/site-packages/botocore/awsrequest.py", line 24, in <module>
    import botocore.utils
  File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 32, in <module>
    import botocore.httpsession
  File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 28, in <module>
    from urllib3.contrib.pyopenssl import orig_util_SSLContext as SSLContext
  File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 50, in <module>
    import OpenSSL.SSL
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import SSL, crypto
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 19, in <module>
    from OpenSSL.crypto import (
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/crypto.py", line 3232, in <module>
    name="load_pkcs7_data",
TypeError: deprecated() got an unexpected keyword argument 'name'
`
we are running the Prefect agent as a docker container on an EC2 instance.
m

Mason Menges

09/28/2022, 6:58 PM
Thanks, it looks like the error is resulting from the agent not being able to retrieve the flow from storage, so I don't think the agent itself is the issue, at least it doesn't seem that way at the moment. it's difficult to say just from the error message alone. Would you be able to share the Run and Storage configuration this was registered with?
l

Lance Cuthbert

09/28/2022, 7:02 PM
script.py
script.py
This is the error message that I think might be at the root of the issue -
Copy code
UserWarning: Client was created with an API token configured for authentication. API tokens are deprecated, please use API keys instead.
  "Client was created with an API token configured for authentication. "
m

Mason Menges

09/28/2022, 8:17 PM
It definitely could be, though Existing tokens should still work we just don't allow new ones to be created through prefect-cloud, that said it may be beneficial authenticate to cloud using an API Key instead. I'll not that I was able to register and run a simple flow with S3 storage in this format
Copy code
storage = S3(bucket="********bucket", key="test_flow") 

@task
def some_task():
    print("test")


with Flow(name="test_flow", 
            storage=storage, 
            run_config=UniversalRun(env=dict(AWS_ACCESS_KEY_ID=Secret("aws_personal_key_id").get(),
                      AWS_SECRET_ACCESS_KEY=Secret("aws_personal_api_key").get()))) as flow:

    some_task()

flow.register(project_name="Local-test")
with prefect version 1.3.0
l

Lance Cuthbert

09/28/2022, 8:27 PM
I have 4 flows that were all working without issue on this agent for months. Then they all started to fail on the exact same day. They ran correctly at 12:00pm on Sunday (9/25). When they tried to run again at 3pm Sunday (9/25) - they all started to fail. There were no code updates on our side, no permission change, nothing. Could something have expired or was a change introduced on Prefect side?
I took a flow and set it to pull the 1.3.0-python3-7 docker image. This is the output from my attempt to try and run the flow -
Copy code
prefect agent docker start -t EdYAD_SW9wvRsPfsf42yQQ -n DockerAgent -l <http://prefect.segmint.com|prefect.segmint.com> --volume /opt/segmint:/opt/segmint --show-flow-logs

 ____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2022-09-28 20:34:21,989] INFO - DockerAgent | Starting DockerAgent with labels ['<http://prefect.segmint.com|prefect.segmint.com>']
[2022-09-28 20:34:21,989] INFO - DockerAgent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2022-09-28 20:34:21,989] INFO - DockerAgent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2022-09-28 20:34:22,076] INFO - DockerAgent | Waiting for flow runs...
[2022-09-28 20:34:22,300] INFO - DockerAgent | Found 1 flow run(s) to submit for execution.
[2022-09-28 20:34:22,437] INFO - DockerAgent | Deploying flow run 437b9710-93ce-49ca-a014-16495d381681
[2022-09-28 20:34:22,439] INFO - DockerAgent | Pulling image prefecthq/prefect:1.3.0-python3.7...
[2022-09-28 20:34:37,925] INFO - DockerAgent | Successfully pulled image prefecthq/prefect:1.3.0-python3.7...
+pip install snowflake-connector-python
Collecting snowflake-connector-python
  Downloading snowflake_connector_python-2.8.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (21.3 MB)
Collecting oscrypto<2.0.0
  Downloading oscrypto-1.3.0-py2.py3-none-any.whl (194 kB)
Requirement already satisfied: cffi<2.0.0,>=1.9 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (1.15.1)
Requirement already satisfied: charset-normalizer<3,>=2 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (2.1.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (3.3)
Requirement already satisfied: typing-extensions<5,>=4.3 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (4.3.0)
Requirement already satisfied: requests<3.0.0 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (2.28.1)
Collecting filelock<4,>=3.5
  Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (2022.6.15)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (1.26.11)
Collecting pycryptodomex!=3.5.0,<4.0.0,>=3.2
  Downloading pycryptodomex-3.15.0-cp35-abi3-manylinux2010_x86_64.whl (2.3 MB)
Requirement already satisfied: setuptools>34.0.0 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (57.5.0)
Requirement already satisfied: pyjwt<3.0.0 in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (2.4.0)
Collecting cryptography<37.0.0,>=3.1.0
  Downloading cryptography-36.0.2-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.7 MB)
Collecting asn1crypto<2.0.0,>0.24.0
  Downloading asn1crypto-1.5.1-py2.py3-none-any.whl (105 kB)
Requirement already satisfied: pytz in /usr/local/lib/python3.7/site-packages (from snowflake-connector-python) (2022.1)
Collecting pyOpenSSL<23.0.0,>=16.2.0
  Downloading pyOpenSSL-22.1.0-py3-none-any.whl (57 kB)
Requirement already satisfied: pycparser in /usr/local/lib/python3.7/site-packages (from cffi<2.0.0,>=1.9->snowflake-connector-python) (2.21)
Installing collected packages: asn1crypto, oscrypto, filelock, pycryptodomex, cryptography, pyOpenSSL, snowflake-connector-python
  Attempting uninstall: cryptography
    Found existing installation: cryptography 37.0.4
    Uninstalling cryptography-37.0.4:
      Successfully uninstalled cryptography-37.0.4
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

pyopenssl 22.1.0 requires cryptography<39,>=38.0.0, but you'll have cryptography 36.0.2 which is incompatible.
Successfully installed asn1crypto-1.5.1 cryptography-36.0.2 filelock-3.8.0 oscrypto-1.3.0 pyOpenSSL-22.1.0 pycryptodomex-3.15.0 snowflake-connector-python-2.8.0
WARNING: You are using pip version 20.2.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 53, in flow_run
    result = client.graphql(query)
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 464, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['flow_run'], 'message': 'Missing Authorization header in JWT authentication mode', 'extensions': {'path': '$', 'code': 'invalid-headers', 'exception': {'message': 'Missing Authorization header in JWT authentication mode'}}}]
m

Mason Menges

09/28/2022, 8:38 PM
Hmm we haven't released anything to cloud 1 within the past week at least as far as I am aware, There is an ongoing intermittent issue with secrets more on that here https://prefect.status.io/, but mostly appears intermittently though it is possible this is related since the error originates from accessing the flow storage. It's possible the token may have expired but I would have expected the client to through other errors in that case. Also from the error you're receiving I'm not entirely convinced the issue isn't related to the storage for the flow, Would you be able to re-register the flow updating the storage configuration and see if that works?
l

Lance Cuthbert

09/29/2022, 4:13 PM
@Mason Menges - We figured out the issue. we use the Python snowflake-connector in our flows. The latest version in the connector has dependencies that could not be resolved and it was causing issues downstream. I republish my flows and pinned the snowflake-connector version to an older version and everything seems to be working again.
m

Mason Menges

09/29/2022, 4:16 PM
Awesome I'm glad to hear it got sorted out 😄
l

Lance Cuthbert

09/29/2022, 4:18 PM
Appreciate your help. Thanks again!!
m

Mason Menges

09/29/2022, 4:19 PM
No problem at all 😄
b

Bianca Hoch

09/29/2022, 6:02 PM
Hello @Lance Cuthbert! Would you mind sharing the version of the snowflake-connector that you pinned for this fix?
l

Lance Cuthbert

09/29/2022, 6:25 PM
Sure, the version I pinned it to is pretty old, but it was the version that we used when we originally deployed the flows -
Copy code
env={"EXTRA_PIP_PACKAGES": "snowflake-connector-python==2.4.3"}
✨ 1
gratitude thank you 1
b

Bianca Hoch

09/29/2022, 6:27 PM
Thank you!
4 Views
Powered by