https://prefect.io logo
Join Slack
Powered by
I found a bug introduced since v0.4.7: How to rep...
# prefect-aws
a
I found a bug introduced since v0.4.7: How to reproduce: • prefect worker is running with prefect-aws of v0.4.6 • prefect on client side use v0.4.7 or newer • Having flow runs persists result to S3 bucket Error: • Fetch flow run result throwing error of 
TypeError: unhashable type: 'dict'
My hunch is problem is raised after one of these PRs: • #369 • #373 • #375 This is quite a breaking change and prevent me from upgrading to latest version. Can someone take a look into this?
k
thanks for raising this, I'll take a look
🙌 1
a
Thank you, @Kevin Grismore Look forward to hearing from you regarding the updates 👍
k
@Anh Pham I wasn't able to reproduce this using 
prefect-aws
0.4.10
would you be able to share the entire stack trace and what version of prefect-aws you have installed currently?
a
@Kevin Grismore It’s actually this error message:
Copy code
''dict' object has no attribute 'get_params_override''."
I was using prefect 2.16.0 with prefect-aws 0.4.6 on hosted server
And prefect-aws 0.4.10 with prefect 2.16.0 in local to fetch persisted result.
I think it’s related to this line of code.
As in remote, I have prefect hosted in ECS, so I don’t need to specify 
aws_client_parameters
for worker or server.
k
but the 
aws_client_parameters
field should be of type 
AwsClientParameters
which has the 
get_params_override
method
even if you don't provide anything, it should make one by default
you're using an 
S3Bucket
block, right?
a
Yes. I specify that for result_storage.
k
and you have an 
AwsCredentials
block nested in it?
a
With 
bucket_name
and 
bucket_folder
No, I don’t provide 
credentials
As ECS task has the execution role to load it
k
okay, but the S3Bucket class creates a default credentials object for the credentials field
so it doesn't need to be a saved block, even if you have the credentials field empty in your S3 bucket block it should still create all the right nested fields
would you be ok sharing what your S3 bucket block looks like?
Copy code
>>> from prefect_aws import S3Bucket
>>> bucket = S3Bucket(bucket_name="my-bucket",bucket_folder="a-folder")
>>> bucket.credentials.aws_client_parameters
AwsClientParameters(api_version=None, use_ssl=True, verify=True, verify_cert_path=None, endpoint_url=None, config=None)
>>> bucket.credentials.aws_client_parameters.get_params_override()
{'use_ssl': True, 'verify': True}
a
Yeah. Let me try to pull it out from db what’s saved there.
@Kevin Grismore Here’s the stack trace
Copy code
Traceback (most recent call last):
  File "***/get_flow_run.py", line 60, in get_flow_run
    await flow_run.state.result(fetch=True)  # type: ignore[call-overload]
  File "***/.venv/lib/python3.10/site-packages/prefect/states.py", line 98, in _get_state_result
    result = await state.data.get()
  File "***/.venv/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "***/.venv/lib/python3.10/site-packages/prefect/results.py", line 534, in get
    blob = await self._read_blob(client=client)
  File "***/.venv/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "***/.venv/lib/python3.10/site-packages/prefect/results.py", line 549, in _read_blob
    content = await storage_block.read_path(self.storage_key)
  File "***/.venv/lib/python3.10/site-packages/prefect_aws/s3.py", line 606, in read_path
    return await run_sync_in_worker_thread(self._read_sync, path)
  File "***/.venv/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 95, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "***/.venv/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "***/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "***/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "***/.venv/lib/python3.10/site-packages/prefect_aws/s3.py", line 614, in _read_sync
    s3_client = self._get_s3_client()
  File "***/.venv/lib/python3.10/site-packages/prefect_aws/s3.py", line 469, in _get_s3_client
    return self.credentials.get_client("s3")
  File "***/.venv/lib/python3.10/site-packages/prefect_aws/credentials.py", line 170, in get_client
    return _get_client_cached(ctx=self, client_type=client_type)
  File "***/.venv/lib/python3.10/site-packages/prefect_aws/credentials.py", line 121, in __hash__
    hash(self.aws_client_parameters),
TypeError: unhashable type: 'dict'
I installed prefect-aws v0.4.10. Is this line expected?
Here’s how 
aws_credentials
looks like
Copy code
AwsCredentials(aws_access_key_id=None, aws_secret_access_key=None, aws_session_token=None, profile_name=None, region_name=None, aws_client_parameters={'api_version': None, 'use_ssl': True, 'verify': True, 'verify_cert_path': None, 'endpoint_url': None, 'config': None})
And here’s 
aws_client_parameters
looks like
Copy code
{'api_version': None, 'use_ssl': True, 'verify': True, 'verify_cert_path': None, 'endpoint_url': None, 'config': None}
It’s a 
dict
and can’t be hashed 😕
k
are you manually instantiating it like that?
where is this coming from?
a
No, I didn’t init it manually. It is read directly from db. As you can see in stack trace, I was using flow_run.state.result(fetch=True)
I just added few log lines to the code and provide you the result
I can try to temporarily remove that hash to see if it works
If I replace hash line with 0, I could see that it stuck at a different point
Copy code
Traceback (most recent call last):
  File "***/get_flow_run.py", line 60, in get_flow_run
    await flow_run.state.result(fetch=True)  # type: ignore[call-overload]
  File "***/.venv/lib/python3.10/site-packages/prefect/states.py", line 89, in _get_state_result
    result = await state.data.get()
  File "***/.venv/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "***/.venv/lib/python3.10/site-packages/prefect/results.py", line 534, in get
    blob = await self._read_blob(client=client)
  File "***/.venv/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "***/.venv/lib/python3.10/site-packages/prefect/results.py", line 549, in _read_blob
    content = await storage_block.read_path(self.storage_key)
  File "***/.venv/lib/python3.10/site-packages/prefect_aws/s3.py", line 580, in read_path
    return await run_sync_in_worker_thread(self._read_sync, path)
  File "***/.venv/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 95, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "***/.venv/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "***/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "***/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "***/.venv/lib/python3.10/site-packages/prefect_aws/s3.py", line 588, in _read_sync
    s3_client = self._get_s3_client()
  File "***/.venv/lib/python3.10/site-packages/prefect_aws/s3.py", line 456, in _get_s3_client
    return self.credentials.get_client("s3")
  File "***/.venv/lib/python3.10/site-packages/prefect_aws/credentials.py", line 173, in get_client
    return _get_client_cached(ctx=self, client_type=client_type)
  File "***/.venv/lib/python3.10/site-packages/prefect_aws/credentials.py", line 53, in _get_client_cached
    **ctx.aws_client_parameters.get_params_override(),
AttributeError: 'dict' object has no attribute 'get_params_override'
Is this something related to block init to parse 
aws_client_parameters
from dict?
k
can you try creating a new S3Bucket block?
a
Yes. Let me try out.
Seems like it’s working fine.
k
so, I think maybe make a new one and save that, and load the new one as your result storage. I think the properties on the one you're using are messed up somehow
a
Oh, I think I might found the issue. I have my prefect server running since Sep 2023 and it’s registered in db with old version of block schema which uses 0.4.1 version. Do you think it’s an issue as when prefect server loads the storage type, it might use block schema reference?
Also, my setup with result persistence made prefect server save an anonymous block every time flow run finish.
I guess this is the related thread
j
I have the same problem using Prefect cloud 
AttributeError: 'dict' object has no attribute 'get_params_override'
Did you manage to make it work ? I'm using an S3Bucket block but I still get the error even with a brand new block
p
@Jean-Michel Provencher did you figure it out? I am encountering the same issue right now, using prefect-aws v. 0.4.14 and a S3Bucket block I created via the UI
a
@Kevin Grismore Actually, I found the issue. It’s the order change here between Union types at this release. Do you remember why we need this change? Is there any other solution to make it work?
What I tried and succeeded was switching the order of Union to be 
Union[AwsCredentials, MinIOCredentials]
it works like charm.
I’m not sure why but with current codebase, the 
aws_client_parameters
was not loaded to 
AwsClientParameters
but kept in 
dict
.
k
@Anh Pham can you provide an exact set of reproduction steps for the issue you're encountering? the union order change was made to fix an issue with minio, but maybe it had an unintended side effect
a
https://github.com/PrefectHQ/prefect/pull/13819 I think it’s now fixed with this PR @Kevin Grismore
4 Views
Open in Slack
PreviousNext
Powered by