I need some troubleshooting help I created a deployment that Prefect Community #ask-community

I need some troubleshooting help. I created a dep...

Brian Roepke

01/20/2023, 7:53 PM

I need some troubleshooting help. I created a deployment that uses S3 cloud storage, and when I run it with a local agent, it works great. I'm setting up a LINUX EC2 instance and trying to get the agent to run there. I've successfully installed the libraries I need (

s3fs, boto3, prefect_aws, prefect, prefect-snowflake

). But I'm running into an error when I start the Flow as a run from Prefect Cloud. The agent will pick up the job without a problem - but I'm getting the following error: (note the blank S3 path)

Copy code

Downloading flow code from storage at ''

Flow could not be retrieved from deployment.

Full dump of the error in the thread. Could anyone think of a missing dependency or authentication step I missed? (also - if I go into

python3

and run

from prefect.filesystems import S3; s3_block = S3.load("s3-prefect"); print(s3_block)

that works just fine... )

Brian Roepke

01/20/2023, 7:53 PM

Copy code

Downloading flow code from storage at ''

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.7/site-packages/s3fs/core.py", line 112, in _error_wrapper
    return await func(*args, **kwargs)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/aiobotocore/client.py", line 358, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/engine.py", line 266, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/deployments.py", line 174, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/filesystems.py", line 469, in get_directory
    from_path=from_path, local_path=local_path
  File "/home/ec2-user/.local/lib/python3.7/site-packages/prefect/filesystems.py", line 313, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/fsspec/asyn.py", line 113, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/fsspec/asyn.py", line 98, in sync
    raise return_result
  File "/home/ec2-user/.local/lib/python3.7/site-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/home/ec2-user/.local/lib/python3.7/site-packages/fsspec/asyn.py", line 551, in _get
    rpaths = await self._expand_path(rpath, recursive=recursive)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/fsspec/asyn.py", line 751, in _expand_path
    out = await self._expand_path([path], recursive, maxdepth)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/fsspec/asyn.py", line 769, in _expand_path
    rec = set(await self._find(p, maxdepth=maxdepth, withdirs=True))
  File "/home/ec2-user/.local/lib/python3.7/site-packages/s3fs/core.py", line 777, in _find
    out = [await self._info(path)]
  File "/home/ec2-user/.local/lib/python3.7/site-packages/s3fs/core.py", line 1216, in _info
    **self.req_kw,
  File "/home/ec2-user/.local/lib/python3.7/site-packages/s3fs/core.py", line 340, in _call_s3
    method, kwargs=additional_kwargs, retries=self.retries
  File "/home/ec2-user/.local/lib/python3.7/site-packages/s3fs/core.py", line 139, in _error_wrapper
    raise err
PermissionError: Forbidden

Christopher Boyd

01/20/2023, 8:23 PM

do you ahve getWebcredentials set on the role for your ec2 instance?

Christopher Boyd

01/20/2023, 8:24 PM

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html

Christopher Boyd

01/20/2023, 8:24 PM

you should be able to try and access s3 directly from the EC2 instance using boto, if you can’t do that (completely disregarding prefect), you’re likely missing an IAM policy to allow it

Christopher Boyd

01/20/2023, 8:27 PM

specifically this errror is what indicates a role permission:

Copy code

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Brian Roepke

01/20/2023, 8:38 PM

Thanks, @Christopher Boyd!

Brian Roepke

01/20/2023, 8:38 PM

I was able to download with boto3.

Christopher Boyd

01/20/2023, 8:39 PM

Do you have the head permission on that role too though ?

Christopher Boyd

01/20/2023, 8:40 PM

Those should be separate permissions on the role

Christopher Boyd

01/20/2023, 8:42 PM

You should be able to test that with head_object() which is the error

Brian Roepke

01/20/2023, 8:44 PM

i'm checking out the IAM role now from your link. there wasn't an explicit IAM role attached at creation.

Christopher Boyd

01/20/2023, 8:44 PM

Yea, it just feels like a permissions issue - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object

Christopher Boyd

01/20/2023, 8:45 PM

whatever that flow or object is in the bucket, or in the s3 bucket itself

Brian Roepke

01/20/2023, 8:49 PM

The new IAM role I created has S3full access and EC2 full access. That should cover the permissions to the ones described in the docs.

Brian Roepke

01/20/2023, 8:58 PM

Copy code

>>> object = s3.head_object(Bucket='prefect-test-1',Key='test.txt')
>>> object
{'ResponseMetadata': {'RequestId': 'C4ZBXV4KN062ZNWP', 'HostId': 'kQPT+tnlHAp0mn3DK2vQ+nziiV6dWRhYrto7vrjERzPa+5S8U13bfqHGDs0P3ZPnnJtx2RJYzDE=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'kQPT+tnlHAp0mn3DK2vQ+nziiV6dWRhYrto7vrjERzPa+5S8U13bfqHGDs0P3ZPnnJtx2RJYzDE=', 'x-amz-request-id': 'C4ZBXV4KN062ZNWP', 'date': 'Fri, 20 Jan 2023 20:58:05 GMT', 'last-modified': 'Fri, 20 Jan 2023 20:37:24 GMT', 'etag': '"bcd7301fca81d73fff76a2ca2f4b04ff"', 'accept-ranges': 'bytes', 'content-type': 'text/plain', 'server': 'AmazonS3', 'content-length': '48'}, 'RetryAttempts': 0}, 'AcceptRanges': 'bytes', 'LastModified': datetime.datetime(2023, 1, 20, 20, 37, 24, tzinfo=tzutc()), 'ContentLength': 48, 'ETag': '"bcd7301fca81d73fff76a2ca2f4b04ff"', 'ContentType': 'text/plain', 'Metadata': {}}

🙌 1

Christopher Boyd

01/20/2023, 9:00 PM

that looks good

Brian Roepke

01/20/2023, 9:01 PM

It's still throwing the same error - I think it's because it's passing an empty string... The line at the top of the stack.

Downloading flow code from storage at ''

Christopher Boyd

01/20/2023, 9:02 PM

How is your s3 block configured / saved?

Christopher Boyd

01/20/2023, 9:02 PM

As well as how the deployment is configured?

Brian Roepke

01/20/2023, 9:04 PM

Block...

Brian Roepke

01/20/2023, 9:04 PM

Copy code

prefect deployment build snow_conn_test.py:snowflake_query_flow -n lambda-tmdb-s3 -sb s3/s3-prefect -q lamdba-tmdb -o snowflake_query_flow-deployment.yaml

Brian Roepke

01/20/2023, 9:04 PM

yaml

Brian Roepke

01/20/2023, 9:04 PM

basically followed the tutorial verbatim

Brian Roepke

01/20/2023, 9:05 PM

Copy code

prefect deployment apply snowflake_query_flow-deployment.yaml

Christopher Boyd

01/20/2023, 9:05 PM

yea, that does look odd - what tutorial did you try? I’d like to try and reproduce

Brian Roepke

01/20/2023, 9:05 PM

https://docs.prefect.io/tutorials/storage/

Christopher Boyd

01/20/2023, 9:05 PM

gotcha , I’ll need to walk through it - nothing you shared looks out of sorts, so it shouldn’t be an empty string like that

Brian Roepke

01/20/2023, 9:06 PM

Thank you @Christopher Boyd

Brian Roepke

01/21/2023, 10:35 PM

@Christopher Boyd (sorry to bother on a weekend). I got it resolved! I found this article which contained the error I was getting (I don't know if the mesasge changed or if I modified something to get it...

Copy code

RuntimeError: File system created with scheme 's3' from base path '<s3://prefect-test-1/flows/lambda-tmdb|s3://prefect-test-1/flows/lambda-tmdb>' could not be created. You are likely missing a Python module required to use the given storage protocol.

I decided to reinstall

s3fs,

and it turned out there was a dependency conflict that I didn't notice the first time. It was on the AWS CLI that I installed prior. I removed that and re-installed

s3fs,

and boom. It worked! Thank you for the troubleshooting.

🙌 1

✅ 1

Brian Roepke

01/21/2023, 11:35 PM

BTW - the storage location string is still empty regardless of it working or not.

Christopher Boyd

01/23/2023, 2:35 PM

That’s great to hear! I’ll still take a look the empty string, to see if that’s an issue. Would you mind confirming what version of Prefect you’re using here?

Brian Roepke

01/23/2023, 3:16 PM

2.7.8

4 Views

Open in Slack

Previous Next