https://prefect.io logo
#prefect-server
Title
# prefect-server
a

Amogh Kulkarni

06/28/2022, 6:02 PM
I am getting an error “Flow could not be retrieved from deployment.botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden” when I am running the flow from the orion server set up in EKS. I am using S3 storage and Kubernetes Flow runner and a custom image in ECR. The deployment spec and flow is on my local machine and I deploy it using the prefect cli command (prefect deployment create <path of python file>) Any help on this is really appreciated… This is the sequence of steps that led me to the error: 1. I Modified the config on my local machine to point to our orion server in DEV.
Copy code
prefect config set PREFECT_API_URL=http://<orion-dev-server>:4200/api
2. I then wrote the code of the flow and deployment spec on my local machine. Code of the deployment spec:
Copy code
DeploymentSpec(
    flow_location="<flow location>",
    flow_name="flow_one",
    name="deployment-one",
    flow_runner=KubernetesFlowRunner(
        image="<image is on AWS ECR>",
        namespace="<kubernetes namespace>",
        image_pull_policy="Always"
    )
)
I then run the deployment by prefect deployment create <deployment python file> This runs fine. The file gets copied to S3 storage and I can see the deployment and flow on the orion server.
k

Kevin Kho

06/28/2022, 9:44 PM
Hey Amogh, could you move the traceback to the thread when you get the chance? I think this is saying you dont have permissions to upload to the bucket
z

Zanie

06/28/2022, 9:48 PM
I agree with Kevin this looks like a permissions error, but to read the object from the bucket.
I presume your credentials are available locally for the
create
operation, but are they available in the flow run image?
a

Amogh Kulkarni

06/28/2022, 10:21 PM
Moving traceback error to a thread.
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 212, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_deployment(deployment, client=client)
  File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 105, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 540, in load_flow_from_deployment
    maybe_flow = await client.resolve_datadoc(deployment.flow_data)
  File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 1916, in resolve_datadoc
    return await resolve_inner(datadoc)
  File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 1909, in resolve_inner
    data = await self.retrieve_data(data)
  File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 1477, in retrieve_data
    return await storage_block.read(embedded_datadoc)
  File "/usr/local/lib/python3.8/site-packages/prefect/blocks/storage.py", line 194, in read
    return await run_sync_in_worker_thread(self._read_sync, key)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncio.py", line 53, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.8/site-packages/prefect/blocks/storage.py", line 204, in _read_sync
    s3_client.download_fileobj(Bucket=self.bucket, Key=key, Fileobj=stream)
  File "/usr/local/lib/python3.8/site-packages/boto3/s3/inject.py", line 795, in download_fileobj
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/usr/local/lib/python3.8/site-packages/s3transfer/tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/s3transfer/download.py", line 354, in _submit
    response = client.head_object(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
```
On the AWS envrionment, we have IAM policies in place for the Kubernetes pod to read/write/delete files from S3. I do not need to explictly pass AWS credentials that I use on my local machine.
Same goes for the image.
We have prefect 1.0 running perfectly fine with the same AWS environment.
So, thats why i know that there are no permissioning issues with ECR, S3 and the EKS environment.
z

Zanie

06/28/2022, 10:25 PM
This is a pretty straight-forward call to boto3 though, there’s not much that we’re doing here and it’s definitely a permission error.
Have you included a
*
at the end of your bucket ACL?
a

Amogh Kulkarni

06/28/2022, 10:26 PM
I tried multiple things in the day and I think I have a theory. I think that might be it. I am unfortunately using the bleeding edge (orion branch). That’s why everything was working fine 3 days back but today I am facing an issue running even the most simplest flow.
Let me try setting up everything with the stable version 2.0b7 and I will let you know how that goes…
I dont think its the problem with the ACL because the fllow was running fine 3 days back.
Let me try using 2.0b7 version for all the components and I will get back to you if this theory works for not.
Thanks Kevin and Michael
a

Anna Geller

06/30/2022, 12:26 PM
@Amogh Kulkarni since you cross-posted the same question here and on Discourse, I'm gonna remove the request on Discourse. Please don't post your questions twice, one request is enough, we will get back to you as soon as we can
👍 1
4 Views