I am getting an error “Flow could not be retrieved from deployment.botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden” when I am running the flow from the orion server set up in EKS. I am using S3 storage and Kubernetes Flow runner and a custom image in ECR. The deployment spec and flow is on my local machine and I deploy it using the prefect cli command (prefect deployment create <path of python file>) Any help on this is really appreciated… This is the sequence of steps that led me to the error: 1. I Modified the config on my local machine to point to our orion server in DEV.

prefect config set PREFECT_API_URL=http://<orion-dev-server>:4200/api

2. I then wrote the code of the flow and deployment spec on my local machine. Code of the deployment spec:

DeploymentSpec(
    flow_location="<flow location>",
    flow_name="flow_one",
    name="deployment-one",
    flow_runner=KubernetesFlowRunner(
        image="<image is on AWS ECR>",
        namespace="<kubernetes namespace>",
        image_pull_policy="Always"
    )
)

I then run the deployment by prefect deployment create <deployment python file> This runs fine. The file gets copied to S3 storage and I can see the deployment and flow on the orion server.

Hey Amogh, could you move the traceback to the thread when you get the chance? I think this is saying you dont have permissions to upload to the bucket

I agree with Kevin this looks like a permissions error, but to read the object from the bucket.

Moving traceback error to a thread.

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 212, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_deployment(deployment, client=client)
  File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 105, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/deployments.py", line 540, in load_flow_from_deployment
    maybe_flow = await client.resolve_datadoc(deployment.flow_data)
  File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 1916, in resolve_datadoc
    return await resolve_inner(datadoc)
  File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 1909, in resolve_inner
    data = await self.retrieve_data(data)
  File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 1477, in retrieve_data
    return await storage_block.read(embedded_datadoc)
  File "/usr/local/lib/python3.8/site-packages/prefect/blocks/storage.py", line 194, in read
    return await run_sync_in_worker_thread(self._read_sync, key)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/asyncio.py", line 53, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.8/site-packages/prefect/blocks/storage.py", line 204, in _read_sync
    s3_client.download_fileobj(Bucket=self.bucket, Key=key, Fileobj=stream)
  File "/usr/local/lib/python3.8/site-packages/boto3/s3/inject.py", line 795, in download_fileobj
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.8/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/usr/local/lib/python3.8/site-packages/s3transfer/tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/s3transfer/download.py", line 354, in _submit
    response = client.head_object(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
```

This is a pretty straight-forward call to boto3 though, there’s not much that we’re doing here and it’s definitely a permission error.

I tried multiple things in the day and I think I have a theory. I think that might be it. I am unfortunately using the bleeding edge (orion branch). That’s why everything was working fine 3 days back but today I am facing an issue running even the most simplest flow.

@Amogh Kulkarni since you cross-posted the same question here and on Discourse, I'm gonna remove the request on Discourse. Please don't post your questions twice, one request is enough, we will get back to you as soon as we can