https://prefect.io logo
Channels
Powered by
v

Vadym Dytyniak

08/31/2023, 8:05 AM
Hello. I use pull_from_s3 step and sometimes I see the following error:
Untitled
Image that I use has 
prefect-aws
installed
prefect.yaml:
Copy code
build: null
push:
  - prefect_aws.deployments.steps.push_to_s3:
      id: push-flow
      requires: prefect-aws==0.3.6
      bucket: org-prefect-poc
      folder: flows/sample

pull:
  - prefect_aws.deployments.steps.pull_from_s3:
      id: pull-flow
      bucket: org-prefect-poc
      folder: "{{ push-flow.folder }}"

deployments:
  - name: sample
    entrypoint: sample.py:sample
    work_pool:
        name: default-k8s-pool
        work_queue_name: k8s-queue
        job_variables:
          service_account_name: sample
          env:
            EXTRA_PIP_PACKAGES: org-chronos[complete]==1.14.4
    schedule: null
and there is one more error that is random as well:
Untitled
interesting fact about the flow runs where I see these errors: there is no mention about flow run in worker logs and there is no pod yet. I see errors immediately on UI.
j

Jamie Zieziula

08/31/2023, 5:14 PM
Copy code
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
this is telling you that whatever credentials you are using don’t have the appropriate permissions to complete the api call
I don’t see where you specify the image to use. By default, the worker will use a prefect image which doesn’t have the 
prefect_aws
module installed on it
v

Vadym Dytyniak

09/01/2023, 6:26 AM
@Jamie Zieziula K8s pod has correct role with needed permissions. I see this error even before pod started. Looks like some other machine is trying to run pull from s3 step. I think these errors are related.
I specify an image on work pool level:
@Jamie Zieziula Could you please help me to understand how it works?
Powered by