jcozar
02/04/2023, 12:06 PMprefect deployment build -a
to build and apply de deployment on prefect cloud. But when I run a flow run, the agent crashes because of credentials: botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the RegisterTaskDefinition operation
. The AWS_PROFILE is configured to use the correct credentials in the agent environment, so obviously I am missing something about the workflow in prefect v2.
As you can see I am lost in prefect v2 š Can you share with me some tutorial or link with best practices to work with prefect v2 in AWS?
Thank you very much!Jack
02/04/2023, 2:22 PMjcozar
02/04/2023, 8:36 PMpath
argument from Deployment
class requires to be a subpath of basepath
from LocalFileSystem
.
In order to understand the worflow I used --bs s3/my-s3-block in the deployment. Then, before running a flow run, I manually deleted all the files from the s3 bucket (I could use .prefectignore instead). Then, I run the flow run (I copied the flow.py in /opt/prefect/flows
and used WORKDIR as well). It worked perfectly so the --bs was not used by the image. If I'm right, I have the following question:
⢠A block storage is always required in a deployment. What's the best way to create a deployment for docker images where there is no need to provide one? In prefect v1 there was a Docker "block storage", but not now
Thank you very much!LocalFileSystem
). Without providing a storage block, I can set the path to /opt/prefect/flows/
and it works! The deployment.yml file is created as follows:
...
###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: XXXXX
manifest_path: null
storage: null
path: /opt/prefect/flows/
entrypoint: XXX.py:XXX
parameter_openapi_schema:
title: Parameters
type: object
properties: {}
required: null
definitions: null
timestamp: '2023-02-04T22:32:39.131444+00:00'
Anna Geller
02/05/2023, 8:37 AM--path
argument can be provided to set the path within the image, I agree that the UX could be a little more intuitive but we'll certainly improve that in the future šEthan Veres
03/01/2023, 10:37 PMjcozar
03/02/2023, 10:06 AMflows
folder I have all the definitions for tasks, flows and deployments)
FROM prefecthq/prefect:2-python3.11
RUN python -m pip install --upgrade pip
RUN pip install prefect-aws
WORKDIR /
COPY src/flows/ /src/flows/
Ethan Veres
03/02/2023, 2:20 PMjcozar
03/02/2023, 3:58 PMEthan Veres
03/02/2023, 4:01 PM