Hi I m a little bit confused in Prefect2 I m migrating from Prefect Community #prefect-community

Hi! I'm a little bit confused in Prefect2. I'm mig...

jcozar

02/04/2023, 12:06 PM

Hi! I'm a little bit confused in Prefect2. I'm migrating from v1 to v2, and here is my architecture in AWS: • My tasks just call ECS tasks, so my flows are very simple • The prefect agent is running in a ECS task, so I don't care about permissions (the task role is granted with the required permissions) I read the documentation and I don't understand how the flow deployment process works: • In v1 I use Docker storage (in ECR) to serialize and save the flow. In v2 I only see filesystems like S3 to do that. There is no option to use docker? If I use -sb with a s3 bucket, how it works with Docker? The image is built for every flow run or how it works? • The documentation always refers to blocks. I created a block for ECS infrastructure, and used

prefect deployment build -a

to build and apply de deployment on prefect cloud. But when I run a flow run, the agent crashes because of credentials:

botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the RegisterTaskDefinition operation

. The AWS_PROFILE is configured to use the correct credentials in the agent environment, so obviously I am missing something about the workflow in prefect v2. As you can see I am lost in prefect v2 🙂 Can you share with me some tutorial or link with best practices to work with prefect v2 in AWS? Thank you very much!

Jack

02/04/2023, 2:22 PM

hey @jcozar, I used this to AWS ECS Fargate up and runnings, does this help? https://towardsdatascience.com/prefect-aws-ecs-fargate-github-actions-make-serverless-dataflows-as-easy-as-py-f6025335effc

jcozar

02/04/2023, 8:36 PM

Ok thank you very much! It helped indeed 🙂

jcozar

02/04/2023, 8:43 PM

However I'm not sure why @Anna Geller uses s3 storage... I can see that she builds the Docker image within the flow code inside. Therefore, why using s3 storage that (as long as I understand) copies all the files inside /opt/prefect/flows/?

jcozar

02/04/2023, 9:58 PM

I think that I understand why she did that. Please @Anna Geller sorry if I'm wrong, but I've been playing with it and if I use a LocalFileSystem the flow tries to run the flow from an absolute path "copied" from my local filesystem (not from /opt/prefect/flows/ where I copied the flow in my Docker image). I didn't find a way to modify the path, since

path

argument from

Deployment

class requires to be a subpath of

basepath

from

LocalFileSystem

. In order to understand the worflow I used --bs s3/my-s3-block in the deployment. Then, before running a flow run, I manually deleted all the files from the s3 bucket (I could use .prefectignore instead). Then, I run the flow run (I copied the flow.py in

/opt/prefect/flows

and used WORKDIR as well). It worked perfectly so the --bs was not used by the image. If I'm right, I have the following question: • A block storage is always required in a deployment. What's the best way to create a deployment for docker images where there is no need to provide one? In prefect v1 there was a Docker "block storage", but not now Thank you very much!

jcozar

02/04/2023, 10:38 PM

Ok, after a few more tries I answer myself 😄 If no storage is provided, the flow uses its local path (which is different from using

LocalFileSystem

). Without providing a storage block, I can set the path to

/opt/prefect/flows/

and it works! The deployment.yml file is created as follows:

Copy code

...
###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: XXXXX
manifest_path: null
storage: null
path: /opt/prefect/flows/
entrypoint: XXX.py:XXX
parameter_openapi_schema:
  title: Parameters
  type: object
  properties: {}
  required: null
  definitions: null
timestamp: '2023-02-04T22:32:39.131444+00:00'

💯 1

Anna Geller

02/05/2023, 8:37 AM

that's exactly right - the

--path

argument can be provided to set the path within the image, I agree that the UX could be a little more intuitive but we'll certainly improve that in the future 🙌

👍 1

🙏 1

Ethan Veres

03/01/2023, 10:37 PM

@jcozar I’m running into the exact same issue. Do you by any chance have a working sample I can run off of?

jcozar

03/02/2023, 10:06 AM

Hi @Ethan Veres! I can share what I did: • Infrastructure: ECSTask(...) • Deployment: Deployment.build_from_flow(..., path="/", infrastructure=infrastructure) • Infrastructure Docker Image (in

flows

folder I have all the definitions for tasks, flows and deployments)

Copy code

FROM prefecthq/prefect:2-python3.11
RUN python -m pip install --upgrade pip
RUN pip install prefect-aws

WORKDIR /

COPY src/flows/ /src/flows/

Ethan Veres

03/02/2023, 2:20 PM

Thanks @jcozar ! Is that your whole Dockerfile? No additional CMD or RUN needed? Or is that in your ECS task definition?

jcozar

03/02/2023, 3:58 PM

That's my whole Dockerfile, used to create my infrastructure (ECSTask). Inside of src/flows is all the Prefect code, so when the agent receives this container it can run any flow inside of it. There is another option, which is using S3 for storage, so the agent run the container and download the flow stored in S3, but for me this way was easier!

Ethan Veres

03/02/2023, 4:01 PM

thanks! going to test that out

👍 1

Open in Slack

Previous Next