Hi I m a little bit confused in Prefect2 I m migrating from

Question

Hi I m a little bit confused in Prefect2 I m migrating from v1 to v2 and here is my architecture in AWS My tasks just call ECS tasks so my flows are very simple The prefect agent is running in a ECS task so I don t care about permissions the task role is granted with the required permissions I read the documentation and I don t understand how the flow deployment process works In v1 I use Docker storage in ECR to serialize and save the flow In v2 I only see filesystems like S3 to do that There is no option to use docker If I use sb with a s3 bucket how it works with Docker The image is built for every flow run or how it works The documentation always refers to blocks I created a block for ECS infrastructure and used `prefect deployment build a` to build and apply de deployment on prefect cloud But when I run a flow run the agent crashes because of credentials `botocore errorfactory AccessDeniedException An error occurred AccessDeniedException when calling the RegisterTaskDefinition operation` The AWS PROFILE is configured to use the correct credentials in the agent environment so obviously I am missing something about the workflow in prefect v2 As you can see I am lost in prefect v2 slightly smiling face Can you share with me some tutorial or link with best practices to work with prefect v2 in AWS Thank you very much

Jack · Answer

hey < jcozar> I used this to AWS ECS Fargate up and runnings does this help

jcozar · Answer

Ok thank you very much It helped indeed slightly smiling face

jcozar · Answer

However I m not sure why < Anna Geller> uses s3 storage I can see that she builds the Docker image within the flow code inside Therefore why using s3 storage that as long as I understand copies all the files inside opt prefect flows

jcozar · Answer

I think that I understand why she did that Please < Anna Geller> sorry if I m wrong but I ve been playing with it and if I use a LocalFileSystem the flow tries to run the flow from an absolute path copied from my local filesystem not from opt prefect flows where I copied the flow in my Docker image I didn t find a way to modify the path since `path` argument from `Deployment` class requires to be a subpath of `basepath` from `LocalFileSystem` In order to understand the worflow I used bs s3 my s3 block in the deployment Then before running a flow run I manually deleted all the files from the s3 bucket I could use prefectignore instead Then I run the flow run I copied the flow py in ` opt prefect flows` and used WORKDIR as well It worked perfectly so the bs was not used by the image If I m right I have the following question A block storage is always required in a deployment What s the best way to create a deployment for docker images where there is no need to provide one In prefect v1 there was a Docker block storage but not now Thank you very much

jcozar · Answer

Ok after a few more tries I answer myself smile If no storage is provided the flow uses its local path which is different from using `LocalFileSystem` Without providing a storage block I can set the path to ` opt prefect flows ` and it works The deployment yml file is created as follows ``` DO NOT EDIT BELOW THIS LINE flow name XXXXX manifest path null storage null path opt prefect flows entrypoint XXX py XXX parameter openapi schema title Parameters type object properties required null definitions null timestamp 2023 02 04T22 32 39 131444+00 00 ```

Anna Geller · Answer

that s exactly right the ` path` argument can be provided to set the path within the image I agree that the UX could be a little more intuitive but we ll certainly improve that in the future raised hands

Ethan Veres · Answer

< jcozar> I m running into the exact same issue Do you by any chance have a working sample I can run off of

jcozar · Answer

Hi < Ethan Veres> I can share what I did Infrastructure ECSTask Deployment Deployment build from flow path= infrastructure=infrastructure Infrastructure Docker Image in `flows` folder I have all the definitions for tasks flows and deployments ```FROM prefecthq prefect 2 python3 11 RUN python m pip install upgrade pip RUN pip install prefect aws WORKDIR COPY src flows src flows ```

Ethan Veres · Answer

Thanks < jcozar> Is that your whole Dockerfile No additional CMD or RUN needed Or is that in your ECS task definition

jcozar · Answer

That s my whole Dockerfile used to create my infrastructure ECSTask Inside of src flows is all the Prefect code so when the agent receives this container it can run any flow inside of it There is another option which is using S3 for storage so the agent run the container and download the flow stored in S3 but for me this way was easier

Ethan Veres · Answer

thanks going to test that out