Hi I am trying to use S3 storage for the flow code as follow

Question

Hi I am trying to use S3 storage for the flow code as follows ```STORAGE = S3 bucket= prefect pipelines key=f flows forecaster flow py stored as script=True this will ensure to upload the Flow script to S3 during registration local script path=f flow py ``` The following command is used for registering the flow ```prefect register project forecaster p forecaster ``` But for some reason all the modules under the `forecaster` directory is not getting uploaded under the S3 key specified Need help in solving this Thanks in advance

Saurabh Sharma · Answer

Just for reference

Anna Geller · Answer

S3 storage will upload only your flow code it won t package your dependencies You could package your code dependencies and custom modules into a Docker image <https discourse prefect io t when i run my flow i see an error failed to load and execute flows environment modulenotfounderror no module named users username what is happening 33|This Discourse topic> provides an in depth explanation of the issue and how you can approach it section 2 Missing code dependencies in the flow s execution environment is relevant in your use case

Saurabh Sharma · Answer

Awesome thanks Let me try this out

Saurabh Sharma · Answer

Hi I tried this but for some reason the code packaging was not happening as expected So to reproduce the issue I switched to Docker Storage for my local workflow using the ECR image as base image as follows ```with Flow FLOW NAME storage=Docker registry url= base image= image tag= latest as flow ``` But when I run my `flow py` I get EOF error while it attempts to push the image to registry Logs for the same are below ``` python flow py Users saurabh pyenv versions askria financial template service lib python3 9 site packages prefect core flow py 1708 UserWarning No result handler was specified on your Flow Cloud features such as input caching and resuming task runs from failure may not work properly registered flow = client register 2022 04 22 13 58 32+0530 INFO prefect Docker | Building the flow s Docker storage Step 1 9 FROM gt d2addd0e99fc Step 2 9 ENV PREFECT USER CONFIG PATH= opt prefect config toml gt Running in be01aa09d8d3 Removing intermediate container be01aa09d8d3 gt dc9c3512b05d Step 3 9 RUN pip install pip upgrade gt Running in fd9adcabcc72 Requirement already satisfied pip in usr local lib python3 9 site packages 22 0 4 WARNING Running pip as the root user can result in broken permissions and conflicting behaviour with the system package manager It is recommended to use a virtual environment instead Removing intermediate container fd9adcabcc72 gt d7f400326e18 Step 4 9 RUN pip show prefect || pip install git+ gt Running in 97b539c2551a Name prefect Version 1 1 0 Summary The Prefect Core automation and scheduling engine Home page Author Prefect Technologies Inc Author email License Apache License 2 0 Location usr local lib python3 9 site packages Requires click cloudpickle croniter dask distributed docker importlib resources marshmallow marshmallow oneofschema msgpack mypy extensions packaging pendulum python box python dateutil python slugify pytz pyyaml requests tabulate toml urllib3 Required by Removing intermediate container 97b539c2551a gt ac4b00fad79e Step 5 9 RUN pip install wheel gt Running in 00dcc88ede0b Requirement already satisfied wheel in usr local lib python3 9 site packages 0 37 1 WARNING Running pip as the root user can result in broken permissions and conflicting behaviour with the system package manager It is recommended to use a virtual environment instead Removing intermediate container 00dcc88ede0b gt cd84c116e8c9 Step 6 9 RUN mkdir p opt prefect gt Running in e5d1fa741f51 Removing intermediate container e5d1fa741f51 gt 2ec2b40d4fdb Step 7 9 COPY forecaster flow opt prefect flows forecaster prefect gt 13cfddd0b18b Step 8 9 COPY healthcheck py opt prefect healthcheck py gt 11a2da36e87e Step 9 9 RUN python opt prefect healthcheck py opt prefect flows forecaster prefect 3 9 gt Running in a51b8b2f74aa Beginning health checks System Version check OK Cloudpickle serialization check OK Result check OK All health checks passed Removing intermediate container a51b8b2f74aa gt 976fd83bf28f Successfully built 976fd83bf28f Successfully tagged 2022 04 22 13 58 45+0530 INFO prefect Docker | Pushing image to the registry Traceback most recent call last File Users saurabh Dropbox Projects professional Askria code financial template service workflows forecaster flow py line 104 in lt module gt flow register project name= forecaster File Users saurabh pyenv versions askria financial template service lib python3 9 site packages prefect core flow py line 1708 in register registered flow = client register File Users saurabh pyenv versions askria financial template service lib python3 9 site packages prefect client client py line 848 in register serialized flow = flow serialize build=build type Any File Users saurabh pyenv versions askria financial template service lib python3 9 site packages prefect core flow py line 1497 in serialize storage = self storage build type Optional Storage File Users saurabh pyenv versions askria financial template service lib python3 9 site packages prefect storage docker py line 325 in build self build image push=push File Users saurabh pyenv versions askria financial template service lib python3 9 site packages prefect storage docker py line 399 in build image self push image full name self image tag File Users saurabh pyenv versions askria financial template service lib python3 9 site packages prefect storage docker py line 613 in push image raise InterruptedError line get error InterruptedError EOF``` Post this I even did the following step on shell to ensure the docker login is done for ECR ```aws ecr get login password region us west 1 profile askria deploy | docker login username AWS password stdin Login Succeeded``` Not sure what else is missing Would appreciate any sort of help Thanks

Anna Geller · Answer

Can you try to push the image manually yourself to see whether it works this way you can do that using ```docker push <http xxxxxxxxxxx dkr ecr us west 1 amazonaws com forecaster latest|xxxxxxxxxxx dkr ecr us west 1 amazonaws com forecaster latest>```

Anna Geller · Answer

you could also follow the setup described in section Deploying your flows to a remote Kubernetes cluster on AWS EKS in <https www prefect io blog orchestrating elt on kubernetes with prefect dbt and snowflake part 2 |this blog post>

Saurabh Sharma · Answer

Sure trying this

Saurabh Sharma · Answer

So the image is getting pushed to the ECR repo ``` docker push The push refers to repository 0ef046cb1c71 Pushed cbcd7f3aa78a Pushed 0e71c950bd25 Pushed 130afd05f548 Pushed 9244b3735769 Layer already exists 3599ac45f041 Layer already exists 0caacca1e28a Layer already exists cc4b440e810b Layer already exists f99c2bd0bdfc Layer already exists 8bcf066bab18 Layer already exists 7fce09c1d950 Layer already exists 1401df2b50d5 Layer already exists latest digest sha256 9a023585c2c97cc186c074785efa694f73cc2ff74443ab2ba56dbd1ffdb81609 size 2840```

Anna Geller · Answer

woohoo partying face

Saurabh Sharma · Answer

Let me follow the blog post you shared and revert