Jacob Goldberg
10/01/2021, 4:33 PMSuccessfully pulled image XXXXX
and agent | Completed deployment of flow run XXXXX
, but the flows content is not being updated . When I change the runtime Labels on Prefect cloud and execute the same (updated) flow in another environment (docker agent running on mac osx) the updates i made to the flow are reflected…Kevin Kho
Jacob Goldberg
10/01/2021, 4:38 PM<http://XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/cal_val_etl_flows:latest|XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/cal_val_etl_flows:latest>
accroding to the EC2 terminal output. I dont realyl have experience with ECR outside of storing Prefect images, but i have not changed any configuration settings, so I am not sure why it would be wrong if it is pulling ‘latest’. it has always worked in the past and it is working on another environmentKevin Kho
Jacob Goldberg
10/01/2021, 4:42 PMJacob Goldberg
10/01/2021, 4:47 PMFailed to load and execute Flow's environment: StorageError('An error occurred while unpickling the flow:\n ModuleNotFoundError("No module named \'tasks\'")\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - python: (flow built with \'3.8.6\', currently running with \'3.8.10\')\nThis also may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')
Jacob Goldberg
10/01/2021, 4:50 PMJacob Goldberg
10/01/2021, 4:50 PMKevin Kho
Jacob Goldberg
10/01/2021, 5:08 PMKevin Kho
Kevin Kho
Jacob Goldberg
10/01/2021, 5:24 PMJacob Goldberg
10/01/2021, 5:24 PMprefect.storage.docker.Docker()
function to build the docker image, and have relied on default for the base_image arg. The docs say that the
"the base image for this when building this image (e.g. python:3.6), defaults to the prefecthq/prefect image matching your python version and prefect core library version used at runtime."
Therefore I am not sure what could be causing the python version mismatchJacob Goldberg
10/01/2021, 5:25 PMKevin Kho
Jacob Goldberg
10/01/2021, 5:26 PMKevin Kho
ECSRun(..., env="PREFECT___LOGGING___LEVEL":"DEBUG")
Kevin Kho
Kevin Kho
Jacob Goldberg
10/01/2021, 5:42 PMJacob Goldberg
10/01/2021, 5:42 PMKevin Kho
Kevin Kho
Jacob Goldberg
10/01/2021, 5:47 PM"PREFECT___LOGGING___LEVEL":"DEBUG"
, updated the image, and re-registered the flow and I am still seeing the same message with a basic test flow. i.e. no additional information from debuggingJacob Goldberg
10/01/2021, 5:47 PMJacob Goldberg
10/01/2021, 5:47 PMKevin Kho
Kevin Kho
Jacob Goldberg
10/01/2021, 5:52 PMKevin Kho
Jacob Goldberg
10/01/2021, 6:05 PMKevin Kho
Jacob Goldberg
10/01/2021, 7:22 PMprefect.storage.docker.Docker()
is causing the problem. I am starting outside of the EC2 environment, and working on my Mac OSX environment where everything has always worked fine until the last 48 hours. I have ensured the build environment of the image + registration is running python 3.8.5 and that the agent is running under python 3.8.5 however I still get this message when trying to execute flows:
Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n ModuleNotFoundError("No module named \'tasks\'")\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - python: (flow built with \'3.8.5\', currently running with \'3.8.12\')\nThis also may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')
Jacob Goldberg
10/01/2021, 7:23 PMprefect.storage.docker.Docker()
that is choosing a base image with python 3.8.12 causing this error. Am i missing something here? This still may not be root cause of my issues, but i do want to ensure the python versions are alignedJacob Goldberg
10/01/2021, 7:24 PMKevin Kho
Kevin Kho
Jacob Goldberg
10/01/2021, 7:27 PMwith open("../requirements.txt") as f:
requirements = f.read().splitlines()
STORAGE = Docker(
registry_url="<http://XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/|XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/>",
image_name="cal_val_etl_flows",
image_tag=f"latest-{round(time.time())}",
python_dependencies=requirements,
# copy module files to Docker container
files={
os.path.dirname(
os.path.dirname(os.path.realpath(__file__))
): "/custom_modules/calval_etl"
},
# add module to python path of Docker container
env_vars={"PYTHONPATH": "$PYTHONPATH:custom_modules/calval_etl"},
# adding this, because run into healthcheck error where API modules are not authorized to fetch creds from aws secrets.
ignore_healthchecks=True,
)
Jacob Goldberg
10/01/2021, 7:27 PMKevin Kho
Kevin Kho
Jacob Goldberg
10/01/2021, 7:29 PMKevin Kho
Kevin Kho
Kevin Kho
Jacob Goldberg
10/01/2021, 7:33 PMKevin Kho
Jacob Goldberg
10/01/2021, 7:34 PMKevin Kho
Jacob Goldberg
10/01/2021, 7:43 PM404 Client Error for <http+docker://localhost/v1.41/containers/create?name=stoic-roadrunner>: Not Found ("No such image: <http://XXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/cal_val_etl_flows:latest-1633114956|XXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/cal_val_etl_flows:latest-1633114956>")
Jacob Goldberg
10/01/2021, 7:43 PMJacob Goldberg
10/01/2021, 7:43 PMJacob Goldberg
10/01/2021, 7:53 PMaws ecr list-images --repository-name "cal_val_etl_flows"
i can see the image that supposedly cant be foundJacob Goldberg
10/01/2021, 7:54 PMKevin Kho
Jacob Goldberg
10/01/2021, 7:55 PMKevin Kho
AWS_CREDENTIALS
as a secret on Prefect Cloud? (am hoping not)Jacob Goldberg
10/01/2021, 7:55 PMKevin Kho
Jacob Goldberg
10/01/2021, 7:56 PMRUN_CONFIG = DockerRun(
labels=["ec2", "prod"],
env={
"AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
"AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
"AWS_DEFAULT_REGION": os.environ["AWS_DEFAULT_REGION"],
"PREFECT__LOGGING__LEVEL": "DEBUG"
},
)
Kevin Kho
Jacob Goldberg
10/01/2021, 8:00 PMJacob Goldberg
10/01/2021, 8:01 PMKevin Kho
docker pull
directly and see if that works?Jacob Goldberg
10/01/2021, 8:03 PMJacob Goldberg
10/01/2021, 8:04 PMwrite /var/lib/docker/tmp/GetImageBlob150753823: no space left on device
Jacob Goldberg
10/01/2021, 8:04 PMKevin Kho
Kevin Kho
dockerpy
side so maybe it really doesn’t find it? It does say 404Kevin Kho
Jacob Goldberg
10/01/2021, 8:26 PMJacob Goldberg
10/01/2021, 8:27 PMKevin Kho
Jacob Goldberg
10/01/2021, 8:28 PMKevin Kho
docker image prune
through the ShellTask lolJacob Goldberg
10/01/2021, 8:31 PMJacob Goldberg
10/01/2021, 8:32 PM