Joshua Grant
01/30/2023, 11:48 PMprefect==2.7.10
using ECSTask
from prefect-aws==0.2.4
, in which running the flow is baked into the Docker image but prefect
copies over the code, which results in a shutil.Error
. Before upgrading, we used the GitHub
storage block in the command to Deployment.build_from_flow()
as we encountered this error when using the path
parameter with prefect==2.6.9
. Details in ๐งต.Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.9/site-packages/prefect/engine.py", line 269, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/opt/bitnami/python/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/opt/bitnami/python/lib/python3.9/site-packages/prefect/deployments.py", line 175, in load_flow_from_flow_run
await storage_block.get_directory(from_path=deployment.path, local_path=".")
File "/opt/bitnami/python/lib/python3.9/site-packages/prefect/filesystems.py", line 942, in get_directory
copytree(
File "/opt/bitnami/python/lib/python3.9/shutil.py", line 566, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "/opt/bitnami/python/lib/python3.9/shutil.py", line 522, in _copytree
raise Error(errors)
from os import getenv
from pydantic import SecretStr
from prefect.deployments import Deployment
from prefect.filesystems import GitHub
from prefect_aws.ecs import AwsCredentials, ECSTask
from pyaml_env import parse_config
from flow_file import my_flow
aws_creds = AwsCredentials()
github_block = GitHub(repository="url_to_repository",
reference=getenv('GIT_TAG'), access_token=SecretStr(getenv('GIT_API_KEY')))
task_def = parse_config('task_definition.yaml')
ecs_task_fg = ECSTask(
aws_credentials=aws_credentials_block,
image=docker_image,
cpu=4096,
memory=8192,
stream_output=True,
configure_cloudwatch_logs=False,
cluster=os.getenv('ECS_CLUSTER'),
execution_role_arn=os.getenv('FLOW_EXECUTION_ROLE'),
task_role_arn=getenv('FLOW_TASK_ROLE'),
vpc_id=getenv('VPC_ID'),
task_definition=task_def,
)
deployment_fg = Deployment.build_from_flow(
flow=excel_flow,
name=f'{getenv("IMAGE_NAME")}-fg',
version='1',
work_queue_name=getenv('IMAGE_NAME'),
infrastructure=ecs_task_fg,
storage=github_block,
)
aws_credentials_block.save('aws-creds', overwrite=True)
ecs_task_fg.save(f'{getenv("IMAGE_NAME")}-ecs-task-fg', overwrite=True)
deployment_fg.apply()
GitHub
block. However, since upgrading, both the GitHub
storage block or specifying path='/app/'
result in the copy error.storage=github_block
method was working, but specifying path='/app/'
resulted in the copy error, now both methods result in the copy error.task_definition.yaml
species sidecar containers to load with the task for routing logs to DataDog. It uses the awsfirelens
logDriver, which incorrectly reports:
23:35:26.125 | WARNING | prefect.infrastructure.ecs-task - ECSTask '588acc00-5ee0-4ce8-b65d-1086106f49ec': Logging configuration uses unsupported driver {container_def['logConfiguration'].get('logDriver')!r}. Output cannot be streamed.
but that is a separate issue and is only a warning.GitHub
storage solution works if specifying path=None
when calling Deployment.build_from_flow()
. However, I would still like to be able to use the flow baked in to the docker image.Peyton Runyan
01/31/2023, 2:24 PMshutil
error with a specified path, using the github block sounds a lot an issue that another user was facing. Does this help: https://github.com/PrefectHQ/prefect/pull/8193g = GitHub(
repository="<https://github.com/PrefectHQ/prefect.git>",
include_git_objects=False,
)
Joshua Grant
01/31/2023, 2:29 PMpath=None
, otherwise it assigns the WORKDIR
of the image as the pathPeyton Runyan
01/31/2023, 2:35 PMJoshua Grant
01/31/2023, 2:38 PMpath=None
in the call to Deployment.build_from_flow()
. Looking forward for the baked in flow support! Thanks!