Joshua Grant
02/21/2023, 9:54 PMMarvin
02/21/2023, 9:54 PMPATH
and PREFECT__FLOW_PATH
. You can find more detailed information about this in the [Prefect Documentation](https://docs.prefect.io/ui/flows/), especially in the topics on Flows, Deployments, and Logging. Hope this helps!Zanie
02/21/2023, 10:05 PMJoshua Grant
02/21/2023, 10:06 PMPREFECT_LOCAL_STORAGE_PATH=
, I am trying to tell prefect where to look for a flow when building a deployment from flow without specifying storage (flow logic is in the container, as is the flow files).FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows'
Zanie
02/21/2023, 10:07 PMJoshua Grant
02/21/2023, 10:08 PMECSTask
)Zanie
02/21/2023, 10:09 PMJoshua Grant
02/21/2023, 10:10 PMZanie
02/21/2023, 10:10 PMif __name__ == "__main__":
flow_call()
Joshua Grant
02/21/2023, 10:11 PMDockerfile
?FROM public.ecr.aws/bitnami/python:3.9-prod
WORKDIR /app
RUN apt-get update && apt-get install git libmagic-dev -y && \
apt-get clean && \
pip install --upgrade pip && \
pip install pipenv
COPY . .
ARG GIT_API_KEY
ARG COMMON_PKG_VERSION="dev"
ENV PREFECT_LOCAL_STORAGE_PATH="/app"
RUN pipenv lock --clear && pipenv install --system
Zanie
02/21/2023, 10:12 PMJoshua Grant
02/21/2023, 10:15 PMshutil.copytree
error when using local storageAnna Geller
02/22/2023, 1:09 AMFROM prefecthq/prefect:2-python3.9
WORKDIR /opt/prefect/flows
Joshua Grant
02/22/2023, 3:33 PMPREFECT_LOCAL_STORAGE_PATH
envar breaks the flow. Here is a brief overview of how we are deploying Prefect v2 with this method:
1. a flow-centric image is created in AWS CodeBuild based on Dockerfile and is pushed to ECR.
2. CodeBuild fires off a script that spins up an ECS Task in the targeted AWS environment
3. ECS Task creates a task definition using the image created in CodeBuild, it overrides some things and adds sidecars to the new task definition
4. Above task definition is passed to prefect deployment script, which creates an ECSTask
infrastructure block and then builds the flow (or flows) and registers them with prefecttask_definitoin.yaml
that can be parsed with pyaml-env
) are stored in a common python package
We are using prefect==0.15.2
simultaneously, but run into some issues with GraphQL. We are migrating from that prefect version to prefect 2Anna Geller
02/22/2023, 5:56 PMJoshua Grant
02/22/2023, 6:12 PMECSTask
creation had to be modified (providing task_definition_arn
and auto_deregister_task=False
) to allow for us to use sidecars for logging to DataDog (among other things) and avoid the too-many-revisions. In our workflow, once a flow is "deployed" it is the source of truth until overwritten by another "deployment". The source of all this discussion was removing the need for a Github
block and using the flow stored in the image, which the addition of the envar PREFECT_LOCAL_STORAGE_PATH
solves (magically). This strategy allows us to deploy multiple ECSTask
infrastructure blocks (`EC2`|`Fargate`) with varying amounts of memory and cpu. Downstream workflow logic determines the infrastructure flow to run (e.g. -ec2-light
or -fg-heavy
) in the flow name, which for light processing that is needed to complete quickly we use EC2
with low resource request.