https://prefect.io logo
Title
j

Joshua Grant

02/21/2023, 9:54 PM
@Marvin in prefect 2 what is the environmental variable to set to specify the flow path?
m

Marvin

02/21/2023, 9:54 PM
It looks like the environment variable you're asking about is
PATH
and
PREFECT__FLOW_PATH
. You can find more detailed information about this in the [Prefect Documentation](https://docs.prefect.io/ui/flows/), especially in the topics on Flows, Deployments, and Logging. Hope this helps!
z

Zanie

02/21/2023, 10:05 PM
There is no environment variable for this in Prefect 2
What are you trying to do?
j

Joshua Grant

02/21/2023, 10:06 PM
I think it is
PREFECT_LOCAL_STORAGE_PATH=
, I am trying to tell prefect where to look for a flow when building a deployment from flow without specifying storage (flow logic is in the container, as is the flow files).
error:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows'
z

Zanie

02/21/2023, 10:07 PM
What’s the entrypoint for your deployment?
j

Joshua Grant

02/21/2023, 10:08 PM
there isn't one. we found setting one caused execution of the entrypoint and the flow when the flow ran on the deployment infrastructure (
ECSTask
)
z

Zanie

02/21/2023, 10:09 PM
That’s a little confusing — the entrypoint is how we load the flow for execution
Are you saying it ran twice?
Once at load, then again after?
j

Joshua Grant

02/21/2023, 10:10 PM
correct.
z

Zanie

02/21/2023, 10:10 PM
Sounds like you’re calling your flow in the file then?
You should guard that with
if __name__ == "__main__":
   flow_call()
j

Joshua Grant

02/21/2023, 10:11 PM
it was guarded. No matter what we put in the entrypoint, it was executed along with the flow.
but that was back when we were using github storage. We want to use local storage now.
possible incorrect configuration of the
Dockerfile
?
FROM public.ecr.aws/bitnami/python:3.9-prod
WORKDIR /app
RUN apt-get update && apt-get install git libmagic-dev -y && \
      apt-get clean && \
      pip install --upgrade pip && \
      pip install pipenv
COPY . .
ARG GIT_API_KEY
ARG COMMON_PKG_VERSION="dev"
ENV PREFECT_LOCAL_STORAGE_PATH="/app"
RUN pipenv lock --clear && pipenv install --system
z

Zanie

02/21/2023, 10:12 PM
Setting the local storage path won’t make us look for a flow there
There needs to be an entrypoint of some sort, I’m pretty confused about your setup tbh.
j

Joshua Grant

02/21/2023, 10:15 PM
well, it's running fine no entrypoint with github storage, but with I think Prefect 2.5.x > < 2.7.x it reported a
shutil.copytree
error when using local storage
a

Anna Geller

02/22/2023, 1:09 AM
it would be easiest to use Prefect base image and then using /opt/prefect/flows as work dir should make Prefect look in the right place, could you try that?
FROM prefecthq/prefect:2-python3.9
WORKDIR /opt/prefect/flows
also --path CLI argument will point your deployment to the path within the docker image
j

Joshua Grant

02/22/2023, 3:33 PM
Interesting. We are probably not using prefect as intended then. We now have it working with the above Dockerfile (removal of the
PREFECT_LOCAL_STORAGE_PATH
envar breaks the flow. Here is a brief overview of how we are deploying Prefect v2 with this method: 1. a flow-centric image is created in AWS CodeBuild based on Dockerfile and is pushed to ECR. 2. CodeBuild fires off a script that spins up an ECS Task in the targeted AWS environment 3. ECS Task creates a task definition using the image created in CodeBuild, it overrides some things and adds sidecars to the new task definition 4. Above task definition is passed to prefect deployment script, which creates an
ECSTask
infrastructure block and then builds the flow (or flows) and registers them with prefect
some business requirements: • all of the deployment logic is handled in Python • build environment is siloed from deploy environments; we use (dev, alpha, beta, prod) • shared `prefect.task`s and deployment functions (and data like a
task_definitoin.yaml
that can be parsed with
pyaml-env
) are stored in a common python package We are using
prefect==0.15.2
simultaneously, but run into some issues with GraphQL. We are migrating from that prefect version to prefect 2
a

Anna Geller

02/22/2023, 5:56 PM
Prefect 1 and 2 are entirely different products and they have very different deployment patterns • v1 ECS examplev2 ECS example
j

Joshua Grant

02/22/2023, 6:12 PM
I think our biggest challenge migrating to v2 was to allow the addition of sidecars and avoiding getting the too-many-revisions to a task definition issue. Our current deployment strategy is working as expected albeit probably not what it was designed to do. Some things with
ECSTask
creation had to be modified (providing
task_definition_arn
and
auto_deregister_task=False
) to allow for us to use sidecars for logging to DataDog (among other things) and avoid the too-many-revisions. In our workflow, once a flow is "deployed" it is the source of truth until overwritten by another "deployment". The source of all this discussion was removing the need for a
Github
block and using the flow stored in the image, which the addition of the envar
PREFECT_LOCAL_STORAGE_PATH
solves (magically). This strategy allows us to deploy multiple
ECSTask
infrastructure blocks (`EC2`|`Fargate`) with varying amounts of memory and cpu. Downstream workflow logic determines the infrastructure flow to run (e.g.
-ec2-light
or
-fg-heavy
) in the flow name, which for light processing that is needed to complete quickly we use
EC2
with low resource request.