https://prefect.io
Join Slack
<@ULVA73B9P> in prefect 2 what is the environmental variable to set to specify the flow path?
# ask-marvin
j
@Marvin in prefect 2 what is the environmental variable to set to specify the flow path?
m
It looks like the environment variable you're asking about is 
PATH
and 
PREFECT__FLOW_PATH
. You can find more detailed information about this in the [Prefect Documentation](https://docs.prefect.io/ui/flows/), especially in the topics on Flows, Deployments, and Logging. Hope this helps!
z
There is no environment variable for this in Prefect 2
What are you trying to do?
j
I think it is 
PREFECT_LOCAL_STORAGE_PATH=
, I am trying to tell prefect where to look for a flow when building a deployment from flow without specifying storage (flow logic is in the container, as is the flow files).
error: 
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows'
z
What’s the entrypoint for your deployment?
j
there isn't one. we found setting one caused execution of the entrypoint and the flow when the flow ran on the deployment infrastructure (
ECSTask
)
z
That’s a little confusing — the entrypoint is how we load the flow for execution
Are you saying it ran twice?
Once at load, then again after?
j
correct.
z
Sounds like you’re calling your flow in the file then?
You should guard that with
Copy code
if __name__ == "__main__":
   flow_call()
j
it was guarded. No matter what we put in the entrypoint, it was executed along with the flow.
but that was back when we were using github storage. We want to use local storage now.
possible incorrect configuration of the 
Dockerfile
?
Copy code
FROM public.ecr.aws/bitnami/python:3.9-prod
WORKDIR /app
RUN apt-get update && apt-get install git libmagic-dev -y && \
      apt-get clean && \
      pip install --upgrade pip && \
      pip install pipenv
COPY . .
ARG GIT_API_KEY
ARG COMMON_PKG_VERSION="dev"
ENV PREFECT_LOCAL_STORAGE_PATH="/app"
RUN pipenv lock --clear && pipenv install --system
z
Setting the local storage path won’t make us look for a flow there
There needs to be an entrypoint of some sort, I’m pretty confused about your setup tbh.
j
well, it's running fine no entrypoint with github storage, but with I think Prefect 2.5.x > < 2.7.x it reported a 
shutil.copytree
error when using local storage
a
it would be easiest to use Prefect base image and then using /opt/prefect/flows as work dir should make Prefect look in the right place, could you try that?
Copy code
FROM prefecthq/prefect:2-python3.9
WORKDIR /opt/prefect/flows
also --path CLI argument will point your deployment to the path within the docker image
j
Interesting. We are probably not using prefect as intended then. We now have it working with the above Dockerfile (removal of the 
PREFECT_LOCAL_STORAGE_PATH
envar breaks the flow. Here is a brief overview of how we are deploying Prefect v2 with this method: 1. a flow-centric image is created in AWS CodeBuild based on Dockerfile and is pushed to ECR. 2. CodeBuild fires off a script that spins up an ECS Task in the targeted AWS environment 3. ECS Task creates a task definition using the image created in CodeBuild, it overrides some things and adds sidecars to the new task definition 4. Above task definition is passed to prefect deployment script, which creates an 
ECSTask
infrastructure block and then builds the flow (or flows) and registers them with prefect
some business requirements: • all of the deployment logic is handled in Python • build environment is siloed from deploy environments; we use (dev, alpha, beta, prod) • shared `prefect.task`s and deployment functions (and data like a 
task_definitoin.yaml
that can be parsed with 
pyaml-env
) are stored in a common python package We are using 
prefect==0.15.2
simultaneously, but run into some issues with GraphQL. We are migrating from that prefect version to prefect 2
a
Prefect 1 and 2 are entirely different products and they have very different deployment patterns • v1 ECS example • v2 ECS example
j
I think our biggest challenge migrating to v2 was to allow the addition of sidecars and avoiding getting the too-many-revisions to a task definition issue. Our current deployment strategy is working as expected albeit probably not what it was designed to do. Some things with 
ECSTask
creation had to be modified (providing 
task_definition_arn
and 
auto_deregister_task=False
) to allow for us to use sidecars for logging to DataDog (among other things) and avoid the too-many-revisions. In our workflow, once a flow is "deployed" it is the source of truth until overwritten by another "deployment". The source of all this discussion was removing the need for a 
Github
block and using the flow stored in the image, which the addition of the envar 
PREFECT_LOCAL_STORAGE_PATH
solves (magically). This strategy allows us to deploy multiple 
ECSTask
infrastructure blocks (`EC2`|`Fargate`) with varying amounts of memory and cpu. Downstream workflow logic determines the infrastructure flow to run (e.g. 
-ec2-light
or 
-fg-heavy
) in the flow name, which for light processing that is needed to complete quickly we use 
EC2
with low resource request.
9 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by