Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
c
Chris Gunderson
05/18/2023, 2:14 PMWe are starting to migrate to Prefect 2. I was setting up the directory structure to have blocks, deployment scripts, flows, tasks, and utilities. The flows folder will have a sub-folder to break up the projects. The same docker image will be used for all project, mainly due to the overlap of libraries we use.
When deploying the flow, I received this error:
ValueError: 'C:\\Users\\chris.gunderson\\source\\repos\\prefect2-scripts\\src\\main\\prefect\\flows\\allocations\\prefect_fidelity_allocations.py' is not in the subpath of 'C:\\Users\\chris.gunderson\\source\\repos\\prefect2-scripts\\src\\main\\prefect\\deploy' OR one path is relative and the other is absolute.
image.png
I can move the deployment script to the flows/allocations folder and it works, but I'd like the deployment scripts grouped together. Is that possible?
from prefect.deployments import Deployment
from prefect.filesystems import S3
from prefect.orion.schemas.schedules import CronSchedule
from prefect_aws.ecs import ECSTask
from src.main.prefect.flows.allocations.prefect_fidelity_allocations import FidelityAllocationsFlow
s3block = S3.load("sra-s3")
ecs_task_block = ECSTask.load("default")
deployment = Deployment \
.build_from_flow(name="FidelityAllocationsFlow_Deployment",
work_queue_name="ecs-worker",
flow=FidelityAllocationsFlow,
storage=s3block,
schedule=(CronSchedule(cron="13 15 * * 1-5", timezone="America/Chicago")),
# Parameters when registering flow to production removed for post
infrastructure=ecs_task_block
# output="fidelity.yaml" Troubleshooting yaml
)
deployment.apply()n
Nate
05/18/2023, 5:52 PMhi @Chris Gunderson - if you're just establishing your setup here, i might recommend checking out projects / workers / work pools to handle ECS flow run infra / deployments
here's a guide (that will be merged soon) and a rundown of the new concepts
c
Chris Gunderson
05/18/2023, 6:01 PM@Nate We have already created the AWS template for the Prefect agent, and the configuration for the ECS Tasks. The only thing that I'm looking for is how to separate the flows and deployment scripts. Please see the error and screen shot above.
n
Nate
05/18/2023, 6:17 PMI understand, I recommended the new approach because i saw
We are starting to migrate to Prefect 2and projects / workers will eclipse infra blocks / agents as a method of creating deployments in the future projects are built specifically to declaratively group many deployment configs and apply them many at a time, which seemed like what you were driving at as for your error, this highlights a limitation of the older deployment paradigm
ValueError: 'C:\\Users\\chris.gunderson\\source\\repos\\prefect2-scripts\\src\\main\\prefect\\flows\\allocations\\prefect_fidelity_allocations.py' is not in the subpath of 'C:\\Users\\chris.gunderson\\source\\repos\\prefect2-scripts\\src\\main\\prefect\\deploy'is not in the subpath of