Devin Flake
01/28/2025, 9:04 PMPREFECT__STORAGE_BASE_PATH to me?Marvin
01/28/2025, 9:04 PMPREFECT__STORAGE_BASE_PATH environment variable is used in Prefect to specify a default local path for file storage. This path acts as a base directory where Prefect will store or access files unless another path is specified. It ensures that file operations are restricted to a specific directory, providing a level of security and organization.
For more detailed information, you might want to check out the Prefect documentation on file storage here.Nate
01/28/2025, 9:05 PMDevin Flake
01/28/2025, 9:13 PM.deploy() where it seems like the entrypoint_type isn't being used/detected, hence the suggestion to use a prefect.yaml where you can specify the directory for the flow, ex:
entrypoint: flows/export/set_export_freq.py:set_export_freq
path: /opt/prefect/arte_tasks/
pull:
- prefect.deployments.steps.set_working_directory:
directory: /opt/prefect/arte_tasksDevin Flake
01/28/2025, 9:18 PMprefect.yaml and simply have all the config for the Docker image build in the .deploy() since our Docker build is kind of complex needing to get env variables in order build correctly:
A_TOKEN = os.getenv("A_TOKEN")
set_export_freq.deploy(
name="set_export_freq",
work_pool_name="arte-k8s",
image=DockerImage(
name="docker-arte-release.dr-uw2.adobeitc.com/arte-flows-dflake",
tag="0.4",
dockerfile="containers/arte-flows/Dockerfile",
buildargs={"username": "arte", "pypitoken": A_TOKEN},
),
entrypoint_type="arte_tasks/flows/export/set_export_freq.py:set_export_freq",
parameters={"started_by": "dflake", "dry_run": True, "dpc_type": "prod", "verbose": True},
#cron="0 9 * * *",
description="Find rsids with > 1M daily hits. Set the export interval to 30 min if interval > 30 min.",
tags=["automation", "export"]
)Nate
01/28/2025, 9:21 PMentrypoint_type should probably be private, and you shouldnt have to set it. if you go to definition on that you'll see it defaults to an enum member which is literally "file_path"
https://github.com/prefecthq/prefect/blob/main/src/prefect/flows.py#L1095
whereas entrypoint is the kwarg you'd set in from_source if you're not baking your code into your imageNate
01/28/2025, 9:22 PMentrypoint_type from your .deploy() to start, and then maybe see if there's some pathing issue in the dockerfileDevin Flake
01/28/2025, 9:25 PMfrom_source right? I'm not sure how the flow will be able to find the entrypoint thenDevin Flake
01/28/2025, 9:28 PMDevin Flake
01/28/2025, 9:32 PMNate
01/28/2025, 9:34 PM.deploy if you're baking code into your image then you should only need to set your image and then you can handle pathing inside the Dockerfile itselfDevin Flake
01/28/2025, 9:34 PM