Devin Flake
01/28/2025, 9:04 PMPREFECT__STORAGE_BASE_PATH
to me?Marvin
01/28/2025, 9:04 PMPREFECT__STORAGE_BASE_PATH
environment variable is used in Prefect to specify a default local path for file storage. This path acts as a base directory where Prefect will store or access files unless another path is specified. It ensures that file operations are restricted to a specific directory, providing a level of security and organization.
For more detailed information, you might want to check out the Prefect documentation on file storage here.Nate
01/28/2025, 9:05 PMDevin Flake
01/28/2025, 9:13 PM.deploy()
where it seems like the entrypoint_type
isn't being used/detected, hence the suggestion to use a prefect.yaml
where you can specify the directory for the flow, ex:
entrypoint: flows/export/set_export_freq.py:set_export_freq
path: /opt/prefect/arte_tasks/
pull:
- prefect.deployments.steps.set_working_directory:
directory: /opt/prefect/arte_tasks
Devin Flake
01/28/2025, 9:18 PMprefect.yaml
and simply have all the config for the Docker image build in the .deploy()
since our Docker build is kind of complex needing to get env variables in order build correctly:
A_TOKEN = os.getenv("A_TOKEN")
set_export_freq.deploy(
name="set_export_freq",
work_pool_name="arte-k8s",
image=DockerImage(
name="docker-arte-release.dr-uw2.adobeitc.com/arte-flows-dflake",
tag="0.4",
dockerfile="containers/arte-flows/Dockerfile",
buildargs={"username": "arte", "pypitoken": A_TOKEN},
),
entrypoint_type="arte_tasks/flows/export/set_export_freq.py:set_export_freq",
parameters={"started_by": "dflake", "dry_run": True, "dpc_type": "prod", "verbose": True},
#cron="0 9 * * *",
description="Find rsids with > 1M daily hits. Set the export interval to 30 min if interval > 30 min.",
tags=["automation", "export"]
)
Nate
01/28/2025, 9:21 PMentrypoint_type
should probably be private, and you shouldnt have to set it. if you go to definition on that you'll see it defaults to an enum member which is literally "file_path"
https://github.com/prefecthq/prefect/blob/main/src/prefect/flows.py#L1095
whereas entrypoint
is the kwarg you'd set in from_source
if you're not baking your code into your imageNate
01/28/2025, 9:22 PMentrypoint_type
from your .deploy()
to start, and then maybe see if there's some pathing issue in the dockerfileDevin Flake
01/28/2025, 9:25 PMfrom_source
right? I'm not sure how the flow will be able to find the entrypoint thenDevin Flake
01/28/2025, 9:28 PMDevin Flake
01/28/2025, 9:32 PMNate
01/28/2025, 9:34 PM.deploy
if you're baking code into your image then you should only need to set your image
and then you can handle pathing inside the Dockerfile
itselfDevin Flake
01/28/2025, 9:34 PM