Joseph Haaga
11/20/2020, 5:31 PMFailed to load and execute Flow's environment: KeyError('my-flow-name')
The Flow is uses DockerStorage, and I’ve got an image containing only this Flow; are there some prefect
cli invocations I can do inside the container to debug this KeyError?nicholas
11/20/2020, 5:32 PMJoseph Haaga
11/20/2020, 5:44 PMbuild
and register
step into separate scripts (we’re trying to write a gold-standard template repo for devs to fork when writing Flows), but this is basically what’s going on
# src/flow.py
from prefect import Flow, task
@task
def task_one():
print("hi")
return 1
@task
def task_two(x):
return x + 1
with Flow("my-flow-name") as flow:
a = task_one()
b = task_two(a)
# build_flow.py
from src.flow import flow
from prefect.environments.storage import Docker
storage = Docker(
dockerfile='Dockerfile',
registry_url="<http://myregistry.com|myregistry.com>",
image_name="my-flow-name",
)
storage.add_flow(flow)
storage.build()
# register_flow.py
flow.storage = Docker(
registry_url=args.repo,
image_name="my-flow-name",
image_tag=args.tag,
)
flow.environment = DaskKubernetesEnvironment(
min_workers=1,
max_workers=3,
worker_spec_file="worker_spec.yaml"
)
flow.register(project_name=args.project, build=False, labels=labels)
nicholas
11/20/2020, 7:59 PMbuild_flow.py
and another in register_flow.py
- and it's a little tough to tell but it looks like the actual flow you're registering doesn't ever have its storage builtJoseph Haaga
11/20/2020, 8:02 PMregister_flow.py
and are now encountering another (albeit more encouraging) error:
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'src'")
We usually just write a setup.py
and RUN pip install -e .
to circumvent this issue, but curious if anyone knows a better way to resolve this!nicholas
11/20/2020, 8:06 PMsrc
directory in your dockerfile? if not, it won't be present when you try to run your flow in the containerJoseph Haaga
11/20/2020, 8:14 PMnicholas
11/20/2020, 9:05 PMJoseph Haaga
11/20/2020, 9:22 PMnicholas
11/20/2020, 9:24 PM__init__.py
file in that directory?