Thread
#prefect-server
    Joseph Haaga

    Joseph Haaga

    1 year ago
    Getting the following error when attempting to run my Flow
    Failed to load and execute Flow's environment: KeyError('my-flow-name')
    The Flow is uses DockerStorage, and I’ve got an image containing only this Flow; are there some
    prefect
    cli invocations I can do inside the container to debug this KeyError?
    nicholas

    nicholas

    1 year ago
    Hi @Joseph Haaga - can you provide some more context to this and perhaps some minimum amount of code?
    Joseph Haaga

    Joseph Haaga

    1 year ago
    yeah it’s a little hard since we’ve actually broken out the
    build
    and
    register
    step into separate scripts (we’re trying to write a gold-standard template repo for devs to fork when writing Flows), but this is basically what’s going on
    # src/flow.py
    from prefect import Flow, task
    
    
    @task
    def task_one():
        print("hi")
        return 1
    
    
    @task
    def task_two(x):
        return x + 1
    
    
    with Flow("my-flow-name") as flow:
        a = task_one()
        b = task_two(a)
    
    
    
    # build_flow.py
    from src.flow import flow
    from prefect.environments.storage import Docker
    
    storage = Docker(
        dockerfile='Dockerfile',
        registry_url="<http://myregistry.com|myregistry.com>",
        image_name="my-flow-name",
    )
    storage.add_flow(flow)
    storage.build()
    
    
    
    # register_flow.py
    flow.storage = Docker(
        registry_url=args.repo,
        image_name="my-flow-name",
        image_tag=args.tag,
    )
    
    flow.environment = DaskKubernetesEnvironment(
        min_workers=1,
        max_workers=3,
        worker_spec_file="worker_spec.yaml"
    )
    flow.register(project_name=args.project, build=False, labels=labels)
    nicholas

    nicholas

    1 year ago
    @Joseph Haaga it looks like you've got 2 storage objects between those scripts, 1 in
    build_flow.py
    and another in
    register_flow.py
    - and it's a little tough to tell but it looks like the actual flow you're registering doesn't ever have its storage built
    Joseph Haaga

    Joseph Haaga

    1 year ago
    yeah I share your suspicion - we’ve consolidated everything into
    register_flow.py
    and are now encountering another (albeit more encouraging) error:
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'src'")
    We usually just write a
    setup.py
    and
    RUN pip install -e .
    to circumvent this issue, but curious if anyone knows a better way to resolve this!
    nicholas

    nicholas

    1 year ago
    Hm, are you explicitly creating the
    src
    directory in your dockerfile? if not, it won't be present when you try to run your flow in the container
    Joseph Haaga

    Joseph Haaga

    1 year ago
    oh yeah we’re COPYing src in to the container
    nicholas

    nicholas

    1 year ago
    So is this failing at the build step or when running?
    Joseph Haaga

    Joseph Haaga

    1 year ago
    when running - everything builds and registers nicely, and these error messages are pulled from the UI on page for the FLow Run
    nicholas

    nicholas

    1 year ago
    gotcha, that's a little odd - do you have an
    __init__.py
    file in that directory?