Hello all I have a deployment that runs a main flow that dep Prefect Community #ask-community

Hello all! I have a deployment that runs a main fl...

alex

03/31/2023, 4:22 PM

Hello all! I have a deployment that runs a main flow that deploys additional deployments. I was able to have the main deployment run after running into the issue here but my children deployments are failing. Does anyone have any recommendations on how to fix this? Details in thread.

alex

03/31/2023, 4:23 PM

main_flow.py

Copy code

from prefect import flow, task

from nested_deployments.child_flow import child_flow
from nested_deployments.deployments import get_deployment, apply_deployment


@task
def get_values():
    return [1, 2, 3]


@flow
def master_flow():
    values = get_values()

    for val in values:
        deploy = get_deployment(
            flow=child_flow,
            deployment_name=f"child-deployment-{val}",
            parameters={
                "val": val
            }
        )
        apply_deployment(deploy)


def deploy_main_flow():
    deploy = get_deployment(
        flow=master_flow,
        deployment_name="main-deployment",
        parameters={}
    )
    apply_deployment(deploy)


if __name__ == '__main__':
    deploy_main_flow()

child_flow.py

Copy code

from prefect import flow


@flow()
def child_flow(val: int):
    print(f"value is {val}")

deployments.py

Copy code

import os

from prefect.deployments import Deployment
from prefect.filesystems import LocalFileSystem
from prefect.settings import PREFECT_UI_URL


def get_ui_deployment_url(deployment_id) -> str:
    ui_url = PREFECT_UI_URL.value() or "<http://ephemeral-orion/api>"
    return f"{ui_url}/deployments/deployment/{deployment_id}"


def get_deployment(flow, deployment_name, parameters):
    base_path = os.getcwd()

    fs = LocalFileSystem(basepath=base_path)

    deployment = Deployment.build_from_flow(
        flow=flow,
        name=deployment_name,
        schedule=None,
        storage=fs,
        path=base_path,
        parameters=parameters,
        load_existing=False,
        skip_upload=True,
    )
    return deployment


def apply_deployment(deployment):
    _id = deployment.apply()

    url = get_ui_deployment_url(_id)
    print(f"Deployment '{deployment.name}' deployed. Url: {url}")

alex

03/31/2023, 4:24 PM

The child deployments are deployed successfully but fail with

Copy code

12:19:09.037 | ERROR   | Flow run 'hulking-salmon' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/Users/a/code/prefect-minimal/venv/lib/python3.9/site-packages/prefect/engine.py", line 262, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/Users/a/code/prefect-minimal/venv/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/a/code/prefect-minimal/venv/lib/python3.9/site-packages/prefect/deployments.py", line 170, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/Users/a/code/prefect-minimal/venv/lib/python3.9/site-packages/prefect/filesystems.py", line 143, in get_directory
    copytree(from_path, local_path, dirs_exist_ok=True)
  File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 566, in copytree
    with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/qh/3t4219m142g_k67hmw5nr7xm0000gp/T/tmps_epjldzprefect'

alex

03/31/2023, 4:32 PM

With a kubernetes run I get this error

Copy code

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 277, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.9/site-packages/prefect/client/utilities.py", line 40, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/deployments.py", line 194, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 147, in get_directory
    copytree(from_path, local_path, dirs_exist_ok=True)
  File "/usr/local/lib/python3.9/shutil.py", line 566, in copytree
    with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/home/deployment'

Open in Slack

Previous Next