Thread
#prefect-community
    David Yang

    David Yang

    5 months ago
    I have a master flow that executes other flows. It runs well so far using git storage. but it failed with this error message: "No heartbeat detected from the remote task; marking the run as failed" tired to first sub flow called in the master flow. Only one line code changed in the master flow. Any suggestions?
    Kevin Kho

    Kevin Kho

    5 months ago
    Are you using threaded heartbeats already?
    David Yang

    David Yang

    5 months ago
    same error. It seems that sub flows are not kicked off.
    Kevin Kho

    Kevin Kho

    5 months ago
    That’s pretty weird. Does the subflow kick off and could I see your code?
    David Yang

    David Yang

    5 months ago
    no... true. It's weird. All I did is just the master flow's storage change. One line code and it run perfectly before the change. Code is pretty simple. progject_name ="DEV" flow_storage = Azure(container="prefectflows") with Flow("EDW_Master_Flow", schedule=weekday_schedule, executor=LocalDaskExecutor(), storage= flow_storage, run_config = DockerRun(image=image_url,env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"}) ) as flow: flow_GeoLocationCal = create_flow_run(flow_name="xxxxxx", project_name=progject_name,task_args={"name": "Run xxxxx Calculation"}) wait_for_flow_GeoLocationCal = wait_for_flow_run(flow_GeoLocationCal,stream_logs=True, raise_final_state=True) flow_dbt = create_flow_run(flow_name="xxxxx", project_name=progject_name, task_args={"name": "Run dbt tasks"}) wait_for_flow_dbt = wait_for_flow_run(flow_dbt,stream_logs=True, raise_final_state=True) flow_dbt.set_upstream(wait_for_flow_GeoLocationCal) wait_for_flow_dbt.set_upstream(flow_dbt) if name == "main": flow.schedule = None flow.run()
    Kevin Kho

    Kevin Kho

    5 months ago
    Is it the create_flow_run or wait_for_flow_run that dies?
    What was the storage before you changed it? Does reverting it back will work?
    David Yang

    David Yang

    5 months ago
    create_flow_run step. because I could not see the sub flow run in the prefect cloud. It works after I revert the storage back. flow_storage = Git( repo="xxxxxxxx",

    name of repo

    flow_path="./flows/xxxxxx.py", repo_host="xxxxxxx",

    repo host name

    branch_name= "main", git_clone_url_secret_name="azureflowurl" ) # flow_storage = Azure(container="prefectflows")
    Kevin Kho

    Kevin Kho

    5 months ago
    So it seems like something is up with the Azure storage and maybe causing it to hang? Does your flow exist in Azure when you go there?
    David Yang

    David Yang

    5 months ago
    yeah....
    my filename is EDW_Master_Flow.py
    I also checked the docker image environment variable. AZURE_STORAGE_CONNECTION_STRING is configured properly.
    Anna Geller

    Anna Geller

    5 months ago
    In general, flow of flows is primarily a backend feature so local flow run is a bit tricky. Can you try registering your child flows, then the parent flow and run it with the backend?
    Also, make sure that each flow, both child and parent flow, have their own storage file. I remember having similar issue as you do when having both child and parent flow defined within the same file
    David Yang

    David Yang

    5 months ago
    They have their own storage file and they have own docker images. I still got the same error.
    Anna Geller

    Anna Geller

    5 months ago
    I see - can you share a simplified flow code for both? alternatively, can you check if following this example helps you configure your flow of flows: https://github.com/anna-geller/orchestrator-pattern/tree/master/simple_parent_child_example?
    Kevin Kho

    Kevin Kho

    5 months ago
    I think something might be hanging when you pull from Azure. Can you try going into your container and seeing if you can pull the file? If you have Python, you can do something like:
    storage = Azure(...)
    storage.get_flow("flow_name:)
    David Yang

    David Yang

    5 months ago
    @Kevin Kho, no flows in the storage. storage.flows returns empty list.
    Child flows use azure storage as well and they run successfully
    Kevin Kho

    Kevin Kho

    5 months ago
    That’s very weird. I can’t pinpoint immediately. When you run the child Flow, is that the same agent?
    David Yang

    David Yang

    5 months ago
    @Kevin Kho, yes same agent. An azure VM runs prefect agent. and all flows run on the same agent and on dedicated docker container. All flows use azure storage. If I switch the master flow's storage from azure to git only, one line code change, it works perfectly. I could change it back to git storage, but do like to know the root cause.
    Kevin Kho

    Kevin Kho

    5 months ago
    Is your Azure script based as well or does it use serialized?
    David Yang

    David Yang

    5 months ago
    use serialized.
    Kevin Kho

    Kevin Kho

    5 months ago
    Ok I assume it will work with using
    stored_as_script
    ? Is there a chance your registration versions of Python/Prefect/cloudpickle are different from the execution versions?
    David Yang

    David Yang

    5 months ago
    BTW, the azure blob storage is ADLS Gen2
    Python version is different. registration uses 3.8 and execution is 3.9
    Kevin Kho

    Kevin Kho

    5 months ago
    That should throw an error. Am surprised it’s not but could you try aligning the versions?
    I am looking in ADLS Gen2. I dont know off the top of my head if the storage is written for Gen1 or Gen2 or both
    David Yang

    David Yang

    5 months ago
    No error if I use git. and the log shows that Prefect try to run child flow before downloading the master flow. 14-Apr-22 03:44:41 EDT agent INFO Submitted for execution: Container ID: fd21f7253cbe0190a1b39bd2790b60db17e62997bd841ea93743f548a70b9453 14-Apr-22 03:59:38 EDT prefect-server.ZombieKiller.TaskRun ERROR No heartbeat detected from the remote task; marking the run as failed. 14-Apr-22 04:06:03 EDT prefect-server.Lazarus.FlowRun INFO Rescheduled by a Lazarus process. This is attempt 1. 14-Apr-22 04:06:07 EDT agent INFO Submitted for execution: Container ID: 7ab95a04b9da7ff0b4d618facfbf3d67a41c5009218648ff28d41ff0a5df1e6a 14-Apr-22 04:21:19 EDT prefect-server.ZombieKiller.TaskRun ERROR No heartbeat detected from the remote task; marking the run as failed. 14-Apr-22 04:28:35 EDT prefect-server.Lazarus.FlowRun INFO Rescheduled by a Lazarus process. This is attempt 2. 14-Apr-22 04:28:43 EDT agent INFO Submitted for execution: Container ID: 7210573b4897cb50e0e888d5a5e482c77e6986815793324cdba206580490b8c4 14-Apr-22 04:28:44 EDT prefect.Azure INFO Downloading edwmaster/2022-04-14t19-40-46-492607-00-00 from prefectflows 14-Apr-22 04:28:44 EDT prefect.CloudFlowRunner INFO Beginning Flow run for 'edwmaster' 14-Apr-22 04:28:45 EDT prefect.CloudTaskRunner INFO Task 'wait_for_flow_run': Starting task run... 14-Apr-22 04:28:45 EDT prefect.CloudTaskRunner INFO Task 'wait_for_flow_run': Finished task run for task with final state: 'TriggerFailed' 14-Apr-22 04:28:45 EDT prefect.CloudFlowRunner INFO Flow run FAILED: some reference tasks failed.
    Kevin Kho

    Kevin Kho

    5 months ago
    No no I mean you can still use Script based storage on Azure by doing
    stored_as_script=True
    Been looking and am positive the current Azure storage should work for ADLS Gen2. And the fact that the Flow upload succeeded makes me more confident
    David Yang

    David Yang

    5 months ago
    @Kevin Kho, I got this error after I did these steps: 1: change the code to use script. storage= Azure(container="prefectflows",stored_as_script=True,blob_name="EDWMasterFlow.py",overwrite=True) 2: upload the flow file to the container manually. The error is: Failed to load and execute flow run: NameError("name 'file' is not defined")