I have a master flow that executes other flows. It runs well so far using git storage. but it failed with this error message: "No heartbeat detected from the remote task; marking the run as failed" tired to first sub flow called in the master flow. Only one line code changed in the master flow. Any suggestions?

Are you using threaded heartbeats already?

same error. It seems that sub flows are not kicked off.

That’s pretty weird. Does the subflow kick off and could I see your code?

no... true. It's weird. All I did is just the master flow's storage change. One line code and it run perfectly before the change. Code is pretty simple. progject_name ="DEV" flow_storage = Azure(container="prefectflows") with Flow("EDW_Master_Flow", schedule=weekday_schedule, executor=LocalDaskExecutor(), storage= flow_storage, run_config = DockerRun(image=image_url,env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"}) ) as flow: flow_GeoLocationCal = create_flow_run(flow_name="xxxxxx", project_name=progject_name,task_args={"name": "Run xxxxx Calculation"}) wait_for_flow_GeoLocationCal = wait_for_flow_run(flow_GeoLocationCal,stream_logs=True, raise_final_state=True) flow_dbt = create_flow_run(flow_name="xxxxx", project_name=progject_name, task_args={"name": "Run dbt tasks"}) wait_for_flow_dbt = wait_for_flow_run(flow_dbt,stream_logs=True, raise_final_state=True) flow_dbt.set_upstream(wait_for_flow_GeoLocationCal) wait_for_flow_dbt.set_upstream(flow_dbt) if name == "__main__": flow.schedule = None flow.run()

Is it the create_flow_run or wait_for_flow_run that dies?

create_flow_run step. because I could not see the sub flow run in the prefect cloud. It works after I revert the storage back. flow_storage = Git( repo="xxxxxxxx", # name of repo flow_path="./flows/xxxxxx.py", repo_host="xxxxxxx", # repo host name branch_name= "main", git_clone_url_secret_name="azureflowurl" ) _# flow_storage = Azure(container="prefectflows")_

So it seems like something is up with the Azure storage and maybe causing it to hang? Does your flow exist in Azure when you go there?

yeah....

In general, flow of flows is primarily a backend feature so local flow run is a bit tricky. Can you try registering your child flows, then the parent flow and run it with the backend?

They have their own storage file and they have own docker images. I still got the same error.

I see - can you share a simplified flow code for both? alternatively, can you check if following this example helps you configure your flow of flows: https://github.com/anna-geller/orchestrator-pattern/tree/master/simple_parent_child_example?

I think something might be hanging when you pull from Azure. Can you try going into your container and seeing if you can pull the file? If you have Python, you can do something like:

storage = Azure(...)
storage.get_flow("flow_name:)

@Kevin Kho, no flows in the storage. storage.flows returns empty list.

That’s very weird. I can’t pinpoint immediately. When you run the child Flow, is that the same agent?

@Kevin Kho, yes same agent. An azure VM runs prefect agent. and all flows run on the same agent and on dedicated docker container. All flows use azure storage. If I switch the master flow's storage from azure to git only, one line code change, it works perfectly. I could change it back to git storage, but do like to know the root cause.

Is your Azure script based as well or does it use serialized?

use serialized.

Ok I assume it will work with using

stored_as_script

? Is there a chance your registration versions of Python/Prefect/cloudpickle are different from the execution versions?

BTW, the azure blob storage is ADLS Gen2

That should throw an error. Am surprised it’s not but could you try aligning the versions?

No error if I use git. and the log shows that Prefect try to run child flow before downloading the master flow. 14-Apr-22 034441 EDT agent INFO Submitted for execution: Container ID: fd21f7253cbe0190a1b39bd2790b60db17e62997bd841ea93743f548a70b9453 14-Apr-22 035938 EDT prefect-server.ZombieKiller.TaskRun ERROR No heartbeat detected from the remote task; marking the run as failed. 14-Apr-22 040603 EDT prefect-server.Lazarus.FlowRun INFO Rescheduled by a Lazarus process. This is attempt 1. 14-Apr-22 040607 EDT agent INFO Submitted for execution: Container ID: 7ab95a04b9da7ff0b4d618facfbf3d67a41c5009218648ff28d41ff0a5df1e6a 14-Apr-22 042119 EDT prefect-server.ZombieKiller.TaskRun ERROR No heartbeat detected from the remote task; marking the run as failed. 14-Apr-22 042835 EDT prefect-server.Lazarus.FlowRun INFO Rescheduled by a Lazarus process. This is attempt 2. 14-Apr-22 042843 EDT agent INFO Submitted for execution: Container ID: 7210573b4897cb50e0e888d5a5e482c77e6986815793324cdba206580490b8c4 14-Apr-22 042844 EDT prefect.Azure INFO Downloading edwmaster/2022-04-14t19-40-46-492607-00-00 from prefectflows 14-Apr-22 042844 EDT prefect.CloudFlowRunner INFO Beginning Flow run for 'edwmaster' 14-Apr-22 042845 EDT prefect.CloudTaskRunner INFO Task 'wait_for_flow_run': Starting task run... 14-Apr-22 042845 EDT prefect.CloudTaskRunner INFO Task 'wait_for_flow_run': Finished task run for task with final state: 'TriggerFailed' 14-Apr-22 042845 EDT prefect.CloudFlowRunner INFO Flow run FAILED: some reference tasks failed.

No no I mean you can still use Script based storage on Azure by doing

stored_as_script=True

@Kevin Kho, I got this error after I did these steps: 1: change the code to use script. storage= Azure(container="prefectflows",stored_as_script=True,blob_name="EDWMasterFlow.py",overwrite=True) 2: upload the flow file to the container manually. The error is: Failed to load and execute flow run: NameError("name '__file__' is not defined")