https://prefect.io logo
Title
d

David Yang

04/13/2022, 3:49 PM
I have a master flow that executes other flows. It runs well so far using git storage. but it failed with this error message: "No heartbeat detected from the remote task; marking the run as failed" tired to first sub flow called in the master flow. Only one line code changed in the master flow. Any suggestions?
k

Kevin Kho

04/13/2022, 3:50 PM
Are you using threaded heartbeats already?
d

David Yang

04/13/2022, 5:28 PM
same error. It seems that sub flows are not kicked off.
k

Kevin Kho

04/13/2022, 5:29 PM
That’s pretty weird. Does the subflow kick off and could I see your code?
d

David Yang

04/13/2022, 5:34 PM
no... true. It's weird. All I did is just the master flow's storage change. One line code and it run perfectly before the change. Code is pretty simple. progject_name ="DEV" flow_storage = Azure(container="prefectflows") with Flow("EDW_Master_Flow", schedule=weekday_schedule, executor=LocalDaskExecutor(), storage= flow_storage, run_config = DockerRun(image=image_url,env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"}) ) as flow: flow_GeoLocationCal = create_flow_run(flow_name="xxxxxx", project_name=progject_name,task_args={"name": "Run xxxxx Calculation"}) wait_for_flow_GeoLocationCal = wait_for_flow_run(flow_GeoLocationCal,stream_logs=True, raise_final_state=True) flow_dbt = create_flow_run(flow_name="xxxxx", project_name=progject_name, task_args={"name": "Run dbt tasks"}) wait_for_flow_dbt = wait_for_flow_run(flow_dbt,stream_logs=True, raise_final_state=True) flow_dbt.set_upstream(wait_for_flow_GeoLocationCal) wait_for_flow_dbt.set_upstream(flow_dbt) if name == "__main__": flow.schedule = None flow.run()
k

Kevin Kho

04/13/2022, 5:37 PM
Is it the create_flow_run or wait_for_flow_run that dies?
What was the storage before you changed it? Does reverting it back will work?
d

David Yang

04/13/2022, 5:46 PM
create_flow_run step. because I could not see the sub flow run in the prefect cloud. It works after I revert the storage back. flow_storage = Git( repo="xxxxxxxx", # name of repo flow_path="./flows/xxxxxx.py", repo_host="xxxxxxx", # repo host name branch_name= "main", git_clone_url_secret_name="azureflowurl" ) _# flow_storage = Azure(container="prefectflows")_
k

Kevin Kho

04/13/2022, 5:58 PM
So it seems like something is up with the Azure storage and maybe causing it to hang? Does your flow exist in Azure when you go there?
d

David Yang

04/13/2022, 5:59 PM
yeah....
my filename is EDW_Master_Flow.py
I also checked the docker image environment variable. AZURE_STORAGE_CONNECTION_STRING is configured properly.
a

Anna Geller

04/13/2022, 6:28 PM
In general, flow of flows is primarily a backend feature so local flow run is a bit tricky. Can you try registering your child flows, then the parent flow and run it with the backend?
Also, make sure that each flow, both child and parent flow, have their own storage file. I remember having similar issue as you do when having both child and parent flow defined within the same file
d

David Yang

04/14/2022, 6:02 PM
They have their own storage file and they have own docker images. I still got the same error.
a

Anna Geller

04/14/2022, 6:11 PM
I see - can you share a simplified flow code for both? alternatively, can you check if following this example helps you configure your flow of flows: https://github.com/anna-geller/orchestrator-pattern/tree/master/simple_parent_child_example?
k

Kevin Kho

04/14/2022, 6:12 PM
I think something might be hanging when you pull from Azure. Can you try going into your container and seeing if you can pull the file? If you have Python, you can do something like:
storage = Azure(...)
storage.get_flow("flow_name:)
d

David Yang

04/14/2022, 7:47 PM
@Kevin Kho, no flows in the storage. storage.flows returns empty list.
Child flows use azure storage as well and they run successfully
k

Kevin Kho

04/14/2022, 7:49 PM
That’s very weird. I can’t pinpoint immediately. When you run the child Flow, is that the same agent?
d

David Yang

04/14/2022, 7:57 PM
@Kevin Kho, yes same agent. An azure VM runs prefect agent. and all flows run on the same agent and on dedicated docker container. All flows use azure storage. If I switch the master flow's storage from azure to git only, one line code change, it works perfectly. I could change it back to git storage, but do like to know the root cause.
k

Kevin Kho

04/14/2022, 7:58 PM
Is your Azure script based as well or does it use serialized?
d

David Yang

04/14/2022, 8:26 PM
use serialized.
k

Kevin Kho

04/14/2022, 8:33 PM
Ok I assume it will work with using
stored_as_script
? Is there a chance your registration versions of Python/Prefect/cloudpickle are different from the execution versions?
d

David Yang

04/14/2022, 8:34 PM
BTW, the azure blob storage is ADLS Gen2
Python version is different. registration uses 3.8 and execution is 3.9
k

Kevin Kho

04/14/2022, 8:37 PM
That should throw an error. Am surprised it’s not but could you try aligning the versions?
I am looking in ADLS Gen2. I dont know off the top of my head if the storage is written for Gen1 or Gen2 or both
d

David Yang

04/14/2022, 8:40 PM
No error if I use git. and the log shows that Prefect try to run child flow before downloading the master flow. 14-Apr-22 03:44:41 EDT agent INFO Submitted for execution: Container ID: fd21f7253cbe0190a1b39bd2790b60db17e62997bd841ea93743f548a70b9453 14-Apr-22 03:59:38 EDT prefect-server.ZombieKiller.TaskRun ERROR No heartbeat detected from the remote task; marking the run as failed. 14-Apr-22 04:06:03 EDT prefect-server.Lazarus.FlowRun INFO Rescheduled by a Lazarus process. This is attempt 1. 14-Apr-22 04:06:07 EDT agent INFO Submitted for execution: Container ID: 7ab95a04b9da7ff0b4d618facfbf3d67a41c5009218648ff28d41ff0a5df1e6a 14-Apr-22 04:21:19 EDT prefect-server.ZombieKiller.TaskRun ERROR No heartbeat detected from the remote task; marking the run as failed. 14-Apr-22 04:28:35 EDT prefect-server.Lazarus.FlowRun INFO Rescheduled by a Lazarus process. This is attempt 2. 14-Apr-22 04:28:43 EDT agent INFO Submitted for execution: Container ID: 7210573b4897cb50e0e888d5a5e482c77e6986815793324cdba206580490b8c4 14-Apr-22 04:28:44 EDT prefect.Azure INFO Downloading edwmaster/2022-04-14t19-40-46-492607-00-00 from prefectflows 14-Apr-22 04:28:44 EDT prefect.CloudFlowRunner INFO Beginning Flow run for 'edwmaster' 14-Apr-22 04:28:45 EDT prefect.CloudTaskRunner INFO Task 'wait_for_flow_run': Starting task run... 14-Apr-22 04:28:45 EDT prefect.CloudTaskRunner INFO Task 'wait_for_flow_run': Finished task run for task with final state: 'TriggerFailed' 14-Apr-22 04:28:45 EDT prefect.CloudFlowRunner INFO Flow run FAILED: some reference tasks failed.
k

Kevin Kho

04/14/2022, 8:43 PM
No no I mean you can still use Script based storage on Azure by doing
stored_as_script=True
Been looking and am positive the current Azure storage should work for ADLS Gen2. And the fact that the Flow upload succeeded makes me more confident
d

David Yang

04/19/2022, 1:25 PM
@Kevin Kho, I got this error after I did these steps: 1: change the code to use script. storage= Azure(container="prefectflows",stored_as_script=True,blob_name="EDWMasterFlow.py",overwrite=True) 2: upload the flow file to the container manually. The error is: Failed to load and execute flow run: NameError("name '__file__' is not defined")