Georg Zangl
10/18/2020, 9:04 AM[program:fhn]
command=sudo prefect agent start local -p /usr/dcc/fhn/data_import -p /usr/dcc/fhn/calc_td -p /usr/dcc/fhn/vol_back_all -f
I can run only two flows, the third one fails with the message "Failed to load and execute Flow's environment: ModuleNotFoundError".
It doesn't matter which flow is third, it is always the last one, which fails.
So I have created two separate programs:
[program:fhn1]
command=sudo prefect agent start local -p /usr/dcc/fhn/data_import -p /usr/dcc/fhn/calc_td -f -l Import
[program:fhn_2]
command=prefect agent start local -p /usr/dcc/fhn/vol_back_all -f -l Vol
to handle all three flows.
But now, the success of the flows is unstable. Sometimes they fail with the "Failed to load.." message, sometimes they succeed. Even if I trigger them manually, they sometimes fail and sometimes succeed. More than 50% of flow runs fail.
Here is a log from supervisord:
2020-10-18 08:55:05,888 DEBG 'fhn1' stdout output:
[2020-10-18 08:55:05] INFO - prefect.CloudFlowRunner | Beginning Flow run for 'Calculate Equations FHN'
2020-10-18 08:55:05,903 DEBG 'fhn1' stdout output:
[2020-10-18 08:55:05] ERROR - prefect.Local | Failed to load Flow from /usr/dcc/fhn/flows/back-allocation-fhn.prefect
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/prefect/environments/storage/local.py", line 103, in get_flow
return prefect.core.flow.Flow.load(flow_location)
File "/usr/local/lib/python3.7/dist-packages/prefect/core/flow.py", line 1495, in load
return cloudpickle.load(f)
File "/usr/local/lib/python3.7/dist-packages/cloudpickle/cloudpickle.py", line 562, in subimport
__import__(name)
ModuleNotFoundError: No module named 'read_wt'
2020-10-18 08:55:05,954 DEBG 'fhn1' stdout output:
No module named 'read_wt'
The two agents are running fine. All code and agents are local, everything runs on one machine. I am running looped tasks in each of the flow.
Here is the code for one of the flows:
flow = Flow("Data Import FHN")
flow.set_dependencies(loop_conns, keyword_tasks={"iloop": looplist}, mapped=True)
with Flow("Data Import FHN") as flow:
connect = db_conn()
mapped_result =loop_conns.map(iloop=looplist)
flow.storage = Local(directory="/usr/dcc/fhn/flows")
flow.storage.build()
flow.register(project_name="FHN")
Any help would be appreciate to understand the unstable performance better.Spencer
10/18/2020, 1:00 PMread_wt
is missing. Verify that the necessary packages are installed for the python that each agent is using.Georg Zangl
10/18/2020, 2:11 PMSpencer
10/18/2020, 3:54 PMGeorg Zangl
10/18/2020, 4:09 PMnicholas
10/19/2020, 3:08 PMGeorg Zangl
10/19/2020, 7:48 PMnicholas
10/19/2020, 7:48 PMGeorg Zangl
10/20/2020, 6:35 AMGeorg Zangl
10/20/2020, 2:16 PMcommand=sudo prefect agent start local -p /usr/dcc/fhn/data_import -p /usr/dcc/fhn/vol_back_all -p /usr/dcc/fhn/calc_td -f -l FHN
only the first two flows run
If I split it into two agents (another example):
[program:platform_1]
command=sudo prefect agent start local -p /usr/dcc/platform/data_import -p /usr/dcc/platform/calc_td -f -l Platform1
[program:platform_2]
command=sudo prefect agent start local -p /usr/dcc/platform/vol_back_all -f -l Platform2
only two flows run. Can't get the third flow running. If I switch directories, behavior is the same. Always the flow of the second (in case of two agents) or third (in case of one agent) directory will fail.
Now I'm running one directory/flow per agent, this works fine____ __ _ _ _
| _ \ _ __ ___ / _| ___ ___| |_ / \ __ _ ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __| / _ \ / _` |/ _ \ '_ \| __|
| __/| | | __/ _| __/ (__| |_ / ___ \ (_| | __/ | | | |_
|_| |_| \___|_| \___|\___|\__| /_/ \_\__, |\___|_| |_|\__|
|___/
[2020-10-20 06:56:26,133] INFO - agent | Starting LocalAgent with labels ['FHN1', 'ip-10-0-9-4', 'azure-flow-storage', 'gcs-flow-storage', 's3-flow-storage', 'github-flow-storage', 'webhook-flow-storage', 'gitlab-flow-storage']
[2020-10-20 06:56:26,133] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
2020-10-20 06:56:26,134 DEBG 'fhn_1' stdout output:
[2020-10-20 06:56:26,133] INFO - agent | Agent connecting to the Prefect API at <http://localhost:4200>
This is the message of the agent when it starts, no mentioning of the pathnicholas
10/20/2020, 2:34 PMGeorg Zangl
10/20/2020, 4:27 PM