Stéphan Taljaard
02/01/2023, 2:27 PMflow_run = await client.read_flow_run(flow_run_id)
It's sporadic; it does not happen to many but not all of my flow runs.Christopher Boyd
02/01/2023, 4:02 PMStéphan Taljaard
02/01/2023, 4:07 PM2a0672be-6e64-4081-bdfd-a4dfded5a802
Workspace: b70a86f1-659f-408f-add7-9665c6bfa327
Example flow run: d0a4a7f6-35e9-405f-a9a7-66d2751313a1
Deployment saved in a GCP Bucket; it's successfully pulled when running the deployment's flow
Infra: stock-standard Process block
Default task executor; no `.submit`ed tasks, so all are run immediately/sequentially
Some tasks are async, but used non-awaited inside the non-async flow func.
The flow runs sucessfully for 2 minutes, one of the last tasks is to read the deployment name using async OrionClient, which is where task run a4194641-d636-4f79-8d27-ebb6d1bf5e70
crashed, with this trace (failure at 04:05:25 PM UTC+2)PREFECT_API_REQUEST_TIMEOUT
to 600
https://pastebin.com/DgTStCFP@task
async def get_project_dir_name() -> str:
task_run_context = get_run_context()
flow_run_id = task_run_context.task_run.flow_run_id
async with prefect.client.orion.get_client() as client:
flow_run = await client.read_flow_run(flow_run_id)
flow_id = flow_run.flow_id
flow = await client.read_flow(flow_id)
flow_name = flow.name
if deployment_id := flow_run.deployment_id:
deployment = await client.read_deployment(deployment_id)
deployment_name = deployment.name
else:
deployment_name = "Testing"
dir_name = f"{flow_name}/{deployment_name}"
return project_name_to_project_dir(dir_name, allow_dir=True)
def project_name_to_project_dir(project_name: str, allow_dir: bool = False) -> str:
# This is the opposite of cli.register_flows.format_project_name
fn = sanitize_filepath if allow_dir else sanitize_filename
return fn(project_name.lower().replace(" ", "-"))
Christopher Boyd
02/01/2023, 5:23 PMStéphan Taljaard
02/01/2023, 5:27 PMChristopher Boyd
02/02/2023, 2:05 PMStéphan Taljaard
02/02/2023, 2:27 PMChristopher Boyd
02/02/2023, 2:28 PMStéphan Taljaard
02/02/2023, 2:28 PM8a8e26b7-423e-46ff-8618-aae5f359f274
, task run f3ee56c5-7e4e-4f41-8b07-73b919a0e0af
Here's my test flow I used to reproduce + its logs. I transferred over some functions from my real flows to kind of keep the steps the same. Still, the test flow is less complex workflow, uses less RAM, but higher CPU usage than my actual flows of yesterday.
It might seem like a CPU problem, but I don't think so, since my real ETL flows of yesterday had enough headroom left.