Naga Sravika Bodapati
05/02/2022, 11:36 AMAnna Geller
05/02/2022, 11:46 AMCan u name a scenario/reason why this could happen?1. Not enough memory on your VM/server 2. Not enough memory allocated to Kubernetes pod or Docker container, ECS task 3. General execution layer issues outside of Prefect I could name more but not sure this would be helpful. Can you explain your use case more? Long-running jobs are generally difficult to troubleshoot, check this for more background
Naga Sravika Bodapati
05/02/2022, 12:02 PMAnna Geller
05/02/2022, 12:04 PMhow will making change with respect to heart beat address the issue?by default, it uses processes, and threads work better for long-running jobs, that's the only reason
Naga Sravika Bodapati
05/02/2022, 12:06 PMflow.run_config = LocalRun(labels=["<agent>"])
flow.storage = GCS(bucket="<bucketname>")
Anna Geller
05/02/2022, 12:07 PMThe flows make a connection to our databases in a loop and fetch new/updated data rows into big querythe reason I'm asking about the flow code - we've seen similar issues when a user was sharing DB connections between tasks. This is not supported in Prefect 2.0 unless you use a resource manager https://docs.prefect.io/core/idioms/resource-manager.html
Naga Sravika Bodapati
05/02/2022, 12:35 PMAnna Geller
05/02/2022, 2:10 PMwith Flow(flow_name, schedule=schedule) as flow:
flow.run_config = LocalRun(labels=[""])
flow.storage = GCS(bucket="")
Try:
with Flow(
flow_name,
schedule=schedule,
run_config=LocalRun(labels=["your_host_name"]),
storage=GCS(bucket="your_bucket"),
) as flow:
the label of empty string seems weird - are you sure you explicitly created a label with an empty string? if you didn't specify any labels on a local agent, usually this would need to be the name of the host (your VM/server name)if __name__ == '__main__':
flow.register(project_name=project_name)
Naga Sravika Bodapati
05/02/2022, 2:31 PMAnna Geller
05/02/2022, 2:50 PM