Hi all - I'm running into some strange 502 errors ...
# ask-communityz
Zachary Loertscher
09/19/2023, 1:42 PMHi all - I'm running into some strange 502 errors with Prefect that happen seemingly at random (though usually at the same time of the day).
These are Airbyte tasks, but Airbyte doesn't actually fail. Prefect attempts to check the status 3 times (# of retries), gets a 502 each time, and fails the task.
I know our EC2 is up and running at this time - it just fails on random days. Is there some kind of prefect cloud maintenance that happens during this time frame (~3:15am EST -> 3:40am EST) that cause API spottiness? (we've been experiencing this issue for several weeks)
My only ideas for a solution are:
ā¢ change the retry delay to be 30 minutes
ā¢ Change the time that we run these flows to 4am EST
Flow run id's for the below flows:
952909cb-fc80-4942-8868-8995b83b00b762a4e24c-c6ad-47ac-b017-6d0836c62b8cā
1
Zachary Loertscher
09/19/2023, 1:42 PMkicking off task for connection name: 'InsuranceClaim.ConfirmationOfBenefitOriginalBenefitFactors'
031502 AM
prd - Airbyte
prefect.task_runs
INFO
Triggering Airbyte Connection 3124649e-b5e5-4b87-bfda-420e1d43a0cd, in workspace at 'http://ec2-100-24-158-3.compute-1.amazonaws.com:8000/api/v1'
031502 AM
prd - Airbyte
prefect.task_runs
ERROR
Encountered exception during execution:
032002 AM
prd - Airbyte
prefect.task_runs
INFO
kicking off task for connection name: 'InsuranceClaim.ConfirmationOfBenefitOriginalBenefitFactors'
032012 AM
prd - Airbyte
prefect.task_runs
INFO
Triggering Airbyte Connection 3124649e-b5e5-4b87-bfda-420e1d43a0cd, in workspace at 'http://ec2-100-24-158-3.compute-1.amazonaws.com:8000/api/v1'
032012 AM
prd - Airbyte
prefect.task_runs
ERROR
Encountered exception during execution:
032512 AM
prd - Airbyte
prefect.task_runs
INFO
kicking off task for connection name: 'InsuranceClaim.ConfirmationOfBenefitOriginalBenefitFactors'
032533 AM
prd - Airbyte
prefect.task_runs
INFO
Triggering Airbyte Connection 3124649e-b5e5-4b87-bfda-420e1d43a0cd, in workspace at 'http://ec2-100-24-158-3.compute-1.amazonaws.com:8000/api/v1'
032533 AM
prd - Airbyte
prefect.task_runs
ERROR
Encountered exception during execution:
033033 AM
prd - Airbyte
prefect.task_runs
INFO
kicking off task for connection name: 'InsuranceClaim.ConfirmationOfBenefitOriginalBenefitFactors'
033113 AM
prd - Airbyte
prefect.task_runs
INFO
Triggering Airbyte Connection 3124649e-b5e5-4b87-bfda-420e1d43a0cd, in workspace at 'http://ec2-100-24-158-3.compute-1.amazonaws.com:8000/api/v1'
033113 AM
prd - Airbyte
prefect.task_runs
ERROR
Encountered exception during execution:
033613 AM
prd - Airbyte
prefect.task_runs
ERROR
Finished in state Failed('Task run encountered an exception: Traceback (most recent call last):\n File "/home/airbyte/.local/lib/python3.8/site-packages/prefect_airbyte/client.py", line 154, in trigger_manual_sync_connection\n response.raise_for_status()\n File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_models.py", line 749, in raise_for_status\n raise HTTPStatusError(message, request=request, response=self)\nhttpx.HTTPStatusError: Server error \'502 Bad Gateway\' for url \'http://ec2-100-24-158-3.compute-1.amazonaws.com:8000/api/v1/connections/sync/\'\nFor more information check: https://httpstatuses.com/502\n\nThe above exception was the direct cause of the following exception:\n\nprefect_airbyte.exceptions.AirbyteServerNotHealthyException\n')
j
Jake Kaplan
09/19/2023, 2:30 PMHi, looking at the traceback it looks the the 502 themselves are not from Prefect Cloud.
The URL that's failing to be hit looks like your hosted airbyte server? prefect (client) is failing the task because it can't reach the airbyte server after a certain number of retries
Jake Kaplan
09/19/2023, 2:31 PMI would check your airbyte server logs for 502s or any other load balancers or anything in the network path to confirm
z
Zachary Loertscher
09/19/2023, 2:31 PMInteresting.. Yes I bet it's Airbyte throwing a 502 somewhere - I will take a look!
š 1
4 Views