https://prefect.io
Join Slack
Hi all - I'm running into some strange 502 errors with Prefect that happen seemingly at random (thou...
# ask-community
z
Hi all - I'm running into some strange 502 errors with Prefect that happen seemingly at random (though usually at the same time of the day). These are Airbyte tasks, but Airbyte doesn't actually fail. Prefect attempts to check the status 3 times (# of retries), gets a 502 each time, and fails the task. I know our EC2 is up and running at this time - it just fails on random days. Is there some kind of prefect cloud maintenance that happens during this time frame (~3:15am EST -> 3:40am EST) that cause API spottiness? (we've been experiencing this issue for several weeks) My only ideas for a solution are: ā€¢ change the retry delay to be 30 minutes ā€¢ Change the time that we run these flows to 4am EST Flow run id's for the below flows: 
952909cb-fc80-4942-8868-8995b83b00b7
, 
62a4e24c-c6ad-47ac-b017-6d0836c62b8c
Prefect version: 2.8.7 Full trace-back of an example task error included in thread
āœ… 1
kicking off task for connection name: 'InsuranceClaim.ConfirmationOfBenefitOriginalBenefitFactors' 031502 AM prd - Airbyte prefect.task_runs INFO Triggering Airbyte Connection 3124649e-b5e5-4b87-bfda-420e1d43a0cd, in workspace at 'http://ec2-100-24-158-3.compute-1.amazonaws.com:8000/api/v1' 031502 AM prd - Airbyte prefect.task_runs ERROR Encountered exception during execution: 032002 AM prd - Airbyte prefect.task_runs INFO kicking off task for connection name: 'InsuranceClaim.ConfirmationOfBenefitOriginalBenefitFactors' 032012 AM prd - Airbyte prefect.task_runs INFO Triggering Airbyte Connection 3124649e-b5e5-4b87-bfda-420e1d43a0cd, in workspace at 'http://ec2-100-24-158-3.compute-1.amazonaws.com:8000/api/v1' 032012 AM prd - Airbyte prefect.task_runs ERROR Encountered exception during execution: 032512 AM prd - Airbyte prefect.task_runs INFO kicking off task for connection name: 'InsuranceClaim.ConfirmationOfBenefitOriginalBenefitFactors' 032533 AM prd - Airbyte prefect.task_runs INFO Triggering Airbyte Connection 3124649e-b5e5-4b87-bfda-420e1d43a0cd, in workspace at 'http://ec2-100-24-158-3.compute-1.amazonaws.com:8000/api/v1' 032533 AM prd - Airbyte prefect.task_runs ERROR Encountered exception during execution: 033033 AM prd - Airbyte prefect.task_runs INFO kicking off task for connection name: 'InsuranceClaim.ConfirmationOfBenefitOriginalBenefitFactors' 033113 AM prd - Airbyte prefect.task_runs INFO Triggering Airbyte Connection 3124649e-b5e5-4b87-bfda-420e1d43a0cd, in workspace at 'http://ec2-100-24-158-3.compute-1.amazonaws.com:8000/api/v1' 033113 AM prd - Airbyte prefect.task_runs ERROR Encountered exception during execution: 033613 AM prd - Airbyte prefect.task_runs ERROR Finished in state Failed('Task run encountered an exception: Traceback (most recent call last):\n File "/home/airbyte/.local/lib/python3.8/site-packages/prefect_airbyte/client.py", line 154, in trigger_manual_sync_connection\n response.raise_for_status()\n File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_models.py", line 749, in raise_for_status\n raise HTTPStatusError(message, request=request, response=self)\nhttpx.HTTPStatusError: Server error \'502 Bad Gateway\' for url \'http://ec2-100-24-158-3.compute-1.amazonaws.com:8000/api/v1/connections/sync/\'\nFor more information check: https://httpstatuses.com/502\n\nThe above exception was the direct cause of the following exception:\n\nprefect_airbyte.exceptions.AirbyteServerNotHealthyException\n')
j
Hi, looking at the traceback it looks the the 502 themselves are not from Prefect Cloud. The URL that's failing to be hit looks like your hosted airbyte server? prefect (client) is failing the task because it can't reach the airbyte server after a certain number of retries
I would check your airbyte server logs for 502s or any other load balancers or anything in the network path to confirm
z
Interesting.. Yes I bet it's Airbyte throwing a 502 somewhere - I will take a look!
šŸ‘ 1
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by