https://prefect.io logo
Powered by
Title
r

Ross Teach

09/02/2022, 1:21 PM
Using Prefect V2 Cloud, noticed a few of the following errors (8AM EST + 9AM EST). Seems to be intermittent. Is there a known issue? 
prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/043b2649-9d07-4c5e-8225-521ba2275e68/workspaces/689b139b-a725-4c2b-b167-86a705b8789d/task_runs/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
✅ 1
j

Jeff Hale

09/02/2022, 2:14 PM
Thanks for reporting. What are the circumstances? Is this still occuring?
r

Ross Teach

09/02/2022, 2:16 PM
Yeah I'm still seeing the same error. Running the agent on ec2 if that helps.
k

Khuyen Tran

09/02/2022, 2:43 PM
which command did you run before seeing this error?
r

Ross Teach

09/02/2022, 2:44 PM
ahh gotcha, yeah I'm seeing these in my flow-runs
I've seen the same flow run successfully then throw this error an hour later type of thing. Happens randomly in the middle of run
k

Khuyen Tran

09/02/2022, 2:46 PM
Did you create the flow run through deployment?
r

Ross Teach

09/02/2022, 2:46 PM
yeah my deployments are using docker-images
deployment = Deployment.build_from_flow(
    flow=main,
    name='prod',
    description='Google Campaign Manager 360 DBT Job',
    tags=[Tag.PROD.value, Tag.GOOGLE.value, Tag.DBT.value],
    version=version,
    work_queue_name='prod',
    schedule=CronSchedule(cron='30 7 * * *', timezone='America/New_York'),
    storage=S3.load('prod'),
    infrastructure=DockerContainer(
        image=f'<http://ghcr.io/kortxio/hub-prefect:{version}|ghcr.io/kortxio/hub-prefect:{version}>').dict(),
    infra_overrides=dict(env={'PREFECT_LOGGING_LEVEL': 'DEBUG'}),
)
k

Khuyen Tran

09/02/2022, 2:48 PM
so some flow runs are successful while some are not?
r

Ross Teach

09/02/2022, 2:49 PM
yes
I've 500s for https://api.prefect.cloud/api/accounts/043b2649-9d07-4c5e-8225-521ba2275e68/workspace[…]89d/task_runs/875fd2bb-334f-4b3c-b5f1-8374a133f892/set_state as well
refect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/043b2649-9d07-4c5e-8225-521ba2275e68/workspaces/689b139b-a725-4c2b-b167-86a705b8789d/task_runs/875fd2bb-334f-4b3c-b5f1-8374a133f892/set_state>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
j

Jeff Hale

09/02/2022, 3:01 PM
Thank you. Folks are checking on our end.
Hi Ross. The service is working okay, generally. Unfortunately you are seeing something that shouldn’t generally happen. Are you using 2.3.1? If not, I suggest upgrading and trying to run your flow again. If you are still seeing this error, please let us know some more specifics about the flow code and deployment to the extent that is possible.
#prefect-communityJoin Slack
Powered by